{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data insertion and retrieval with Django models\n",
    "\n",
    "In the following we show some examples in order to perform insertions and retrievals of metadata "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imagedb.models import Instrument\n",
    "\n",
    "instrument = Instrument.objects.get(instrumentName='NISP')\n",
    "\n",
    "print(instrument.telescopeName)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating new objects\n",
    "\n",
    "In the following we will create a NispRawFrame object. Since the NispRawFrame must have also a DataCotainer (i.e. a file to reference to), we first create a DataContainer instance.\n",
    "\n",
    "We can see also that the fields that we have defined as composite fields provide some synctactic sugar when we initialize them with simple dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imagedb.models import ImageType, Pointing, NispDetector, DataContainer, NispRawFrame\n",
    "\n",
    "from datetime import datetime\n",
    "\n",
    "dataFile = DataContainer(\n",
    "  fileFormat = 'fits',\n",
    "  formatIdentifier = 'le1.nisprawframe',\n",
    "  formatVersion = '1.0',\n",
    "  url = \"http://ia2-owncloud.oats.inaf.it/fake/7ff2f203/data/EUC_LE1_NISP_53892-Y-1_20170712T155430.1Z_00.00.fits\"\n",
    ")\n",
    "\n",
    "# We have to save the data container to the DB before assigning it to a NispRawFrame\n",
    "dataFile.save()\n",
    "\n",
    "image = NispRawFrame(exposureTime = 105,\n",
    "                     imgNumber = 16,\n",
    "                     naxis1 = 2040,\n",
    "                     naxis2 = 2040,\n",
    "                     imageType = {'category':'SCIENCE', \n",
    "                                  'firstType':'OBJECT', \n",
    "                                  'secondType':'STD'},\n",
    "                     observationDateTime = datetime.strptime(\"2025-06-21T18:27:23.000001\", \n",
    "                                                             \"%Y-%m-%dT%H:%M:%S.%f\"),\n",
    "                     observationId = 53892,\n",
    "                     ditherNumber = 1,\n",
    "                     instrument = instrument,\n",
    "                     commandedPointing = {'rightAscension':8.48223045516,\n",
    "                                          'declination':8.48223045516,\n",
    "                                          'orientation':64.8793517547},\n",
    "                     filterWheelPosition = \"Y\",\n",
    "                     grismWheelPosition = \"OPEN\"\n",
    "                    )\n",
    "\n",
    "\n",
    "\n",
    "image.frameFile = dataFile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before saving an object, its surrogate key is still empty, i.e. equal to None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(image.id)\n",
    "image.save()\n",
    "print(image.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# We can start creating a detector\n",
    "\n",
    "d11 = NispDetector(detectorId = \"11\", gain = 1.0, readoutNoise = 0.0, rawFrame = image)\n",
    "d11.save()\n",
    "\n",
    "# Or we can use the create() in order to create and save immediately the new object\n",
    "NispDetector.objects.create(detectorId = \"12\", gain = 1.0, readoutNoise = 0.0, rawFrame = image)\n",
    "\n",
    "# or we can create the detector starting from the NispRawFrame, using the reversed relationship,\n",
    "# using again the create method\n",
    "\n",
    "image.detectors.create(\n",
    "  detectorId = \"13\",\n",
    "  gain = 1.0,\n",
    "  readoutNoise = 0.0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objects retrieval\n",
    "\n",
    "To retrieve objects from your database, construct a **QuerySet** via a **Manager** on your model class. \n",
    "\n",
    "A **QuerySet** represents a collection of objects from your database. It can have zero, one or many filters. Filters narrow down the query results based on the given parameters. In SQL terms, a **QuerySet** equates to a **SELECT** statement, and a **filter** is a limiting clause such as **WHERE** or **LIMIT**.\n",
    "\n",
    "You get a **QuerySet** by using your model’s **Manager**. Each model has at least one Manager, and it’s called **objects** by default.\n",
    "\n",
    "The simplest way to retrieve objects from a table is to get all of them. To do this, use the **all()** method on a **Manager**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(NispRawFrame.objects.all())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But usually we want to filter the results. For this purpose we can use the **filter** method, both provided by the **Manager** and the **QuerySet**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieving all frames with observation id 53877 and filter Y\n",
    "# and ordering the results by the ditherNumber\n",
    "\n",
    "result = NispRawFrame.objects.filter(observationId=53877, \n",
    "                                     filterWheelPosition='Y').order_by('ditherNumber')\n",
    "\n",
    "for obj in result:\n",
    "    print(obj.observationId, obj.filterWheelPosition, obj.ditherNumber)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also limit the number of results of a **QuerySet**, using the Python array-slice syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "# Retrieving all NISP raw frames with observation date and time \n",
    "# greater then or equal to 2026-06-22T17:00\n",
    "len(NispRawFrame.objects.filter(observationDateTime__gte=datetime(2025,6,22,17,0)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now limiting the result to 5 items\n",
    "\n",
    "len(NispRawFrame.objects.filter(observationDateTime__gte=datetime(2025,6,22,17,0))[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keyword argument queries – in filter(), etc. – are “AND”ed together. If you need to execute more complex queries (for example, queries with OR statements), you can use **Q objects**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now retrieving all frames with observation ids 53877 and 54349 and filter H\n",
    "from django.db.models import Q\n",
    "\n",
    "result = NispRawFrame.objects.filter(Q(observationId=53877) | Q(observationId=54349), \n",
    "                                     filterWheelPosition='H').order_by('observationId', 'ditherNumber')\n",
    "\n",
    "for obj in result:\n",
    "    print(obj.observationId, obj.filterWheelPosition, obj.ditherNumber)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also traverse relations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting all NispDetectors whose frame observation id is 53877 or 54349 and \n",
    "# filterWheelPosition is Y.\n",
    "# Notice that here we also use the 'in' operator\n",
    "\n",
    "result = NispDetector.objects.filter(rawFrame__observationId__in=[53877, 54349],\n",
    "                                     rawFrame__filterWheelPosition='Y').order_by(\n",
    "                                     'rawFrame__observationId', 'rawFrame__ditherNumber', \n",
    "                                     'detectorId')\n",
    "\n",
    "for obj in result:\n",
    "    print(obj.rawFrame.observationId, obj.rawFrame.ditherNumber, obj.detectorId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OR we can find the NispRawFrame whose referenced file url contains \"LE1_NISP_52926-J-2\"\n",
    "\n",
    "result = NispRawFrame.objects.filter(frameFile__url__contains=\"LE1_NISP_52926-J-2\")\n",
    "\n",
    "for obj in result:\n",
    "    print(obj.observationId, obj.frameFile.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to retrive a single object, instead of using **filter** we can use the **get** method. This method returns one object and raise exceptions if no object is found or if multiple objects satisfy the query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obj = NispRawFrame.objects.get(observationId=53892)\n",
    "\n",
    "# now we delete such object from the database\n",
    "\n",
    "obj.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serializers\n",
    "\n",
    "Let's see some differences between the plain Django serializers and the ModelSerializer class provided by the Django REST framework.\n",
    "The first example uses the Django core serializers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from django.core import serializers\n",
    "\n",
    "data = serializers.serialize('json',NispRawFrame.objects.filter(observationId=53877, \n",
    "                            filterWheelPosition='Y').order_by('ditherNumber'))\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example, instead, uses the Django REST framework ModelSerializer class. In particular, see the file imagedb/serializers.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imagedb.serializers import NispRawFrameSerializer\n",
    "import json\n",
    "\n",
    "frame = NispRawFrameSerializer(NispRawFrame.objects.get(id=1))\n",
    "print(json.dumps(frame.data, indent=2))\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Django Shell-Plus",
   "language": "python",
   "name": "django_extensions"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}