01_ObjDetect.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"images/DLI Header.png\" alt=\"Header\" width=\"400\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.0 Object Detection Application\n",
    "The DeepStream SDK offers a complete set of sample reference applications and pre-trained neural networks to jump-start development.  In this lab, you'll work with the `deepstream-test1` reference application to find objects in a video stream, annotate them with bounding boxes, and output the annotated stream along with a count of the objects found."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_threethingsio.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll follow the steps below to build your own applications based on the reference app:\n",
    "\n",
    "1.1 **[Build a Basic DeepStream Pipeline](#01_overview)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.1.1 [Sample Application `deepstream-test1`](#test1)<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.1.2 [Sample Application plus RTSP - `deepstream-test1-rtsp-out`](#rtsp)<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.1.3 [Exercise: Build and Run the Base Application](#01_ex_base)<br>\n",
    "1.2 **[Configure an Object Detection Model](#01_change_objects)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.2.1 [Gst-nvinfer Configuration File](#01_config)<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.2.2 [Exercise: Detect Only Two Object Types](#01_ex_change)<br>\n",
    "1.3 **[Modify Metadata to Perform Analysis](#01_count_objects)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.3.1 [Extracting Metadata with a GStreamer Probe](#01_probe)<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.3.2 [Exercise: Count Vehicles and Bikes](#01_ex_count)<br>\n",
    "1.4 **[Put It All Together](#01_final)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 1.4.1 [Exercise: Detect and Count three Object Types](#01_ex_challenge)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_overview'></a>\n",
    "# 1.1 Build a Basic DeepStream Pipeline\n",
    "The framework used to build a DeepStream application is a GStreamer **pipeline** consisting of a video input stream, a series of **elements** or **plugins** to process the stream, and an insightful output stream. Each plugin has a defined input, also called its **sink**, and defined output, known as its **source**.  In the pipeline, the source pad of one plugin connects to the sink pad of the next in line.  The source includes information extracted from the processing, the **metadata**, which can be used for annotation of the video and other insights about the input stream.      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_building_blocks.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='test1'></a>\n",
    "## 1.1.1 Sample Application - `deepstream-test1`\n",
    "The DeepStream SDK includes plugins for building a pipeline, and some reference test applications. For example, the `deepstream_test1` application can take a street scene video file as input, use object detection to find vehicles, people, bicycles, and road signs within the video, and output a video stream with bounding boxes around the objects found."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_exampleio2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reference test applications are in the `deepstream_sdk_v4.0.2_jetson/sources/apps/sample_apps/` directory.  You can take a look at the C code for the `deepstream-test1` app at [deepstream_sdk_v4.0.2_jetson/sources/apps/sample_apps/deepstream-test1/deepstream_test1_app.c](../deepstream_sdk_v4.0.2_jetson/sources/apps/sample_apps/deepstream-test1/deepstream_test1_app.c)<br><br>\n",
    "Looking at the code, we can find where all the plugins are instantiated in `main` using the `gst_element_factory_make` method.  This is a good way to see exactly which plugins are in the pipeline *(Note: the sample snippets below are abbreviated code for clarity purposes)*:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```c\n",
    "...\n",
    "  /* Create gstreamer elements */\n",
    "  /* Create Pipeline element that will form a connection of other elements */\n",
    "  pipeline = gst_pipeline_new (\"dstest1-pipeline\");\n",
    "\n",
    "  /* Source element for reading from the file */\n",
    "  source = gst_element_factory_make (\"filesrc\", \"file-source\");\n",
    "\n",
    "  /* Since the data format in the input file is elementary h264 stream,\n",
    "   * we need a h264parser */\n",
    "  h264parser = gst_element_factory_make (\"h264parse\", \"h264-parser\");\n",
    "\n",
    "  /* Use nvdec_h264 for hardware accelerated decode on GPU */\n",
    "  decoder = gst_element_factory_make (\"nvv4l2decoder\", \"nvv4l2-decoder\");\n",
    "\n",
    "  /* Create nvstreammux instance to form batches from one or more sources. */\n",
    "  streammux = gst_element_factory_make (\"nvstreammux\", \"stream-muxer\");\n",
    "\n",
    "  /* Use nvinfer to run inferencing on decoder's output,\n",
    "   * behaviour of inferencing is set through config file */\n",
    "  pgie = gst_element_factory_make (\"nvinfer\", \"primary-nvinference-engine\");\n",
    "\n",
    "  /* Use convertor to convert from NV12 to RGBA as required by nvosd */\n",
    "  nvvidconv = gst_element_factory_make (\"nvvideoconvert\", \"nvvideo-converter\");\n",
    "\n",
    "  /* Create OSD to draw on the converted RGBA buffer */\n",
    "  nvosd = gst_element_factory_make (\"nvdsosd\", \"nv-onscreendisplay\");\n",
    "\n",
    "  /* Finally render the osd output */\n",
    "  transform = gst_element_factory_make (\"nvegltransform\", \"nvegl-transform\");\n",
    "  sink = gst_element_factory_make (\"nveglglessink\", \"nvvideo-renderer\");\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the input is a file source, `filesrc`, in H.264 video format, which is decoded and then run through the `nvinfer` inference engine to detect objects.  A buffer is created with `nvvideoconvert` so that bounding boxes can be overlaid on the video images with the `nvdsosd` plugin.  Finally, the output is rendered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='rtsp'></a>\n",
    "## 1.1.2 Sample Application plus RTSP - `deepstream-test1-rtsp-out`\n",
    "For the purposes of this lab, which runs headless on a Jetson Nano connected to a laptop, the video stream must be converted to a format that can be transferred to the laptop media player.  This is accomplished by customizing the sample app with additional plugins and some logic. Some specific customized apps are included in this lab in the `dli_apps` directory.  Take a look at the C code in [/home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out/deepstream_test1_app.c](../deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out/deepstream_test1_app.c).<br><br>\n",
    "Scrolling down to `main`, we can see that there are a few differences in the rendering plugins used for the RTSP protocol transfer of the video stream *(Note: the sample snippets below are abbreviated code for clarity purposes)*:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```c\n",
    "...\n",
    "  /* Finally render the osd output */\n",
    "  transform = gst_element_factory_make (\"nvvideoconvert\", \"transform\");\n",
    "  cap_filter = gst_element_factory_make (\"capsfilter\", \"filter\");\n",
    "  caps = gst_caps_from_string (\"video/x-raw(memory:NVMM), format=I420\");\n",
    "  g_object_set (G_OBJECT (cap_filter), \"caps\", caps, NULL);\n",
    "\n",
    "  encoder = gst_element_factory_make (\"nvv4l2h264enc\", \"h264-encoder\");\n",
    "  rtppay = gst_element_factory_make (\"rtph264pay\", \"rtppay-h264\");\n",
    "\n",
    "  g_object_set (G_OBJECT (encoder), \"bitrate\", 4000000, NULL);\n",
    "\n",
    "#ifdef PLATFORM_TEGRA\n",
    "  g_object_set (G_OBJECT (encoder), \"preset-level\", 1, NULL);\n",
    "  g_object_set (G_OBJECT (encoder), \"insert-sps-pps\", 1, NULL);\n",
    "  g_object_set (G_OBJECT (encoder), \"bufapi-version\", 1, NULL);\n",
    "#endif\n",
    "  sink = gst_element_factory_make (\"udpsink\", \"udpsink\");\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plugins are put in a pipeline bin with the `gst_bin_add_many()` methods  :\n",
    "\n",
    "```c\n",
    "...\n",
    "  /* Set up the pipeline */\n",
    "  /* we add all elements into the pipeline */\n",
    "  gst_bin_add_many (GST_BIN (pipeline),\n",
    "      source, h264parser, decoder, streammux, pgie,\n",
    "      nvvidconv, nvosd, transform, cap_filter, encoder, rtppay, sink, NULL);\n",
    "...\n",
    "```\n",
    "\n",
    "Next, a sink pad (input) for the `streammux` element is created and linked to the `decoder` source pad (output):\n",
    "\n",
    "```c\n",
    "...\n",
    "  GstPad *sinkpad, *srcpad;\n",
    "  gchar pad_name_sink[16] = \"sink_0\";\n",
    "  gchar pad_name_src[16] = \"src\";\n",
    "\n",
    "  sinkpad = gst_element_get_request_pad (streammux, pad_name_sink);\n",
    "  if (!sinkpad) {\n",
    "    g_printerr (\"Streammux request sink pad failed. Exiting.\\n\");\n",
    "    return -1;\n",
    "  }\n",
    "\n",
    "  srcpad = gst_element_get_static_pad (decoder, pad_name_src);\n",
    "  if (!srcpad) {\n",
    "    g_printerr (\"Decoder request src pad failed. Exiting.\\n\");\n",
    "    return -1;\n",
    "  }\n",
    "\n",
    "  if (gst_pad_link (srcpad, sinkpad) != GST_PAD_LINK_OK) {\n",
    "      g_printerr (\"Failed to link decoder to stream muxer. Exiting.\\n\");\n",
    "      return -1;\n",
    "  }\n",
    "\n",
    "...\n",
    "```\n",
    "\n",
    "Finally, the elements are linked together using the `gst_element_link_many()` method.  The start of the pipeline through the `decoder` are linked together, and the `streammux` and beyond are linked together, to form the entire pipeline.\n",
    "\n",
    "```c\n",
    "...\n",
    "  /* we link the elements together */\n",
    "  /* file-source -> h264-parser -> nvh264-decoder ->\n",
    "   * nvinfer -> nvvidconv -> nvosd -> video-renderer */\n",
    "\n",
    "  if (!gst_element_link_many (source, h264parser, decoder, NULL)) {\n",
    "    g_printerr (\"Elements could not be linked: 1. Exiting.\\n\");\n",
    "    return -1;\n",
    "  }\n",
    "\n",
    "  if (!gst_element_link_many (streammux, pgie,\n",
    "      nvvidconv, nvosd, transform, cap_filter, encoder, rtppay, sink, NULL)) {\n",
    "    g_printerr (\"Elements could not be linked: 2. Exiting.\\n\");\n",
    "    return -1;\n",
    "  }\n",
    "\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, the pipeline for this app consists of the following plugins (ordered):\n",
    "\n",
    "- `GstFileSrc` - reads the video data from file\n",
    "- `GstH264Parse` - parses the incoming H264 stream\n",
    "- `Gst-nvv4l2decoder` - hardware accelerated decoder; decodes video streams using NVDEC\n",
    "- `Gst-nvstreammux` - batch video streams before sending for AI inference\n",
    "- `Gst-nvinfer` - runs inference using TensorRT\n",
    "- `Gst-nvvideoconvert` - performs video color format conversion (I420 to RGBA)\n",
    "- `Gst-nvdsosd` - draw bounding boxes, text and region of interest (ROI) polygons\n",
    "- `Gst-nvvideoconvert` - performs video color format conversion (RGBA to I420)\n",
    "- `GstCapsFilter` - enforces limitations on data (no data modification)\n",
    "- `Gst-nvv4l2h264enc` - encodes RAW data in I420 format to H264\n",
    "- `GstRtpH264Pay` - converts H264 encoded Payload to RTP packets (RFC 3984)\n",
    "- `GstUDPSink` - sends UDP packets to the network. When paired with RTP payloader (`Gst-rtph264pay`) it can implement RTP streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_ex_base'></a>\n",
    "## 1.1.3 Exercise: Build and Run the Base Application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `deepstream-test1` example, object detection is performed on a per-frame-basis. Counts for `Vehicle` and `Person` objects are also tracked.  Bounding boxes are drawn around the objects identified, and a counter display is overlayed in the upper left corner of the video. \n",
    "\n",
    "#### Build the DeepStream app\n",
    "Execute the following cell to build the application:\n",
    "- Click on the cell to select it\n",
    "- Press [SHIFT][ENTER] or [CONTROL][ENTER] on your keyboard to execute the instructions in the code cell.  Alternatively, you can click the run button at the top of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out\n",
      "rm -rf deepstream_test1_app.o deepstream-test1-app\n",
      "cc -c -o deepstream_test1_app.o -DPLATFORM_TEGRA -I../../../includes `pkg-config --cflags gstreamer-1.0` deepstream_test1_app.c\n",
      "cc -o deepstream-test1-app deepstream_test1_app.o `pkg-config --libs gstreamer-1.0` -L/opt/nvidia/deepstream/deepstream-4.0/lib/ -lnvdsgst_meta -lnvds_meta -lgstrtspserver-1.0 -Wl,-rpath,/opt/nvidia/deepstream/deepstream-4.0/lib/\n"
     ]
    }
   ],
   "source": [
    "# Build the app\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out\n",
    "!make clean\n",
    "!make"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the DeepStream app\n",
    "Open the VLC media player on your laptop:\n",
    "- Click \"Media\" and open the  \"Open Network Stream\" dialog\n",
    "- Set the URL to `rtsp://192.168.55.1:8554/ds-test`\n",
    "- Start execution of the cell below\n",
    "- Click \"Play\" on your VLC media player right after you start executing the cell.  \n",
    "\n",
    "The stream will start shortly from the Jetson Nano and display in the media player.  If you find you've missed it due to a time out in the media player, try the process again, this time waiting a little longer before starting the media player."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out\n",
      "\n",
      " *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***\n",
      "\n",
      "Now playing: /home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.h264\n",
      "Opening in BLOCKING MODE \n",
      "Opening in BLOCKING MODE \n",
      "Creating LL OSD context new\n",
      "Running...\n",
      "NvMMLiteOpen : Block : BlockType = 261 \n",
      "\n",
      "(deepstream-test1-app:10211): GLib-GObject-\u001b[1;33mWARNING\u001b[0m **: \u001b[34m08:43:52.057\u001b[0m: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'\n",
      "NVMEDIA: Reading vendor.tegra.display-size : status: 6 \n",
      "NvMMLiteBlockCreate : Block : BlockType = 261 \n",
      "Creating LL OSD context new\n",
      "NvMMLiteOpen : Block : BlockType = 4 \n",
      "===== NVMEDIA: NVENC =====\n",
      "NvMMLiteBlockCreate : Block : BlockType = 4 \n",
      "H264: Profile = 66, Level = 0 bjects =   5 Vehicle Count =   3 Person Count =   2\n",
      "Frame Number =  25 Number of objects =   8 Vehicle Count =   6 Person Count =   2\n",
      "(deepstream-test1-app:10211): GLib-GObject-\u001b[1;33mWARNING\u001b[0m **: \u001b[34m08:43:55.147\u001b[0m: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'\n",
      "Frame Number = 1441 Number of objects =   0 Vehicle Count =   0 Person Count =   0\n",
      "End of stream\n",
      "Returned, stopping playback\n",
      "Deleting pipeline\n"
     ]
    }
   ],
   "source": [
    "# Run the app\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out\n",
    "!./deepstream-test1-app /home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.h264"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_change_objects'></a>\n",
    "# 1.2 Configure an Object Detection Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample application shows counts for two types of objects: `Vehicle` and `Person`.  However, the model that is used can actually detect four types of objects, as revealed in the application C code (line 46):\n",
    "\n",
    "```c\n",
    "gchar pgie_classes_str[4][32] = { \"Vehicle\", \"TwoWheeler\", \"Person\",\n",
    "  \"Roadsign\"\n",
    "};\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_config'></a>\n",
    "## 1.2.1 `Gst-nvinfer` Configuration File\n",
    "This information is specific to the model used for the inference, which in this case is a sample model provided with the DeepStream SDK.  The `Gst-nvinfer` plugin employs a configuration file to specify the model and various properties. Open the configuration file for the app we are using at [/home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out/dstest1_pgie_config.txt](../deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out/dstest1_pgie_config.txt).  The `Gst-nvinfer` configuration file uses a “Key File” format, with details on key names found in the DeepStream Plugin Manual (use the link provided in the class pages for more details). \n",
    "- The **\\[property\\]** group configures the general behavior of the plugin. It is the only mandatory group.\n",
    "- The **\\[class-attrs-all\\]** group configures detection parameters for all classes.\n",
    "- The **\\[class-attrs-\\<class-id\\>\\]** group configures detection parameters for a class specified by \\<class-id\\>. For example, the \\[class-attrs-2\\] group configures detection parameters for class ID 2\\. This type of group has the same keys as \\[class-attrs-all\\]. \n",
    "\n",
    "Note that the number of classes and the ordered `labels.txt` file are specified in the \\[property\\] group along with the model engine. For this exercise, we are more interested in configuring the \\[class-attrs-all\\] and \\[class-attrs-\\<class-id\\>\\] groups.  In the sample, we see the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```c\n",
    "[class-attrs-all]\n",
    "threshold=0.2\n",
    "eps=0.2\n",
    "group-threshold=1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `threshold=0.2` key sets the detection confidence score. This tells us that all objects with a 20% confidence score or better will be marked as detected. If the threshold were greater than 1.0, then no objects could ever be detected.  \n",
    "\n",
    "This \"all\" grouping is not granular enough if we only want to detect a subset of the objects possible, or if we want to use a different confidence level with different objects.  For example, we might want to detect only vehicles, or we might want to identify people with a different confidence level than road signs.  To specify a threshold for the four individual objects available in this model, add a specific group to the config file for each class: \n",
    "* \\[class-attrs-0\\] for vehicles\n",
    "- \\[class-attrs-1\\] for bicycles\n",
    "- \\[class-attrs-2\\] for persons\n",
    "- \\[class-attrs-3\\] for road signs\n",
    "\n",
    "In each group, we can now specify the threshold value.  This will be used to determine object detection for each of the four object categories individually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_ex_change'></a>\n",
    "## 1.2.2 Exercise: Detect Only Two Object Types\n",
    "Create a new app based on `deepstream-test1-rtsp_out` that detects **only** cars and bicycles.  Start by copying the existing app to a new workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new app located at /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst1-two-objects \n",
    "#      based on deepstream-test1-rtsp_out\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps\n",
    "!mkdir -p my_apps/dst1-two-objects\n",
    "!cp -rfv dli_apps/deepstream-test1-rtsp_out/* my_apps/dst1-two-objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using what you just learned, modify the [configuration file](../deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst1-two-objects/dstest1_pgie_config.txt) in your new app to only detect cars and bicycles.  You will need to add *class-specific groups* for each of the four classes to the end of your configuration file.<br>\n",
    "Class-specific example:\n",
    "   ```\n",
    "   # Per class configuration\n",
    "   # car\n",
    "   [class-attrs-0] \n",
    "   threshold=0.2\n",
    "   ```\n",
    "Then, build and run the app to see if it worked!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the app\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst1-two-objects\n",
    "!make clean\n",
    "!make"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the app\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst1-two-objects\n",
    "!./deepstream-test1-app /home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.h264"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How did you do?\n",
    "If you see something like this image, you did it!  If not, keep trying or take a peek at the solution code in the solutions directory. If you aren't satisfied with the detection of the bicycle, you can experiment with the confidence threshold value. <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_bikes_and_cars.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_count_objects'></a>\n",
    "# 1.3 Modify Metadata to Perform Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object detection is working well, but we are only counting the `Person` and `Vehicle` objects detected.  We would like to show the counts for the bicycles instead of people.  The `Gst-nvinfer` plugin finds objects and provides metadata about them as an output on its source pad, which is passed along through the pipeline.   Using a GStreamer **probe**, we can take a look at the metadata and count the objects detected downstream.  This extraction of the information occurs at the input, or \"sink pad\", of the `Gst-nvdsosd` plugin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_test1_app.png\" >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_probe'></a>\n",
    "## 1.3.1 Extracting Metadata with a GStreamer Probe\n",
    "The `osd_sink_pad_buffer_probe` code in [the deepstream-test1 app](../deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test1-rtsp_out/deepstream_test1_app.c) is a callback that is run each time there is new frame data. With this probe, we can snapshot the metadata coming into the `Gst-nvdsosd` plugin, and count the current objects.  The metadata collected that we want to look at will be collected in `obj_meta`: \n",
    "\n",
    "```c\n",
    "NvDsObjectMeta *obj_meta = NULL;\n",
    "```\n",
    "\n",
    "The `NvDsObjectMeta` data structure includes an element for the `class_id`.  This is the same class number used in the config file to identify object types: \n",
    "* 0 for vehicles\n",
    "* 1 for bicycles\n",
    "* 2 for persons\n",
    "* 3 for road signs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The _for_ loop in the probe checks the `obj_meta->class_id` value for every object in the frame and counts them as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```c\n",
    "#define PGIE_CLASS_ID_VEHICLE 0\n",
    "#define PGIE_CLASS_ID_PERSON 2\n",
    "\n",
    "...\n",
    "\n",
    "static GstPadProbeReturn\n",
    "osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,\n",
    "    gpointer u_data)\n",
    "{\n",
    "    GstBuffer *buf = (GstBuffer *) info->data;\n",
    "    guint num_rects = 0; \n",
    "    NvDsObjectMeta *obj_meta = NULL;\n",
    "    guint vehicle_count = 0;\n",
    "    guint person_count = 0;\n",
    "    NvDsMetaList * l_frame = NULL;\n",
    "    NvDsMetaList * l_obj = NULL;\n",
    "    NvDsDisplayMeta *display_meta = NULL;\n",
    "\n",
    "    NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);\n",
    "\n",
    "    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;\n",
    "      l_frame = l_frame->next) {\n",
    "        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);\n",
    "        int offset = 0;\n",
    "        for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;\n",
    "                l_obj = l_obj->next) {\n",
    "            obj_meta = (NvDsObjectMeta *) (l_obj->data);\n",
    "            if (obj_meta->class_id == PGIE_CLASS_ID_VEHICLE) {\n",
    "                vehicle_count++;\n",
    "                num_rects++;\n",
    "            }\n",
    "            if (obj_meta->class_id == PGIE_CLASS_ID_PERSON) {\n",
    "                person_count++;\n",
    "                num_rects++;\n",
    "            }\n",
    "        }\n",
    "...\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The count for each is then added to a display buffer, which is then added to the frame metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```c\n",
    "...\n",
    "    \n",
    "        display_meta = nvds_acquire_display_meta_from_pool(batch_meta);\n",
    "        NvOSD_TextParams *txt_params  = &display_meta->text_params[0];\n",
    "        display_meta->num_labels = 1;\n",
    "        txt_params->display_text = g_malloc0 (MAX_DISPLAY_LEN);\n",
    "        offset = snprintf(txt_params->display_text, MAX_DISPLAY_LEN, \"Person = %d \", person_count);\n",
    "        offset = snprintf(txt_params->display_text + offset , MAX_DISPLAY_LEN, \"Vehicle = %d \", vehicle_count);\n",
    "\n",
    "...\n",
    "    \n",
    "        nvds_add_display_meta_to_frame(frame_meta, display_meta);\n",
    "    }\n",
    "      \n",
    "...\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, there are four places that require changes if we want to modify the counts:\n",
    "* Constants for the class ID values (similar to `PGIE_CLASS_ID_VEHICLE`)\n",
    "* Variables to track the counts (similar to `vehicle_count`\n",
    "* _if_ statements to check the objects and count them\n",
    "* `snprintf` statements to fill the buffer for displaying the counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_ex_count'></a>\n",
    "## 1.3.2 Exercise: Count Vehicles and Bikes\n",
    "Create a new app based on `deepstream-test1-rtsp_out` that shows counts for vehicles and bicycles.  Fill in the following cells with appropriate commands to create, build, and run your app. To edit your files, use the JupyterLab file browser at left to navigate to the correct folder; then, double click on the file you wish to open and edit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Create a new app located at /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst1-counts \n",
    "#     based on deepstream-test1-rtsp_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Edit the C-code to count vehicles and bicycles\n",
    "# Build the app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Run the app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How did you do?\n",
    "If you see something like this image, you did it!  If not, keep trying or take a peek at the solution code in the solutions directory. You can also modify the `g_print` lines to provide bicycle count feedback while the stream is running. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_counts.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_final'></a>\n",
    "# 1.4 Putting It All Together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great job!  You've learned how to build a pipeline, detect various objects, and probe/modify the metadata to count the objects.  It's time to put what you've learned about objects and metadata into one new app."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='01_ex_challenge'></a>\n",
    "## 1.4.1 Exercise: Detect and Count Three Object Types\n",
    "Create a new app based on `deepstream-test1-rtsp_out` that detects and shows counts for only three kinds of objects: persons, vehicles, and bicycles.  Adjust the confidence values if needed for each.  Fill in the following cells with appropriate commands to create, build, and run your app.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Create a new app located at /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst1-three-things"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Edit the C-code to include counts for Persons, Vehicles, and Bikes\n",
    "# Hint: For the offset in the display, you will need to account for two different offsets to properly place the third value.\n",
    "# Build the app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Run the app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How did you do?\n",
    "If you see something like this image, you did it!  If not, keep trying or take a peek at the solution code in the solutions directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/01_three_things.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style=\"color:green;\">Congratulations!</h2>\n",
    "\n",
    "You've created DeepStream apps to detect and count objects in a scene in various configurations.<br>\n",
    "Move on to [2.0 Multiple Networks Application](./02_MultipleNetworks.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"images/DLI Header.png\" alt=\"Header\" width=\"400\"></center>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}