03_MultiStream.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"images/DLI Header.png\" alt=\"Header\" width=\"400\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.0  Multiple Stream Input\n",
    "Multiple input video streams can be batched together through a single inference pipeline, producing an annotated tiled output.  Any input format supported by GStreamer can be used for the input streams, and any number of streams may be tiled together up to the maximum number that the Jetson can handle - eight streams in the case of the Jetson Nano.  In this notebook, you'll work with the `deepstream-test3` reference application to build applications that use varying numbers of input streams and formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/03_test3_example.png\" alt=\"beginning and end image with pipe\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.1 **[Build a Pipeline with Multiple Input Streams in Parallel](#03_overview)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.1.1 [Practice Application `deepstream-test3-rtsp_out`](#03_test3)<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.1.2 [Resolution and Timing](#03_mux)<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.1.3 [Exercise: Build and Run the Base Application](#03_ex_base)<br>\n",
    "3.2 **[Configure Multiple Input Streams with Different Formats](#03_formats)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.2.1 [GStreamer Autoplugging with `uridecodebin`](#03_uri)<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.2.2 [Batch Size Configuration](#03_batch)<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.2.3 [Exercise: Add an Input Source](#03_ex_change)<br>\n",
    "3.3 **[Put It All Together](#03_final)**<br>\n",
    "&nbsp; &nbsp; &nbsp; 3.3.1 [Exercise: Eight Input Streams](#03_ex_challenge)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_overview\"></a>\n",
    "# 3.1 Build a Pipeline with Multiple Input Streams in Parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The application accepts one or more video streams as input.  You can use any GStreamer supported file  or streamed format.  Reference the stream using a **Uniform Resource Identifier (URI)** as input, e.g. a URI with syntax `file:///path/to/file` for a video file.  \n",
    "\n",
    "The sample application creates a **source bin**, which is basically a small decoder pipeline, for each input stream.  It then connects the source bins to the `Gst-nvstreammux` plugin.  This forms a **batch** of frames equal to the number of inputs. The batch of frames is fed to the `Gst-nvinfer` plugin for batched inferencing. A batched buffer is composited into a 2D tile array using the `Gst-nvmultistreamtiler` plugin. The rest of the pipeline is similar to the DeepStream pipeline used in the Object Detection exercises.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_test3\"></a>\n",
    "## 3.1.1 Practice Application `deepstream-test3-rtsp_out` \n",
    "\n",
    "As with other reference applications in this course, a modified version with RSTP output is provided.  Take a look at the `main` code in  [deepstream-test3-rtsp_out/deepstream_test3_app.c](../deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test3-rtsp_out/deepstream_test3_app.c) to analyze the pipeline.  Here's a condensed snippet of the binning and linking portion *(Note: the sample snippets below are abbreviated code for clarity purposes)*:\n",
    "\n",
    "```c\n",
    "...\n",
    "    \n",
    "  gst_bin_add (GST_BIN (pipeline), streammux);\n",
    "\n",
    "...\n",
    "\n",
    "  /* Set up the pipeline */\n",
    "  /* we add all elements into the pipeline */\n",
    "  gst_bin_add_many (GST_BIN (pipeline), pgie, tiler, nvvidconv, nvosd,\n",
    "      transform, cap_filter, encoder, rtppay, sink, NULL);\n",
    "  /* we link the elements together\n",
    "   * nvstreammux -> nvinfer -> nvtiler -> nvvidconv -> nvosd -> video-renderer */\n",
    "  if (!gst_element_link_many (streammux, pgie, tiler, nvvidconv, nvosd,\n",
    "          transform, cap_filter, encoder, rtppay, sink, NULL)) {\n",
    "    g_printerr (\"Elements could not be linked. Exiting.\\n\");\n",
    "    return -1;\n",
    "  }\n",
    "       \n",
    "...\n",
    "```\n",
    "\n",
    "At first glance, it may not be at all clear how the various input streams are linked into the `Gst-nvstreammux` (named `streammux`) element.  The explicitly defined `GstFileSrc`, `GstH264Parse` and `Gst-nvv4l2decoder` elements we saw in `deepstream-test1` and `deepstream-test2` are simply not in this app.  Instead, this app uses the GStreamer `uridecoderbin` element to create the source and decoder elements dynamically, based on the input stream format. This mechanism will be covered in more depth in a later section.\n",
    "\n",
    "In summary, this DeepStream application uses the following elements for its pipeline:\n",
    "- `uridecoderbin` source bins - one or more instances created at runtime to read the video data\n",
    "- `Gst-nvstreammux` - batch video streams before sending for AI inference\n",
    "- `Gst-nvinfer` - runs inference using TensorRT\n",
    "- `Gst-nvmultistreamtiler` - plugin composites a 2D tile from batched buffers\n",
    "- `Gst-nvvideoconvert` - performs video color format conversion (I420 to RGBA)\n",
    "- `Gst-nvdsosd` - draw bounding boxes, text and region of interest (ROI) polygons\n",
    "- `Gst-nvvideoconvert` - performs video color format conversion (RGBA to I420)\n",
    "- `GstCapsFilter` - enforces limitations on data (no data modification)\n",
    "- `Gst-nvv4l2h264enc` - encodes RAW data in I420 format to H264\n",
    "- `GstRtpH264Pay` - converts H264 encoded Payload to RTP packets (RFC 3984)\n",
    "- `GstUDPSink` - sends UDP packets to the network. When paired with RTP payloader `Gst-rtph264pay` it can implement RTP streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_mux\"></a>\n",
    "## 3.1.2 Resolution and Timing\n",
    "\n",
    "Since we are batching more than one input stream, there are a few resolution, timing, and ordering parameters to consider:\n",
    "* Height and width properties must be set on the `Gst-nvstreammux` plugin for the output resolution. If the input frame resolution is different from the `Gst-nvstreammux` plugin height and width, the input frame will be scaled to the `Gst-nvstreammux` output resolution.  \n",
    "* `Gst-nvstreammux` waits for a user-defined timeout before forming the batch. The timeout is set using the \"batched-push-timeout\" property. If the complete batch is formed before the timeout is reached, the batch is pushed to the downstream element. If the timeout is reached before the complete batch can be formed (which can happen when using RTSP sources), the batch is formed from the _available_ input buffers and pushed. Ideally, the `Gst-nvstreammux` timeout should be set based on the framerate of the fastest source. It can also be set to -1 to make the `Gst-nvstreammux` wait indefinitely.\n",
    "* `Gst-nvmultistreamtiler` composites streams based on their stream-ids in row-major order (starting from stream 0, left to right across the top row, then across the next row, etc.).\n",
    "\n",
    "These parameters are set in the C code for the app:\n",
    "```c\n",
    "...\n",
    "    \n",
    "/* The muxer output resolution must be set if the input streams will be of\n",
    " * different resolution. The muxer will scale all the input frames to this\n",
    " * resolution. */\n",
    "#define MUXER_OUTPUT_WIDTH 1920\n",
    "#define MUXER_OUTPUT_HEIGHT 1080\n",
    "\n",
    "/* Muxer batch formation timeout, for e.g. 40 millisec. Should ideally be set\n",
    " * based on the fastest source's framerate. */\n",
    "#define MUXER_BATCH_TIMEOUT_USEC 4000000\n",
    "\n",
    "#define TILED_OUTPUT_WIDTH 1920\n",
    "#define TILED_OUTPUT_HEIGHT 1080\n",
    "    \n",
    "...\n",
    "```\n",
    "For these exercises, we will use video files provided in the DeepStream SDK Samples directory for our streams.  This is for convenience and demonstration purposes.  Any format supported by the GStreamer (a long list) will work.  Execute the following cell to see a list of video files available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_ex_base\"></a>\n",
    "## 3.1.3 Exercise: Build and Run the Base Application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the test3 app\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test3-rtsp_out/\n",
    "!make clean\n",
    "!make"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the DeepStream app\n",
    "Open the VLC media player on your laptop:\n",
    "- Click \"Media\" and open the  \"Open Network Stream\" dialog\n",
    "- Set the URL to `rtsp://192.168.55.1:8554/ds-test`\n",
    "- Start execution of the cell below\n",
    "- Click \"Play\" on your VLC media player right after you start executing the cell.  \n",
    "\n",
    "The stream may be delayed a bit while the required source and decoder elements are dynamically determined. The output stream will then start from the Jetson Nano and display in the media player."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the app with two input streams\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test3-rtsp_out/\n",
    "!./deepstream-test3-app \\\n",
    "file:///home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.h264 \\\n",
    "file:///home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.mp4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_formats\"></a>\n",
    "# 3.2 Configure Multiple Input Streams with Different Formats\n",
    "Input streams with different formats are automatically accommodated with matching decoders at runtime using **autoplugging**.  There is _no configuration necessary_ when changing _formats_ for this app!\n",
    "\n",
    "However, when changing _the number of inputs (batch size)_, configuration is recommended for optimal performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_uri\"></a>\n",
    "## 3.2.1 GStreamer Autoplugging with `uridecodebin`\n",
    "Each input stream is sourced and configured with the aid of the `uridecodebin` GStreamer element at runtime.  This does lengthen the startup time for the application, but also offers robust flexibility. A `for` loop in `main` creates a source bin for each URI input on the command line (see `create_source_bin()` in [deepstream_test3_app.c](../deepstream_sdk_v4.0.2_jetson/sources/apps/dli_apps/deepstream-test3-rtsp_out/deepstream_test3_app.c)). *(Note: the sample snippets below are abbreviated code for clarity purposes)* \n",
    "\n",
    "```c\n",
    "static GstElement *\n",
    "create_source_bin (guint index, gchar * uri)\n",
    "    \n",
    "...\n",
    "    \n",
    "  /* Source element for reading from the uri.\n",
    "   * We will use decodebin and let it figure out the container format of the\n",
    "   * stream and the codec and plug the appropriate demux and decode plugins. */\n",
    "  uri_decode_bin = gst_element_factory_make (\"uridecodebin\", \"uri-decode-bin\");\n",
    "\n",
    "...\n",
    "    \n",
    "  /* Connect to the \"pad-added\" signal of the decodebin which generates a\n",
    "   * callback once a new pad for raw data has been created by the decodebin */\n",
    "  g_signal_connect (G_OBJECT (uri_decode_bin), \"pad-added\",\n",
    "      G_CALLBACK (cb_newpad), bin);\n",
    "  g_signal_connect (G_OBJECT (uri_decode_bin), \"child-added\",\n",
    "      G_CALLBACK (decodebin_child_added), bin);\n",
    "\n",
    "  gst_bin_add (GST_BIN (bin), uri_decode_bin);\n",
    "\n",
    "  /* We need to create a ghost pad for the source bin which will act as a proxy\n",
    "   * for the video decoder src pad. The ghost pad will not have a target right\n",
    "   * now. Once the decode bin creates the video decoder and generates the\n",
    "   * cb_newpad callback, we will set the ghost pad target to the video decoder\n",
    "   * src pad. */\n",
    "  if (!gst_element_add_pad (bin, gst_ghost_pad_new_no_target (\"src\",\n",
    "              GST_PAD_SRC))) {\n",
    "    g_printerr (\"Failed to add ghost pad in source bin\\n\");\n",
    "    return NULL;\n",
    "\n",
    "...\n",
    "    \n",
    "  return bin;\n",
    "}\n",
    "    \n",
    "```\n",
    "Note the element instantiation of a `uridecodebin` element. This element adds and links \"child\" decoder plugin elements to a source bin as needed, based on the actual input stream formatting.  Finally, a source pad is added to the new small pipeline so that it can later be easily linked to the `Gst-nvstreammux` plugin element and the rest of the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_batch\"></a>\n",
    "## 3.2.2 Batch Size Configuration\n",
    "The configuration file includes two parameters related to the number of input streams, or batch size, for the application:\n",
    "* `batch-size` - number of input streams to be batched in parallel\n",
    "* `model-engine-file` - path to the `.engine` file which is built for a specific batch size\n",
    "\n",
    "If the `batch-size` does not match the number of input streams specified at runtime, the app will override the value with the correct number.\n",
    "\n",
    "The `.engine` file is built at runtime if it does not already exist.  If an engine file with the wrong batch size is provided in the configuration file, a warning will appear, and the correct `.engine` will be built and stored prior to running the streams.  This adds time to the start of the run and can be avoided (at least after the first run) by including the correct path for the `.engine`.  Here's some example output with a batch size mismatch:\n",
    "\n",
    "```c\n",
    "0:00:06.165375005  6726   0x5592aa1380 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:checkEngineParams(): Requested Max Batch Size is less than engine batch size\n",
    "0:00:06.166695340  6726   0x5592aa1380 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files\n",
    "0:01:42.046291797  6726   0x5592aa1380 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/models/Primary_Detector/resnet10.caffemodel_b3_fp16.engine\n",
    "```\n",
    "\n",
    "The batch size is coded in the name of the `.engine` file created.  In the above example, the engine file for three input streams was stored as `resnet10.caffemodel_b3_fp16.engine`, whereas the name of the  engine with two input streams is `resnet10.caffemodel_b2_fp16.engine`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_ex_change\"></a>\n",
    "## 3.2.3 Exercise: Add an Input Source\n",
    "Create a new app based on `deepstream-test3-rstp_out` that can accept and tile three video input streams.  The application itself will not need to be modified, but the configuration file will."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new app located at /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst3-three-streams \n",
    "#      based on deepstream-test3-rtsp_out\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps\n",
    "!mkdir -p my_apps/dst3-three-streams\n",
    "!cp -rfv dli_apps/deepstream-test3-rtsp_out/* my_apps/dst3-three-streams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using what you just learned, modify the parameters for `batch-size` and `model-engine-file` in the [configuration file](../deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst3-three-streams/dstest3_pgie_config.txt) of your new app, to expect three input streams.  Then build and run the app to see if it worked!\n",
    "\n",
    "*Note: The first time you build this app, there will be a delay while the correct `.engine` file is built and stored.*   **This can take from one to five minutes to build.** . *To avoid timeouts in the media player during this wait, don't start the media player on your computer until the `.engine` build is complete.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the app\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst3-three-streams\n",
    "!make clean\n",
    "!make"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the app with three input streams\n",
    "%cd /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst3-three-streams/\n",
    "!./deepstream-test3-app \\\n",
    "file:///home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.h264 \\\n",
    "file:///home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.mp4 \\\n",
    "file:///home/dlinano/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_1080p_h265.mp4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How did you do?\n",
    "If you see something like this image, you did it!  If not, keep trying, or take a peek at the solution code in the solutions directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/03_three_streams.png\" alt=\"three streams\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_final\"></a>\n",
    "# 3.3 Put It All Together\n",
    "Great job adding fixing the config file to add another input stream!  Now push Jetson Nano to it's limit by running the maximum number of input streams."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"03_ex_challenge\"></a>\n",
    "## 3.3.1 Exercise: Eight Input Streams\n",
    "Create a new app based on `deepstream-test3-rstp_out` that can accept and tile eight video input streams. Fill in the following cells with appropriate commands to create, build, and run your app. To edit your files, use the JupyterLab file browser at left to navigate to the correct folder; then, double click on the file you wish to open and edit.\n",
    "\n",
    "*Notes:* \n",
    "* *The first time you build this app, there will be a delay while the correct `.engine` file is built and stored.*   **This can take up to five minutes to build.**  *To avoid timeouts in the media player during this wait, don't start the media player on your computer until the `.engine` build is complete.*\n",
    "* *Performance may be degraded with eight streams over the RSTP output.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Create a new app located at /home/dlinano/deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst3-eight-streams\n",
    "#      based on deepstream-test3-rtsp_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Modify the [PGIE config file](../deepstream_sdk_v4.0.2_jetson/sources/apps/my_apps/dst3-eight-streams/dstest3_pgie_config.txt) to process a batch size of eight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Build the app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Run the app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How did you do?\n",
    "If you see something like this image, you've mastered multiple input streams!  If not, keep trying, or take a peek at the solution code in the solutions directory.  <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/03_8streams.png\" alt=\"8 streams\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style=\"color:green;\">Congratulations!</h2>\n",
    "\n",
    "You configured a DeepStream pipeline to accept different input streams and are able to apply inference on those streams.<br>\n",
    "Move on to [4.0 Video File Output](./04_VideoFileOutput.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"images/DLI Header.png\" alt=\"Header\" width=\"400\"></center>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}