Merge pull request #49 from membraneframework/rtmp_to_hls

Broadcasting tutorial

Author: blazpie

broadcasting/00_General_Introduction.md

@@ -0,0 +1,16 @@
+We are glad you have decided to continue your journey into the oceans of multimedia with us!
+In this tutorial, you will be able to see the Membrane in action - as we will prepare a pipeline responsible for converting
+the incoming RTMP stream into an HLS stream, ready to be easily served on the internet.
+Later on, we will substitute the front part of our pipeline, so that to make it compatible with another popular streaming protocol - RTSP.
+That is how we will try to prove the great dose of flexibility that comes with the use of the Membrane Framework!
+
+## General tutorial's structure
+As mentioned earlier, the tutorial consists of two parts:
+* "RTMP to HLS" - this part describes how to create a pipeline capable of receiving RTMP stream, muxing it into CMAF files, and serving them with the use of HTTP
+* "RTSP to HLS" - this part is based on the previous one and describes how to change the pipeline from the previous part so that to make it capable of handling the RTSP stream instead of the RTMP stream.
+
+As a result of each part, you will have a web application capable of playing media streamed with the use of the appropriate protocols - RTMP and RTSP.
+
+We strongly encourage you to follow the tutorial part by part, as the different chapters are tightly coupled. However, in case you wanted just to see the result solutions or even one of them, we invite you to take a look at the appropriate directories of the `membrane_demo` repository:
+* [RTMP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls)
+* [RTSP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtsp_to_hls)

broadcasting/01_RTMP_Introduction.md

@@ -0,0 +1,28 @@
+## What do we want to achieve?
+
+Imagine that we need to build a video streaming platform. That means - we want a particular system user, the streamer, to stream its multimedia on a server, where it will be available for other participants - the viewers.
+We need to take advantage of the fact, that there will be only one person streaming, and take into consideration, that there might be multiple, possibly many, viewers. What's the best solution for such a use case?
+Well, as the tutorial name suggests - we can use RTMP for the streamer to stream its multimedia to the server, and make them accessible for the viewers by broadcasting them via HLS protocol!
+Although it might sound quite complicated, with the Membrane Framework this will be easier than you can expect!
+As a final product of this tutorial, we want to have a completely working solution for broadcasting a stream. It will consist of two parts:
+
+- The pipeline, responsible for receiving an RTMP stream and preparing an HLS stream out of it
+- The web player, capable of playing the HLS stream
+
+The Membrane Framework will be advantageous in the first part where we will create a Membrane's Pipeline for converting the stream.
+When it comes to the web player, we will use an existing solution - the [HLS.js](https://github.com/video-dev/hls.js/) player.
+
+## Why one would need such a solution?
+
+You might wonder why one would need to convert an RTMP stream into an HLS stream at all - couldn't we simply make the streamer broadcast its multimedia with the RTMP to all the viewers?
+Technically speaking we could...but surprisingly it wouldn't be the easiest solution, since each of the viewers would need to act as an RTMP server. And definitely, it wouldn't be a solution that would scale - RTMP is based on TCP which implies, that there is no broadcast mechanism. It would be the streamer who would need to keep the direct connection to each of the viewers.
+In contrast, the solution described above has plenty of advantages - the streamer needs to create a single connection with the RTMP server, and then the multimedia can be shared with the use of a regular HTTP server which is designed to serve multiple clients.
+
+## A brief description of the technology we will use
+
+As stated previously, we will use the preexisting protocols. You might find reading about them beneficial, and that is why we provide you with some links to a brief description of the technology we will be using:
+
+- [How does RTMP work?](https://blog.stackpath.com/rtmp/) - if you are interested in how the connection in RTMP is established, take your time and read this short description!
+- [What is RTMP and why should we care about it?](https://www.wowza.com/blog/rtmp-streaming-real-time-messaging-protocol) - here you can find another short description of RTMP, with the focus laid on the history of the protocol and comparison with other available protocols. Whatsmore, there is a comprehensive explanation of why we need to transcode RTMP to some HTTP-based protocol, just like HLS - which is the use case in our tutorial.
+- [HLS behind the scenes](https://www.toptal.com/apple/introduction-to-http-live-streaming-hls) - dig into the ideas which stand behind the HLS, one of the most common HTTP-based streaming protocols.
+- [Have you heard about CMAF?](https://www.wowza.com/blog/what-is-cmaf) - if you haven't make sure to read what is the purpose of having a `Common Media Application Format`!

broadcasting/02_RTMP_SystemArchitecture.md

@@ -0,0 +1,113 @@
+As noted previously, our system will be divided into two parts:
+
+- the server, which is responsible for receiving the RTMP stream, converting it into HLS, and then publishing the created files as an HTTP server
+- the client, responsible for playing the incoming HLS stream.
+
+Considering our point of view, the first part will be more complex, since we will need to prepare the processing pipeline on our own. Let's start with that part.
+
+## The server
+
+Below you can find the desired flow of data in our application server:
+![Pipeline scheme](assets/RTMP_to_HLS_pipeline.drawio.png)
+
+As you can see, the server architecture consists of two bigger processing units - in Membrane Framework we refer to such units as 'bins'.
+The following bins will be used in our solution:
+
+- RTMP Source Bin - responsible for receiving RTMP stream
+- HLS Sink Bin - responsible for "packing" the data into the container format
+
+Each of these bins consists of some subunits (in Membrane we refer to them as _elements_), responsible for completing an atomic multimedia processing task - i.e. parsing or payloading the incoming stream.
+
+Let's take a quick walk through the whole processing line and describe more specifically what its given parts are meant to do.
+
+### RTMP Source
+
+The very first element, just at the beginning of the RTMP Source Bin is the RTMP Source. It acts as an RTMP server, listening for the incoming stream. It will be exposing a TCP port, and the streamer will be able to connect and start sending RTMP packets through that channel.
+The incoming packets will be demuxed - meaning, that the packets will be unpacked and split, based on the track (video or audio) which data they are transporting.
+
+## Parsers
+
+Buffers containing the given track data will be sent to the appropriate parser - H264 parser for video data and AAC parser for audio data.
+
+### H264 Parser
+
+H264 parser is quite a complex element, designed to read the incoming H264 stream. We have prepared a [separate, supplemental chapter of this tutorial](H264_codec.md) to describe the H264 codec and our parser's implementation - we invite you to read it, however, that knowledge is not necessary to successfully run the application.
+
+### AAC Parser
+
+At the same time, parsing happens to the buffers containing audio data. Since the audio track has been encoded with the use of AAC, we need [AAC parser](https://github.com/membraneframework/membrane_aac_plugin) to decode it.
+For a better understanding of why we need to encode audio data, as well as how it is done, feel free to visit [a part of our Multimedia Introduction Tutorial](<>), divagating on audio processing.
+That part of the tutorial does not describe the AAC codec itself, and in case you are interested in digging into that codec, we highly recommend visiting [that page](https://wiki.multimedia.cx/index.php/Understanding_AAC).
+
+### HLS converter
+
+Once data reaches the HLS bin, it needs to be put into the appropriate container files. Since we will be using Common Media Application Format (CMAF) to distribute our media, we need to put all the tracks into the `fragmented MP4` container. The first step is to payload the track's data. Payloading transforms the media encoded with a given codec into a form that is suitable to be put into the container.
+The audio track is payloaded within the `Membrane.MP4.Payloader.AAC` module and the video track is payloaded with an H264 payloader, implemented by the `Membrane.MP4.Payloader.H264` module.
+The payloaded streams (both the audio stream and the video stream) are then put in the result CMAF file - and the `Membrane.MP4.Muxer.CMAF` is the element responsible for that process. The name 'muxer', relates to the process
+of 'muxing' - putting multiple tracks in one file, which is done by that element. Furthermore, the `CMAF muxer`, splits the streams into so-called segments - stream parts of the desired duration. Along that procedure, the manifest files, which contain metadata about the media and names of the appropriate segment file names, are generated.
+Finally, the output files: '.m4s' files for each of the segments, as well as manifest files, are written on the disk.
+Let's take a look at how do the generated files look like:
+![Pipeline scheme](assets/output_files_structure.png)
+As we can see, there are:
+
+- index.m3u8 - the manifest file which contains metadata about the media, as well as the URI of the manifest files for both the video and audio track.
+
+```
+#EXTM3U
+#EXT-X-VERSION:7
+#EXT-X-INDEPENDENT-SEGMENTS
+#EXT-X-MEDIA:TYPE=AUDIO,NAME="audio_default_name",GROUP-ID="audio_default_id",AUTOSELECT=YES,DEFAULT=YES,URI="audio.m3u8"
+#EXT-X-STREAM-INF:BANDWIDTH=4507734,CODECS="avc1.42e00a",AUDIO="audio_default_id"
+video_g2QABXZpZGVv.m3u8
+```
+
+- audio.m3u8 - the manifest file for the audio track.
+
+```
+#EXTM3U
+#EXT-X-VERSION:7
+#EXT-X-TARGETDURATION:8
+#EXT-X-MEDIA-SEQUENCE:0
+#EXT-X-DISCONTINUITY-SEQUENCE:0
+#EXT-X-MAP:URI="audio_header_g2QABWF1ZGlv_part0_.mp4"
+#EXTINF:8.0,
+audio_segment_9_g2QABWF1ZGlv.m4s
+#EXTINF:8.0,
+audio_segment_10_g2QABWF1ZGlv.m4s
+#EXTINF:8.0,
+audio_segment_11_g2QABWF1ZGlv.m4s
+```
+
+That file contains some metadata (like the desired duration of the segment), along with the URI pointing to the '.mp4' header file of the fMP4 container, and the list of the segment files, in '.m4s' format.
+Each segment is described with `EXTINF` directive, indicating the duration of the segment, in seconds.
+
+- video\_<identifier>.m3u8 - the manifest file for the video track. Its structure is similar to the structure of the audio.m3u8 file. However, it is worth noting, that the desired duration of each segment is equal to 10 seconds - and the '.m4s' files are holding a video stream of that length.
+
+```
+#EXTM3U
+#EXT-X-VERSION:7
+#EXT-X-TARGETDURATION:10
+#EXT-X-MEDIA-SEQUENCE:0
+#EXT-X-DISCONTINUITY-SEQUENCE:0
+#EXT-X-MAP:URI="video_header_g2QABXZpZGVv_part0_.mp4"
+#EXTINF:10.0,
+video_segment_8_g2QABXZpZGVv.m4s
+#EXTINF:10.0,
+video_segment_9_g2QABXZpZGVv.m4s
+```
+
+- audio_header\_<identifier>_part\<part_ordering_number>_.mp4 - a binary file, the header meant to describe the audio track in the format required by the fragmented MP4 container.
+- video_header\_<identifier>\_part\<part_ordering_number>.mp4 - a binary file, the header meant to describe the video track in the format required by the fragmented MP4 container.
+- audio_segment\_\<segment_ordering_number>\_\<>.m4s - particular segments containing fragments of the audio stream.
+- video_segment\_\<segment_ordering_number>\_\<>.m4s - particular segments containing fragments of the video stream.
+
+### HTTP server
+
+The HTTP server sends the requested files to the clients. Its implementation is based on the API provided by the Phoenix Framework.
+
+## The client
+
+When it comes to our application client, we will be using the `Hls.js` library, available in the javascript ecosystem.
+We won't dig into the details of how the client application media player is implemented, however, we would like to point out some steps which take place while playing the media.
+First, the client needs to request the master manifest file. Based on that file, the client asks for the appropriate audio and video track manifest files.
+With these files, the client knows the MP4 header files for both the tracks, as well as knows the filenames of the particular segments, along with their duration. The client downloads the MP4 header file and starts downloading the appropriate segment files, based on which part of the media should be played in the nearest future.

broadcasting/03_RTMP_RunningTheDemo.md

@@ -0,0 +1,46 @@
+This time we won't create the application from the scratch nor will we be based on some kind of template.
+Since most of the application's code is media-agnostic, we don't see a point in focusing on it. Instead, we will indicate the most crucial parts of the
+[RTMP to HLS demo](https://github.com/membraneframework/membrane_demo/rtmp_to_hls), one of many publicly available [Membrane demos](https://github.com/membraneframework/membrane_demo/).
+The Membrane demos are supplementing the Membrane Tutorials on providing a source of knowledge in the field of multimedia processing - we warmly invite you to look at them as you may find some interesting
+use cases of Membrane in real-life scenarios.
+
+## Running the application
+
+In order to run the demo, you need to clone the Membrane demos repository first:
+
+```
+git clone https://github.com/membraneframework/membrane_demo
+cd membrane_demo/rtmp_to_hls
+```
+
+Once in the project directory, you need to get the dependencies of the project:
+
+```
+mix deps.get
+```
+
+Finally, you can run the application with the following command:
+
+```
+mix phx.server
+```
+
+The server will be waiting for an RTMP stream on localhost:9009, and the client of the application will be available on localhost:4000.
+
+## Exemplary stream generation with OBS
+
+You can send RTMP stream onto `localhost:9009` with your favorite streaming tool. As an example, we will show you how to generate an RTMP stream with
+[OBS](https://obsproject.com).
+Once you have OBS installed, you can perform the following steps:
+
+1. Open the OBS application
+1. Open the `Settings` windows
+1. Go to the `Stream` tab and set the value in the `Server` field to: `rtmp://localhost:9009` (the address where the server is waiting for the stream)
+1. Finally, you can go back to the main window and start streaming with the `Start Streaming` button.
+
+Below you can see how to set the appropriate settings (steps 2) and 3) from the list of steps above):
+![OBS settings](doc_assets/OBS_settings.webp)
+
+Once the streaming has started, you can visit `localhost:4000`, where the client application should be available. After a few seconds, you should be able to play
+the video you are streaming. Most likely, a kind of latency will occur and the stream played by the client application will be delayed. That latency can reach values as great as 10 seconds, which is the result
+of a relatively complex processing taking place in the pipeline. Note, that the latency here is not a result of the packets traveling far via the network, as the packets do not leave our computer in that setup - in the case of a real streaming server, we would need to take into consideration the time it takes a packet to reach the destination.