Skip to content
This repository was archived by the owner on Nov 11, 2024. It is now read-only.

Commit 1b0ba99

Browse files
TangZhiZhenczhou26
TangZhiZhen
authored and
czhou26
committed
Update getting started guide
Signed-off-by: Zhizhen Tang <[email protected]>
1 parent 3aa675a commit 1b0ba99

File tree

1 file changed

+46
-205
lines changed

1 file changed

+46
-205
lines changed

‎doc/CDN_Transcode_Sample_Getting_Started_Guide.md

+46-205
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,40 @@
11
# CDN Transcode Sample Getting Started Guide
22

33
* [CDN Transcode Sample Getting Started Guide](#cdn-transcode-sample-getting-started-guide)
4+
* [Architecture](#architecture)
45
* [Prerequisites](#prerequisites)
56
* [Setup CDN-Transcode Server](#setup-cdn-transcode-server)
67
* [Setup Streaming Server \(Optional\)](#setup-streaming-server-optional)
78
* [Setup Client](#setup-client)
89
* [Build](#build)
910
* [Deploy](#deploy)
10-
* [Auto deployment](#auto-deployment)
11-
* [Start CDN transcode service](#start-cdn-transcode-service)
12-
* [Playback](#playback)
13-
* [Web browser playback](#web-browser-playback)
14-
* [VLC playback](#vlc-playback)
11+
* [Auto deployment](#auto-deployment)
12+
* [Auto deployment using docker swarm](#auto-deployment-using-docker-swarm)
13+
* [Auto deployment using Kubernetes](#auto-deployment-using-Kubernetes)
1514
* [Manual deployment](#manual-deployment)
16-
* [Setup network topology](#setup-network-topology)
17-
* [Config Streaming Server](#config-streaming-server)
18-
* [Config CDN-Transcode Server](#config-cdn-transcode-server)
19-
* [Start CDN transcode service](#start-cdn-transcode-service-1)
20-
* [Start zookeeper service](#start-zookeeper-service)
21-
* [Start kafka service](#start-kafka-service)
22-
* [Start live transcode docker](#start-live-transcode-docker)
23-
* [Start VOD transcode service](#start-vod-transcode-service)
24-
* [Start nginx web service](#start-nginx-web-service)
25-
* [Start live transcode service](#start-live-transcode-service)
26-
* [Playback](#playback-1)
27-
* [Web browser playback](#web-browser-playback-1)
28-
* [VLC playback](#vlc-playback-1)
2915

3016

31-
This document describes how to run the CDN Transcode Sample step by step. Please refer to [reference architecture](CDN_Transcode_Sample_RA.md) to understand the sample reference architecture design and how the sample works.
17+
This document describes how to run the CDN Transcode Sample step by step.
18+
19+
The sample provides two kinds of services - `live streaming` and `VOD`, and this guide shows how to use the services in a simplest and typical way which can be scaled out to more complex environment. E.g.: in this guide, the docker images for transcoder server and cdn edge server are hosted on the same physical server. In real case, they can be hosted on different servers located in different places in the CDN network.
20+
21+
# Architecture
22+
The sample implements a reference server-side transcode system over CDN infrastructure, which features `live streaming` and `VOD`.
23+
24+
<IMG src="https://github.com/OpenVisualCloud/CDN-Transcode-Sample/blob/master/volume/html/image/CDN-Transcode-Sample-Arch.png" height="450">
25+
26+
27+
The workflow of live streaming is as follows:
28+
- The live transcode service receives the video stream locally or over rtmp, decapsulate and demux the flv video, transcode to other codecs/bitrate/resolution, in 1:N manner, which means one input and N output. E.g.: one channel HEVC 4k@60fps, transcoded into one channle AVC 720p@30fps, one channel HEVC 1080p@30fps and one channel HEVC 4k@60fps. The transcoded video streams are then be encapsulated in flv over rtmp, be distributed to the CDN service.
29+
- The CDN service receives the video streams from the Live transcode Service, and will translate the video streams into HLS/DASH format.
30+
- The client starts playback the video.
31+
32+
The workflow of VOD is as follows:
33+
- The client starts playback of a video by requesting the video manifest file.
34+
- The CDN service checks if the video has been transcoded. If the DASH/HLS segments exist, the CDN service simply returns the manifest (and any subsequent video segments.) Otherwise, the CDN service schedules transcoding of the video in Kafka, a message system shared between the CDN service and the VOD trancode service.
35+
- The VOD trancode service receives the request and starts transcoding of the video clip.
36+
- The CDN service returns the resulting DASH/HLS manifest and segments to the client.
3237

33-
The sample provides two kinds of services - `live streaming` and `VOD`, and this guide just shows how to use the services in a simplest and typical way which can be scaled out to more complex environment. E.g.: in this guide, the docker images for transcoder server and cdn edge server are hosted on the same physical server. In real case, they can be hosted on different servers located in different places in the CDN network.
3438

3539
# Prerequisites
3640
In this document, we'll use the simplest example to show how to build up the pipeline for different user scenarios. To simply the setup, we'll host docker nodes on the same physical server (named as "CDN-Transcode Server" in this document). One Streaming Server will be used as well to RTMP stream the source video content to CDN-Transcode Server, however you can omit Streaming Server if you want to just use local video content on CDN-Transcode Server. A client system is also needed to playback the transcoded video streams.
@@ -47,21 +51,11 @@ Below is the basic block diagram for the sample setup:
4751
**Note**: for `live streaming` and `VOD` in this document, it refers the interaction between `CDN-Transcode Server` and `Client`, not refers interaction between `Streaming Server` and `CDN-Transcode Server`.
4852
## Setup CDN-Transcode Server
4953
- Install Ubuntu 18.04/CentOS 7.6 on CDN-Transcode Server, and configure the IP address & proxy properlly.
50-
- Install [docker.ce](https://docs.docker.com/install).
51-
- Install [docker compose](https://docs.docker.com/compose/install) by referencing docker composer setup guide. Version 1.20+ is required.
52-
```
53-
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
54+
- Install the third-party dependency Libraries and tools.
5455
```
55-
- Install [kompose](https://github.com/kubernetes/kompose/blob/master/docs/installation.md) by referencing kubernetes kompose setup guide. Version 1.16+ is required.
56+
script/install_dependency.sh
5657
```
57-
sudo curl -L https://github.com/kubernetes/kompose/releases/download/v1.18.0/kompose-linux-amd64 -o /usr/local/bin/kompose
58-
```
59-
- Install python3, pip3 and ruamel.yaml
60-
61-
| Ubuntu 18.04 | CentOS 7.6 |
62-
|:--------------|:-----------|
63-
|(1) sudo apt-get install -y python3-pip |(1) sudo yum install -y python36 python36-pip |
64-
|(2) pip3 install ruamel.yaml |(2) pip3 install ruamel.yaml |
58+
**Note**: This script must be run as root.
6559
6660
- Setup docker network proxy
6761
You may need to setup the docker proxy on CDN-Transcode Server. Below is an example to directly use the host proxy as the docker proxy.
@@ -79,7 +73,7 @@ As mentioned above, Streaing Server is not a must, you can skip this section if
7973
```sh
8074
sudo apt-get install -y ffmpeg
8175
```
82-
- Install Nginx and Nginx Rtmp Module
76+
- Install Nginx and Nginx RTMP Module
8377
```
8478
git clone https://github.com/arut/nginx-rtmp-module.git
8579
@@ -105,176 +99,23 @@ make
10599
# Deploy
106100
The sample supports both auto deployment and manual deployment. The auto deployment will help users to have quick try to use this sample while the manual deployment give more flexibility for customerization.
107101
## Auto deployment
108-
The auto deploy supports both docker swarm and docker compose. You're recommended to use docker compose in this sample. The auto deploy will deploy the live streaming and VOD services automatically using a default video stream "big buck bunny" as an example. To simply the setup, the original source video content are local content but not streamed from the RTMP streaming server.
109-
110-
**Note**: If you want to use other video clips to try the auto deployment, you can simply copy the clips into <CDN-Transcode-Sample>/volume/video/archive folder, then run below command to generate video clip thumbnails. The transcode service will perform 1:N transcoding for these video clips. It's recommended to use .mp4 video clip files.
111-
```
112-
ffmpeg -i <CDN-Transcode-Sample>/volume/video/archive/<clip_name> -vf "thumbnail,scale=640:360" -frames:v 1 -y <CDN-Transcode-Sample>/volume/video/archive/<clip_name>.png
113-
```
114-
### Start CDN transcode service
115-
Run below steps on CDN-Transcode server to stop/start docker swarm service.
116-
117-
- Initialize docker swarm if you have not
118-
```bash
119-
sudo docker swarm init
120-
```
121-
- start/stop docker swarm services
122-
```bash
123-
make start_docker_swarm
124-
make stop_docker_swarm
125-
```
126-
- start/stop docker-compose service
127-
```bash
128-
make start_docker_compose
129-
make stop_docker_compose
130-
```
131-
### Playback
132-
#### Web browser playback
133-
Visit https://\<CDN-Transcode Server IP address\>/ using any web browser, you will see the playlist and then click any of the streams in the playlist to playback.
134-
#### VLC playback
135-
You can also use Windows VLC player to playback the HTTPs streams provided by the sample service. A sample [VLC playback script](client/vlc_playback.bat) is provided for this purpose. You may need to change this script to set the IP address and the VLC path in this script.
102+
The auto deploy supports both docker swarm and kubernetes. The auto deploy will deploy the live streaming and VOD services automatically using a default video stream "big buck bunny" as an example. To simply the setup, the original source video content are local content but not streamed from the RTMP streaming server.
103+
104+
**Note**: If you want to use other video clips to try the auto deployment, you can simply visit https://\<CDN-Transcode Server IP address\>/ using web browser, click the guest button at the top right-hand corner and then choose upload option. It's recommended to use .mp4 video clip files.
105+
106+
### Auto deployment using docker swarm
107+
For details, please refer to [Deploy on Xeon E5 using docker swarm](https://github.com/OpenVisualCloud/CDN-Transcode-Sample/wiki/Deploy-on-Xeon-E5-using-docker-swarm).
108+
### Auto deployment using Kubernetes
109+
1. Setup kubernetes environment
110+
For details, please refer to [Setup Kubernetes environment](https://github.com/OpenVisualCloud/CDN-Transcode-Sample/wiki/Setup-Kubernetes-environment).
111+
**Note**: If you have setup kubernetes environment, please skip this step.
112+
2. Auto deployment on your own platform
113+
* Auto deployment using kubernetes on E3
114+
For details, please refer to [Deploy on Xeon E3 with Gen GFX server using Kubernetes](https://github.com/OpenVisualCloud/CDN-Transcode-Sample/wiki/Deploy-on-Xeon-E3-with-Gen-GFX-using-Kubernetes).
115+
* Auto deployment using kubernetes on E5
116+
For details, please refer to [Deploy on Xeon E5 server using Kubernetes](https://github.com/OpenVisualCloud/CDN-Transcode-Sample/wiki/Deploy-on-Xeon-E5-server-using-Kubernetes).
117+
* Auto deployment using kubernetes on VCA2
118+
For details, please refer to [Deploy on VCA2 using Kubernetes](https://github.com/OpenVisualCloud/CDN-Transcode-Sample/wiki/Deploy-on-VCA2-using-Kubernetes).
136119

137120
## Manual deployment
138-
The manual deployment give an example to users who want to use this sample as a base to build up more complicated CDN transcode solution.
139-
140-
### Setup network topology
141-
In this guide we use a very simple topology diagram to showcase the interconnection between each nodes. Below is an example:
142-
```
143-
+-----------------+-----------------+-----------------+
144-
| Live transcode | VOD transcode 1 | VOD transcode 2 |
145-
| [Ubuntu docker] | [Ubuntu docker] | [Ubuntu docker] |
146-
| 192.168.31.32 | 192.168.31.33 | 192.168.31.34 |
147-
+-----------------+-----------------+-----------------+
148-
| Zookeeper | Kafka | Nginx |
149-
+----------------+ | [Ubuntu docker] | [Ubuntu docker] | [Ubuntu docker] | +-------------+
150-
|Streaming Server| | 192.168.31.29 | 192.168.31.30 | 192.168.31.35 | | Client |
151-
| 10.67.117.70 |----->|-----------------+-----------------+-----------------+----->|10.67.117.80 |
152-
| FFmpeg Nginx | | CDN-Transcode Server: Linux OS 10.67.116.179 | | VLC/browser |
153-
+----------------+ +-----------------------------------------------------+ +-------------+
154-
155-
```
156-
157-
### Config Streaming Server (Optional)
158-
Add below Nginx configuration section into nginx.conf (may be /etc/nginx/nginx.conf) on streaming server, then copy video clips to the path defined in the configuration and start nginx service. E.g: /var/www/clips is used in this sample.
159-
```
160-
rtmp {
161-
server {
162-
application live {
163-
live on;
164-
exec_options on;
165-
exec_pull ffmpeg -re -i http://localhost/$name.mp4 -c:v copy -an -f flv rtmp://localhost/$app/$name;
166-
}
167-
}
168-
}
169-
......
170-
http {
171-
server {
172-
location / {
173-
root /var/www/clips;
174-
autoindex on;
175-
autoindex_exact_size off;
176-
autoindex_localtime on;
177-
}
178-
}
179-
}
180-
```
181-
182-
### Config CDN-Transcode Server
183-
Run below command on CDN-Transcode Server to create docker network:
184-
```
185-
docker network create -d bridge --subnet 192.168.31.0/24 --gateway 192.168.31.1 my_bridge
186-
```
187-
188-
### Start CDN transcode service
189-
#### Start zookeeper service
190-
Run below command on CDN-Transcode Server to create zookeeper docker instance:
191-
```
192-
docker pull zookeeper
193-
docker run -d --name zookeeper-service --env ZOOKEEPER_SERVER_ID=1 --env ZOOKEEPER_CLIENT_PORT=2181 --env ZOOKEEPER_TICK_TIME=2000 --env ZOOKEEPER_MAX_CLIENT_CNXNS=20000 --env ZOOKEEPER_HEAP_OPTS="-Xmx2048m -Xms2048m" --restart=always --network=my_bridge --ip 192.168.31.29 -it zookeeper
194-
```
195-
196-
#### Start kafka service
197-
Run below command on CDN-Transcode Server to create kafka docker instance:
198-
```
199-
docker pull wurstmeister/kafka
200-
docker run -d --name kafka-service --link zookeeper-service --env KAFKA_BROKER_ID=1 --env KAFKA_ADVERTISED_HOST_NAME=kafka-service --env KAFKA_ADVERTISED_PORT=9092 --env KAFKA_ZOOKEEPER_CONNECT=zookeeper-service:2181 --env KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-service:9092 --env KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT --env KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT --env KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 --env KAFKA_DEFAULT_REPLICATION_FACTOR=1 --env KAFKA_AUTO_CREATE_TOPICS_ENABLE=true --env KAFKA_NUM_PARTITIONS=16 --env KAFKA_CREATE_TOPICS="content_provider_sched:16:1" --env KAFKA_HEAP_OPTS="-Xmx1024m -Xms1024m" --restart=always --network=my_bridge --ip 192.168.31.30 -it wurstmeister/kafka
201-
```
202-
203-
#### Start live transcode docker
204-
Run below command on CDN-Transcode Server to create live transcode docker instance:
205-
```
206-
docker run -it --network=my_bridge --ip 192.168.31.32 --name live-transcode-service -v /var/www/dash:/var/www/dash -v /var/www/hls:/var/www/hls ovc_transcode_sw /bin/bash
207-
```
208-
209-
#### Start VOD transcode service
210-
Run below commands on CDN-Transcode Server to create two vod transcode docker instances:
211-
```
212-
docker run -it --network=my_bridge --ip 192.168.31.33 --name vod-transcode-service-1 -v /var/www/dash:/var/www/dash -v /var/www/hls:/var/www/hls -v <CDN-Transcode-Sample>/volume/video/archive:/var/www/archive ovc_transcode_sw /bin/bash -c '/home/main.py'
213-
docker run -it --network=my_bridge --ip 192.168.31.34 --name vod-transcode-service-2 -v /var/www/dash:/var/www/dash -v /var/www/hls:/var/www/hls -v <CDN-Transcode-Sample>/volume/video/archive:/var/www/archive ovc_transcode_sw /bin/bash -c '/home/main.py'
214-
```
215-
216-
#### Start nginx web service
217-
Run below command on CDN-Transcode Server to create cdn docker instance:
218-
```
219-
docker run -it -p 443:8080 --network=my_bridge --ip 192.168.31.35 --name cdn-service -v /var/www/dash:/var/www/dash -v /var/www/hls:/var/www/hls -v <CDN-Transcode-Sample>/volume/video/archive:/var/www/archive -v <CDN-Transcode-Sample>/volume/html:/var/www/html ovc_cdn_service /bin/bash -c '/home/main.py&/home/self-sign.sh&&/usr/sbin/nginx'
220-
```
221-
222-
#### Start live transcode service
223-
Run below commands on live transcode docker instance to show one 1:4 channels of live transcode. It supports 1 channel of H264 decode, 2 channels of SVT-HEVC encode and 2 channels of x264 encode. **Note**: If there is a Streaming Server, you need to replace the IP address below with the actual Streaming Server IP address in your setup. If not, please replace the inputRTMP stream with local stream "/var/www/archive/bbb_sunflower_1080p_30fps_normal.mp4"
224-
```
225-
ffmpeg -i rtmp://10.67.117.70/live/bbb_sunflower_1080p_30fps_normal \
226-
-vf scale=2560:1440 -c:v libsvt_hevc -b:v 15M -f flv \
227-
rtmp://cdn-service/hls/big_buck_bunny_2560x1440 -vf scale=1920:1080 -c:v libsvt_hevc -b:v 10M -f flv \
228-
rtmp://cdn-service/hls/big_buck_bunny_1920x1080 -vf scale=1280:720 -c:v libx264 -b:v 8M -f flv \
229-
rtmp://cdn-service/hls/big_buck_bunny_1280x720 -vf scale=854:480 -c:v libx264 -b:v 6M -f flv \
230-
rtmp://cdn-service/hls/big_buck_bunny_854x480 -abr_pipeline
231-
```
232-
233-
**Note**: for live ABR (using ffmpeg to trancode in 3 variants, and nginx produce one manifest, there are only 3 different resolutions and bit rates supported in live ABR).
234-
Run below commands on live transcode docker instance, the suffix "hi" corresponds to the maximum resolution or bit rate, the suffix "mid" corresponds to the medium resolution or bit rate and the suffix "low" corresponds to the minimum resolution or bit rate.
235-
For DASH
236-
```
237-
ffmpeg -i rtmp://10.67.117.70/live/bbb_sunflower_1080p_30fps_normal -vf scale=1920:1080 -c:v libsvt_hevc -b:v 8M -f flv \
238-
rtmp://cdn-service/dash/big_buck_bunny_hi -vf scale=1280:720 -c:v libsvt_hevc -b:v 4M -f flv \
239-
rtmp://cdn-service/dash/big_buck_bunny_mid -vf scale=854:480 -c:v libsvt_hevc -b:v 2M -f flv \
240-
rtmp://cdn-service/dash/big_buck_bunny_low -abr_pipeline
241-
```
242-
For HLS
243-
```
244-
ffmpeg -i rtmp://10.67.117.70/live/bbb_sunflower_1080p_30fps_normal -vf scale=1920:1080 -c:v libsvt_hevc -b:v 8M -f flv \
245-
rtmp://cdn-service/hls/big_buck_bunny_hi -vf scale=1280:720 -c:v libsvt_hevc -b:v 4M -f flv \
246-
rtmp://cdn-service/hls/big_buck_bunny_mid -vf scale=854:480 -c:v libsvt_hevc -b:v 2M -f flv \
247-
rtmp://cdn-service/hls/big_buck_bunny_low -abr_pipeline
248-
```
249-
250-
Configure below parameters on cdn docker instance, the "max" flag which indicate which representation should have max witdh and height and so use it to create the variant manifest on DASH.
251-
```
252-
rtmp {
253-
server {
254-
......
255-
application hls {
256-
......
257-
hls_variant _low BANDWIDTH=2048000 RESOLUTION=854x480;
258-
hls_variant _mid BANDWIDTH=4096000 RESOLUTION=1280x720;
259-
hls_variant _hi BANDWIDTH=8192000 RESOLUTION=1920x1080;
260-
}
261-
application dash {
262-
......
263-
dash_variant _low bandwidth="2048000" width="854" height="480";
264-
dash_variant _med bandwidth="4096000" width="1280" height="720";
265-
dash_variant _hi bandwidth="8192000" width="1920" height="1080" max;
266-
}
267-
}
268-
}
269-
```
270-
271-
### Playback
272-
#### Web browser playback
273-
Visit https://\<CDN-Transcode Server IP address\>/ using any web browser, you will see the playlist and then click any of the streams in the playlist to playback.
274-
275-
#### VLC playback
276-
Same as auto deployment, you can use VLC client as well following instructions in the above section.
277-
**Note**: for adaptative streaming, please run below commands.
278-
```
279-
vlc https://<CDN-Transcode Server IP address>/dash/big_buck_bunny.mpd
280-
```
121+
The manual deployment give an example to users who want to use this sample as a base to build up more complicated CDN transcode solution. For details, please refer to [Manual Deploy](https://github.com/OpenVisualCloud/CDN-Transcode-Sample/wiki/Manual-Deploy).

0 commit comments

Comments
 (0)