Integrate streaming as a Skyvern Service only in LINUX or WSL platforms

[joseluisll]

Goal

The goal of this PR is to activate the Streaming functionality without changing the behavior or the structure of the original code.

This means that he following known limitations are:

• Compatible only with linux or docker-environments including WSL.
• The original code only manages 1 streaming session. All persistent browser sessions will share the same streaming session. This means that only 1 workflow_run with a persistent browser session should be run at the same time.
• The streaming infraestructure is created only when 1 workflow_run is created with a persistent browser session requirement. When no PBS Workflow_runs have been created, and you go to the PBS list and activate a PBS, the screen is black. If you activate a WR, then all PBS screens will show the live WR screen.

Observations

This PR solves issue #4296. The streaming service could not startup due to broken app context and missing packages into the docker image. Also, the entrypoint did not start the streaming servicies.

Here is the trace that appeared in the log:

[__main__] task_id=None workflow_run_id=wr_494140052738338584 organization_id=o_494139673975597314 error_type=builtins.RuntimeError error_category=ERROR exception_hash=e46b51b703a7fb04

Traceback (most recent call last):

  File "/app/run_streaming.py", line 35, in run

    workflow_run = await app.DATABASE.get_workflow_run(workflow_run_id=workflow_run_id)

                         ^^^^^^^^^^^^

  File "/app/skyvern/forge/__init__.py", line 20, in __getattr__

    raise RuntimeError("ForgeApp is not initialized. Call start_forge_app() before accessing app properties.")

RuntimeError: ForgeApp is not initialized. Call start_forge_app() before accessing app properties.

Changes

• Integrate the run_streaming.py script as an internal forge_app service called StreamingService.
• Modify Dockerfile to add needed packages: x11vnc, websockify
• Modify Entrypoint-skyvern.sh: stop invoking run_streaming.py and call the "scripts/start_streaming.sh"

Future work

I would like to make future changes to improve the integration of the StreamingService into the App.

• Integrate the start_streaming.sh script inside the StreamingService.
• Eliminate the state_file.json . The StreamingService shall read the organizations, workflow_runs and persistentbrowsersessions from the existing App Services.
• Other thoughts, to be explored.

[joseluisll]

Screenshot of the applied fixed. The streaming now works!.

@wintonzheng can you please review this minimal fix?.

Thank you.

[joseluisll]

I have solved also why the live streaming was not working:

• the Dockerfile for the backend does not have the x11vnc and websockify packages installed.
• the entrypoint for the backend does not call the script start_vnc_streaming.sh script, that starts XFVB, VNC and WEBSOCKIFY.

All in all, I check that the tests pass, and also that both the screencapture and the live streaming works using the improved docker image.

[AronPerez]

Looks really good, like this a lot. Have some questions

[joseluisll]

Up to now, I have integrated the streaming service inside the forge_app, and used the start_event to start the service loop. It works, but with the limitations of the screencapture for 1 xfvb display 99. This is the same level of functionality, but now the run_streaming.py is obsolete and not used at all.

[wintonzheng]

we should have a fix for the forge app start call in streaming script: #4703

it looks like this PR is also introducing another functionality here to auto stream screenshot. can we make an update for the title and description of this pr?

[suchintan]

@joseluisll nice work

[joseluisll]

we should have a fix for the forge app start call in streaming script: #4703

it looks like this PR is also introducing another functionality here to auto stream screenshot. can we make an update for the title and description of this pr?

The snapshot functionality was already there in the run_streaming.py, the screencapture for collecting images to the workflow steps. I just integrated it into the StreamingService.

[joseluisll]

@suchintan Please review the PR if possible, to approve?

[AronPerez]

subprocess.run is synchronous. We'd block the entire event loop for the duration of each screenshot capture. It's basically running every second.

Maybe wrap this in asyncio.create_subprocess_shell() or asyncio.to_thread()

[AronPerez]

Xvfb is started twice with different color depths. scripts/start_vnc_streaming.sh starts in 24-bit color, and we start in 16-bit here. Any reason we need both, or can't just rely on ./scripts/start_vnc_streaming.sh directly?

[AronPerez]

In run_streaming.py we have an await initialize_skyvern_state_file(task_id=None, workflow_run_id=None, organization_id=None) check to ensure the state file exists at startup. Do we need that here too?

[joseluisll]

The point of this PR is incorporate run_streaming.py as a STREAMING_SERVICE. The run_streaming.py was an external script that created an app object. I make this proposal to integrate it as an internal service, following the same pattern of the rest of services, so it is integrated in the forge_app like the rest of services.

This PR makes the run_streaming.py obsolete.

[AronPerez]

🔥