Proof-of-concept: Web based Jamulus client

[dtinth]

Video

First, here’s a demo:

webjam.mp4

Quick facts

• Web Audio API can provide low latency audio access to audio output with base latency of 5.33 ms (buffer size of 128 samples) under latencyHint: 'interactive' mode.
• WebRTC can create a low-latency UDP-like data channel under ordered: false, maxRetransmits: 0 settings. We also get end-to-end encryption for free.
• libopus can be compiled into WebAssembly and used inside an AudioWorklet.
• As of writing, all of the above is supported in the latest version of Safari on iOS.

How it works

There are 3 main components:

1. Jamulus server.

2. WebRTC-UDP gateway.

   • Source code: https://github.com/dtinth/rtcjam
   • It should be run on the same server as the Jamulus server. It does nothing more than providing browser access to Jamulus server over a different transport layer.
   • Since it will run on a publicly-accessible server, there will be no need to use STUN/TURN solutions.
   • Once a WebRTC connection is established, all data will be forwarded to localhost:22124 (and vice versa).

3. Web-based Jamulus client.

   • Source code: https://github.com/dtinth/webjam
   • This web-based client speaks the usual Jamulus protocol. However the transport layer is WebRTC instead of plain UDP.
   • An AudioWorklet uses libopus compiled to WebAssembly to decode Opus frames and feeds it to the speaker.

Limitations of the POC

• Right now it can only receive audio, as the encoder has not been implemented yet. The PoC is hardcoded to send silent Opus frames to the server.
• The PoC is hardcoded to use Stereo High Quality mode with 16 frames of jitter buffer. Client name is hardcoded to WebJam.
• Memory and CPU usage is very inefficient — a lot of clones and byte-by-byte operations in JavaScript. Safari on my iPad Pro (2015) crashes (page is forced to reload) after 1 minute of usage 😂. But I believe this can be improved by putting more things in WebAssembly, e.g. Jamulus protocol handler, jitter buffer, etc. There is a tutorial on using WebAssembly with AudioWorklet on Google Developers
• The gateway server currently has no security mechanism at all; it accepts and forwards incoming UDP packets from any host (not just the target server).

Note: I have no intention to develop this any further. All this is just to satisfy my curiosity. However, I think it can be a good starting point to expand Jamulus to the web platform if anyone wants to pursue this goal.

[mcfnord]

My friend Marsha wants to stream a Jamulus session live into AltspaceVR.

AltspaceVR can stream audio from an Edge browser, using an extension called Web Projector.
https://docs.microsoft.com/en-us/windows/mixed-reality/altspace-vr/tutorials/web-projector-streaming

Maybe this could stream the whole jam, but maybe I couldn't set each level like a normal jamulus client. But it appears to require Safari. So close!

[dtinth]

@mcfnord The web client also works on Chrome, and possibly Edge. I said Safari above because for a lot of web APIs, Safari’s support lags behind other browsers.

[WolfganP]

Very interesting pilot, thx @dtinth for sharing it!

[WolfganP]

Interesting announcement from Cloudflare https://blog.cloudflare.com/announcing-our-real-time-communications-platform/ on accelerated WebRTC paths.

[ann0see]

That’s great news! Thanks for your great work! If this moves forward I could imagine it getting a game changer (broadcasting, listener clients, …) maybe you can open a PR and link to it in the tools folder on the repo? I think it should gain more visibility.