Audio plugin for NoVNC

Description

NoVNC drop-in plugin for out-of-band audio playback.

Tested with NoVNC 1.4.0

Features

• Low-latency real-time audio playback using WebSocket and Media Source API.
• Works with both PipeWire and PulseAudio.
• No WebRTC shenanigans necessary.

Installation

NoVNC

• Copy audio-plugin.js to the NoVNC client directory where vnc.html is located.

• Add the following line to the end of the head section of vnc.html.

  <script type="module" crossorigin="anonymous" src="audio-plugin.js"></script>

vnc_lite.html is not supported yet.

Host

VNC protocol in general only handles graphics and not audio. Audio must be transmitted to out-of-band using a separate connection.

PulseAudio/PipeWire

A few modifications to the configuration of PulseAudio/PipeWire is necessary to allow capturing audio and feeding it to the Audio Proxy.

• If using PulseAudio:

  Enable module-simple-protocol-tcp module on PulseAudio:

  # echo "load-module module-simple-protocol-tcp listen=127.0.0.1 format=s16le channels=2 rate=48000 record=true playback=false" > /etc/pulse/default.pa.d/simple-protocol.pa
  $ pulseaudio -k
  $ pulseaudio --start

• If using PipeWire:

  Enable libpipewire-module-protocol-simple module on PipeWire:

  # cat > /etc/pipewire/pipewire.conf.d/simple-protocol.conf << EOF
  context.modules = [
      {
          name = libpipewire-module-protocol-simple
          args = {
              capture = true
              playback = false
  
              stream.capture.sink = true
  
              audio.rate = 48000
              audio.format = S16LE
              audio.channels = 2
  
              server.address = [
                  "tcp:127.0.0.1:4711"
              ]
          }
      }
  ]
  EOF
  $ systemctl --user restart pipewire
  

Audio Proxy

Raw audio output from PulseAudio must be encoded to a codec supported by Media Source (e.g. WebM/Opus) before it can be played back on the browser.

audio-proxy.sh shell script can encode raw audio to the required format using GStreamer.

$ audio-proxy.sh -l 5711
Raw source port: 4711
Raw source format: s16le
Raw source sample rate: 48000
Raw source channels: 2
Server listening on 127.0.0.1:5711

audio-proxy.sh requires socat and gstreamer-tools (along with Base, Good and Bad plugins) to be installed on your system.

• Installing dependencies on Ubuntu/Debian:

$ apt install socat gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad

• Installing dependencies on Alpine Linux:

$ apk add socat gstreamer-tools gst-plugins-base gst-plugins-good gst-plugins-bad

Websockify

Just like VNC, Audio Proxy only accepts raw TCP connections. Websockify will take care of proxying WebSocket traffic to and from audio-proxy.sh.

It's possible to proxy both VNC and Audio and serve the NoVNC client files using a single Websockify instance.

Assuming

• there's a VNC server running on port 5900
• Audio Proxy running on port 5711
• noVNC client files are located at /var/www/noVNC

Websockify can be configured like this:

# cat > /etc/websockify/token.cfg << EOF
	vnc: 127.0.0.1:5900
	audio: 127.0.0.1:5711
EOF
$ websockify \
    --web=/var/www/noVNC \
    --token-plugin=TokenFile \
    --token-source=/etc/websockify/token.cfg \
    8080

• NoVNC client will now be accessible at http://localhost:8080
• VNC WebSocket path will be websockify?token=vnc
• Audio WebSocket path will be websockify?token=audio

Known Issues

• Audio may not start playing if autoconnect is enabled.

TODO

• Add support for Windows
• Add builtin WebSocket server to Audio Proxy

Disclaimer

This is an experimental and hacky piece of software, so expect a few bugs here and there.

Make sure to use TLS with proper HTTP authentication if this software is being exposed to the internet.