Add simple global controls to put the call in picture-in-picture mode (#2573)

* Stop sharing state observables when the view model is destroyed

By default, observables running with shareReplay will continue running forever even if there are no subscribers. We need to stop them when the view model is destroyed to avoid memory leaks and other unintuitive behavior.

* Hydrate the call view model in a less hacky way

This ensures that only a single view model is created per call, unlike the previous solution which would create extra view models in strict mode which it was unable to dispose of. The other way was invalid because React gives us no way to reliably dispose of a resource created in the render phase. This is essentially a memory leak fix.

* Add simple global controls to put the call in picture-in-picture mode

Our web and mobile apps (will) all support putting calls into a picture-in-picture mode. However, it'd be nice to have a way of doing this that's more explicit than a breakpoint, because PiP views could in theory get fairly large. Specifically, on mobile, we want a way to do this that can tell you whether the call is ongoing, and that works even without the widget API (because we support SPA calls in the Element X apps…)

To this end, I've created a simple global "controls" API on the window. Right now it only has methods for controlling the picture-in-picture state, but in theory we can expand it to also control mute states, which is current possible via the widget API only.

* Fix footer appearing in large PiP views

* Add a method for whether you can enter picture-in-picture mode

* Have the controls emit booleans directly

Author: robintown

docs/README.md

@@ -2,5 +2,6 @@
 
 This folder contains documentation for Element Call setup and usage.
 
-- [Url format and parameters](./url-params.md)
 - [Embedded vs standalone mode](./embedded-standalone.md)
+- [Url format and parameters](./url-params.md)
+- [Global JS controls](./controls.md)

docs/controls.md

@@ -0,0 +1,7 @@
+# Global JS controls
+
+A few aspects of Element Call's interface can be controlled through a global API on the `window`:
+
+- `controls.canEnterPip(): boolean` Determines whether it's possible to enter picture-in-picture mode.
+- `controls.enablePip(): void` Puts the call interface into picture-in-picture mode. Throws if not in a call.
+- `controls.disablePip(): void` Takes the call interface out of picture-in-picture mode, restoring it to its natural display mode. Throws if not in a call.

src/@types/global.d.ts

@@ -15,6 +15,7 @@ limitations under the License.
 */
 
 import "matrix-js-sdk/src/@types/global";
+import { Controls } from "../controls";
 
 declare global {
   interface Document {
@@ -23,6 +24,10 @@ declare global {
     webkitFullscreenElement: HTMLElement | null;
   }
 
+  interface Window {
+    controls: Controls;
+  }
+
   interface HTMLElement {
     // Safari only supports this prefixed, so tell the type system about it
     webkitRequestFullscreen: () => void;

src/controls.ts

@@ -0,0 +1,39 @@
+/*
+Copyright 2024 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { Subject } from "rxjs";
+
+export interface Controls {
+  canEnterPip: () => boolean;
+  enablePip: () => void;
+  disablePip: () => void;
+}
+
+export const setPipEnabled = new Subject<boolean>();
+
+window.controls = {
+  canEnterPip(): boolean {
+    return setPipEnabled.observed;
+  },
+  enablePip(): void {
+    if (!setPipEnabled.observed) throw new Error("No call is running");
+    setPipEnabled.next(true);
+  },
+  disablePip(): void {
+    if (!setPipEnabled.observed) throw new Error("No call is running");
+    setPipEnabled.next(false);
+  },
+};

src/room/InCallView.module.css

@@ -58,6 +58,10 @@ limitations under the License.
   );
 }
 
+.footer.hidden {
+  display: none;
+}
+
 .footer.overlay {
   position: absolute;
   inset-block-end: 0;
@@ -67,6 +71,7 @@ limitations under the License.
 }
 
 .footer.overlay.hidden {
+  display: grid;
   opacity: 0;
   pointer-events: none;
 }

src/room/InCallView.tsx

@@ -69,7 +69,7 @@ import { InviteButton } from "../button/InviteButton";
 import { LayoutToggle } from "./LayoutToggle";
 import { ECConnectionState } from "../livekit/useECConnectionState";
 import { useOpenIDSFU } from "../livekit/openIDSFU";
-import { GridMode, Layout, useCallViewModel } from "../state/CallViewModel";
+import { CallViewModel, GridMode, Layout } from "../state/CallViewModel";
 import { Grid, TileProps } from "../grid/Grid";
 import { useObservable } from "../state/useObservable";
 import { useInitial } from "../useInitial";
@@ -93,7 +93,7 @@ const canScreenshare = "getDisplayMedia" in (navigator.mediaDevices ?? {});
 const maxTapDurationMs = 400;
 
 export interface ActiveCallProps
-  extends Omit<InCallViewProps, "livekitRoom" | "connState"> {
+  extends Omit<InCallViewProps, "vm" | "livekitRoom" | "connState"> {
   e2eeSystem: EncryptionSystem;
 }
 
@@ -105,6 +105,8 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
     sfuConfig,
     props.e2eeSystem,
   );
+  const connStateObservable = useObservable(connState);
+  const [vm, setVm] = useState<CallViewModel | null>(null);
 
   useEffect(() => {
     return (): void => {
@@ -113,17 +115,41 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
     // eslint-disable-next-line react-hooks/exhaustive-deps
   }, []);
 
-  if (!livekitRoom) return null;
+  useEffect(() => {
+    if (livekitRoom !== undefined) {
+      const vm = new CallViewModel(
+        props.rtcSession.room,
+        livekitRoom,
+        props.e2eeSystem.kind !== E2eeType.NONE,
+        connStateObservable,
+      );
+      setVm(vm);
+      return (): void => vm.destroy();
+    }
+  }, [
+    props.rtcSession.room,
+    livekitRoom,
+    props.e2eeSystem.kind,
+    connStateObservable,
+  ]);
+
+  if (livekitRoom === undefined || vm === null) return null;
 
   return (
     <RoomContext.Provider value={livekitRoom}>
-      <InCallView {...props} livekitRoom={livekitRoom} connState={connState} />
+      <InCallView
+        {...props}
+        vm={vm}
+        livekitRoom={livekitRoom}
+        connState={connState}
+      />
     </RoomContext.Provider>
   );
 };
 
 export interface InCallViewProps {
   client: MatrixClient;
+  vm: CallViewModel;
   matrixInfo: MatrixInfo;
   rtcSession: MatrixRTCSession;
   livekitRoom: Room;
@@ -138,6 +164,7 @@ export interface InCallViewProps {
 
 export const InCallView: FC<InCallViewProps> = ({
   client,
+  vm,
   matrixInfo,
   rtcSession,
   livekitRoom,
@@ -193,12 +220,6 @@ export const InCallView: FC<InCallViewProps> = ({
   const reducedControls = boundsValid && bounds.width <= 340;
   const noControls = reducedControls && bounds.height <= 400;
 
-  const vm = useCallViewModel(
-    rtcSession.room,
-    livekitRoom,
-    matrixInfo.e2eeSystem.kind !== E2eeType.NONE,
-    connState,
-  );
   const windowMode = useObservableEagerState(vm.windowMode);
   const layout = useObservableEagerState(vm.layout);
   const gridMode = useObservableEagerState(vm.gridMode);