player: sustain 1080p60 playback on Android

[mcuadros]

Summary

This change removes two independent bottlenecks from local playback:

• FFmpeg now scales directly into Avebi's pooled, tightly packed RGBA buffers instead of scaling into an intermediate native frame and copying every row into Go memory.
• Oto audio read-ahead no longer holds the playback-state mutex while FFmpeg decodes the next frame. Native decoder calls remain strictly serialized.

There are no public API changes. Stream playback, media selection, playback controls, audio conversion, and hardware-decoder selection are outside this diff.

Root cause

The previous video path called Scaler.Scale, wrapped the scaler-owned frame, and copied its contents into a pooled Go slice. A 1920x1080 RGBA frame is 8,294,400 bytes; at 60 FPS that extra pass moved approximately 475 MiB/s before ebiten.Image.WritePixels uploaded the frame.

For media with audio, Ebitengine/Oto pulls PCM through ffiLocalController.Read. That callback previously held the controller mutex across decoder.ReadFrame. A native decode can consume most of a frame interval, so Ebitengine's update thread could not inspect the video queue or advance presentation while audio read-ahead was decoding. On the physical Android test this limited the example to approximately 23 TPS / 27 FPS even though MediaCodec and swscale were individually fast enough.

Implementation

1. Scale into the pooled output buffer

ffiDecoder.convertVideoFrameLocked now:

1. obtains an exact width * height * 4 slice from backendVideoBufferPool;
2. wraps that slice in a reusable ffmpeg.Frame using Frame.WrapBuffer;
3. asks Scaler.ScaleTo to write RGBA directly into it; and
4. transfers the slice to the returned backendFrame without another copy.

The output remains tightly packed RGBA, so the contract consumed by ebiten.Image.WritePixels is unchanged.

Buffer ownership and native lifetime

• backendFrame.Video.RGBA owns the pooled slice after ScaleTo returns.
• The controller returns the slice to backendVideoBufferPool only when the queued or displayed frame is replaced, discarded, or closed.
• ffiDecoder.scaleTarget retains FFmpeg's reference to the most recently wrapped slice. WrapBuffer unreferences the previous target before installing the next one.
• A ScaleTo failure frees the native reference before returning the slice to the pool.
• ffiDecoder.Close frees the final wrapped reference.

The native integration test compares ffmpeg.WrappedBufferMemoryUsage before decode, during decode, and after Close: exactly one wrapped video buffer may remain pinned while the decoder is open, and the accounting must return to its original baseline after close.

2. Separate playback state from native decoder serialization

ffiLocalController now gives each lock one responsibility:

Lock	Responsibility
mutex	Playback state, queues, audio-player state, position, and generation.
readMutex	Serializes calls to the controller's io.Reader implementation.
decoderMutex	Serializes every native ReadFrame, Seek, and Close call.

When the audio reader needs another decoded frame, it records decodeGeneration, releases the playback mutex, performs the native read under decoderMutex, and reacquires the playback mutex before touching queues or state. This keeps CurrentVideoFrame and the Ebitengine update loop responsive without allowing concurrent native decoder access.

decodeGeneration increments whenever playback is reset or closed. If a read began before a seek, stop, replay, loop reset, or close and completes afterwards, its frame belongs to the old generation: it is discarded, its RGBA buffer is recycled when appropriate, and it cannot enter the new playback state.

Close marks the controller closed and invalidates the generation before waiting for the serialized decoder close. A read that was already in flight therefore returns EOF after the native call and cannot repopulate a cleared pool.

Review guide

The commits are intentionally separated by concern:

1. backend: scale video into pooled buffers
   • direct swscale output, error cleanup, final wrapped-frame cleanup, and native pin accounting;
2. player: keep decoding outside playback lock
   • lock separation, decoder serialization, and generation invalidation;
3. test: cover concurrent audio decoding
   • deterministic blocked-read tests for responsiveness, seek, and close.

The most important invariants to verify are:

• the scaler never writes to a slice concurrently owned by a displayed or queued frame;
• every returned video buffer is either retained by the controller or recycled exactly once;
• ReadFrame, Seek, and Close never overlap on the native decoder;
• no frame decoded before a lifecycle boundary is committed after that boundary; and
• closing a controller cannot leave a pooled RGBA buffer or wrapped Go buffer pinned.

Tests

The new deterministic tests use a decoder whose ReadFrame blocks until released:

• TestFFmpegAudioDecodeDoesNotBlockVideoPlayback proves CurrentVideoFrame remains available while audio decoding is in flight.
• TestFFmpegAudioReadDiscardsFrameDecodedBeforeSeek starts a seek during the blocked read and verifies the stale frame is rejected and recycled.
• TestFFmpegAudioReadDoesNotRetainFrameDecodedBeforeClose closes during the blocked read and verifies the stale frame is rejected without repopulating the cleared pool.
• TestFFmpegBackendMedia verifies the wrapped-buffer pin count during real decoding and after decoder close.

Validation performed:

• go test -race -count=1 ./...
• 20 repeated race-detector runs of the audio/video, seek, and close concurrency tests
• go vet ./...
• strict checkptr and GOEXPERIMENT=cgocheck2 native integration tests
• real-media suites and 200-cycle go-ebiten-mcp torture runs against FFmpeg 6, 7, 8, and 9
• final go test ./... after updating the branch against current origin/main

Physical Android result

Tested with a clean APK on a physical Android 14 arm64 tablet using a 20-second 1920x1080, 60 FPS H.264/AAC file:

• before: approximately 23 TPS / 27 FPS;
• after: stable 60 TPS / 60 FPS;
• MediaCodec H.264 decoding remained active;
• the stereo 48 kHz AAC test tone was audible, stable, and uninterrupted through the tablet speakers;
• document selection, play, pause, seek, stop, and loop were exercised;
• native heap remained approximately 70 MiB during sustained looping.

This is evidence for the tested device and media, not a general performance guarantee for every codec, resolution, FFmpeg build, or Android device.