Skip to content

Latest commit

 

History

History
3505 lines (3191 loc) · 226 KB

ARCHITECTURE.md

File metadata and controls

3505 lines (3191 loc) · 226 KB

Architecture

Modules

:app    UI (Compose), view model, dependency wiring          Android application
:media  FFmpeg runtime, media probing, file staging, fonts   Android library
:core   models, command generation, conversion workflow      pure Kotlin (JVM)

Dependencies point one way only: :app -> :media -> :core.

:core knows nothing about Android. It describes what it needs from the platform as small interfaces (core/engine/Ports.kt) and :media implements them:

Port (:core) Implementation (:media) Purpose
FfmpegExecutor FfmpegKitExecutor run FFmpeg, stream its log, cancel
SourceStager CacheSourceStager copy content:// sources to local files
MediaProbe AndroidMediaProbe duration, size, tracks, display name
FontProvider SystemFontProvider font file for text overlays
OutputDirectoryProvider AppOutputDirectory where results are written
LoudnessMeter FfmpegLoudnessMeter source loudness, for the preview
TextSourceReader ContentTextReader read subtitle files
ScratchStorage CacheJanitor camera recordings, stills, cache clean-up
StillMaker AndroidStillMaker pictures and freeze frames as PNG stills

There are no singletons. Every collaborator is a class with constructor dependencies, created once in app/.../di/AppContainer.kt, which is owned by FfmpegGuiApp and handed to the view model through ConversionViewModel.Factory.

Data flow

ConversionSettings            editor state: content URIs, timecode strings   (:app)
      |  conversion/RequestMapping.kt
      v
ConversionRequest             what to do, no file locations                  (:core)
      |
      +--> + MediaPaths.aliasesFor(request)  -> CommandBuilder -> preview text
      |
      +--> ConversionJob(request, sources, mode) -> ConversionEngine.run()
               1. stage sources into a private cache directory
               2. re-time the subtitles to the edit; write the concat list
                  (only for stream copy of several clips)
               3. stabilization analysis pass (when requested)
               4. CommandBuilder.build(request, real paths)
                  or ManualCommandResolver for a hand-edited command
               5. FfmpegExecutor.execute(), progress parsed from the log
               6. numbered outputs (parts, frames): packed into one zip
               7. always: delete staged files; on failure/cancel: delete partial output

An output of several files (a video split into parts, a sequence of frames, a hand-written command writing output_%03d.png) is written to a folder of its own (OutputPlanner), then packed into a zip named like it (core/output/PartsArchive.kt; the hand-written one only takes the files of its own counter): a folder cannot be shared or saved, a zip can. Each part is deleted once it is in the zip, so the storage needed stays about the size of the parts; the zip uses the fastest level (media files are already compressed). Packing is a stage of its own (ConversionStage.PACKING, shown on the processing screen and in the notification) and stops when the export is cancelled, leaving no zip behind; a packing that fails (storage full) is FailureReason.PACKING_FAILED, and its files are removed as for any failed export. The share sheet gets the file's real MIME type (application/zip), so the apps that save files offer themselves.

The preview and the executed command come from the same CommandBuilder.build call with different MediaPaths, so they cannot diverge. The aliases shown in the preview (input.mp4, overlay_0.png, output.mkv, ...) are exactly what ManualCommandResolver replaces when the user edits the command by hand.

Timeline

The footage is a sequence of clips, as in a video editor (core/timeline/Timeline.kt, immutable and unit tested):

  • a Clip is a range of one source file (sourceId, startMs, endMs); an EditorSource (app/.../EditorModels.kt) is a file the clips can use;
  • opening a file creates one clip over all of it; splitAt, remove, duplicate, move, trim, append (another file) and insertAt reshape the sequence;
  • "+" picks one file or several at once (OpenMultipleDocuments; ConversionViewModel.addClips): they are inspected a few at a time (probeAll), pictures are made into stills one after the other, and the clips are appended in the order picked, as one undo step. Sounds (addAudioClips) and pictures or videos laid over the video (addOverlays) are picked the same way and go one after another from the playhead; in the picture editor the pictures all show at once, each a little lower and to the right of the one before (STAGGER). The file the project is opened with is still picked alone;
  • a clip can show a still: a picture added with "+" or opened as the project, or a freeze frame (the "Freeze" tool cuts the clip under the playhead and puts that frame in between with insertAt). StillMaker writes every still as an upright PNG in the cache, so FFmpeg, the preview and the thumbnails all read the same file. A still has no length of its own (MediaInfo.isStill, SourceInfo.isStill): its clip lasts 3 s and the clip panel changes that. The export loops it (-loop 1 -framerate <rate> -t <length>) and gives it a silent track; a still is never reversed and never stream-copied. A still clip can have a slow camera move ("Ken Burns", Effects.motion = StillMotion: zoom in, zoom out or pan left / right / up / down, enlarged up to 2x). The export puts zoompan=z=..:x=..:y=..:d=1:s=WxH:fps=F at the head of the clip's chain, with the progress p=min(on/(N-1),1) over the clip's N frames; the preview scales and shifts the picture by the same zoomAt / xAt / yAt;
  • the selected clip's edges on the editor's timeline are handles (tabs in the primary colour, rounded outside and straight inside, with a chevron; the one held grows wider and glows; texts, pictures and sounds wear the same): dragging one trims the clip (Timeline.trim, as the Trim tool does), the clip drawn as it will be while the finger moves with its length in a pill (and its neighbours with it: an end pushes or pulls the clips after it, a start the clips before it, so nothing overlaps or leaves a gap), the edge snapping to the centre line and the preview showing the frame under the edge; one undo step on release. While an edge or an item is dragged the view holds still, and when the finger rests near a side the timeline moves on by itself, frame by frame, faster the nearer it is (edgeScroll), the edge going with it; on release the playhead is brought to the view;
  • a clip held a moment (any clip; on the selected one, not on its handles) is picked up and follows the finger, lifted on its shadow, while the others make room where it would land (ClipMove, moveLayout in EditorTimeline.kt); a tick marks each new place, the timeline moves on by itself near a side, and letting go moves it there (Timeline.move, one undo step), as a placed sound is moved by dragging it;
  • "trim the whole video" does not touch the clips: ConversionSettings keeps trimStartMs / trimEndMs on the timeline and Timeline.window() returns the clips that really play (ConversionSettings.playingClips). The preview and the export both start from that list, so they treat any number of clips as one video.

RequestMapping.kt turns the playing clips into ConversionRequest.segments (each pointing into sources, the distinct files in use). The export opens one seeked input per segment and joins them with the concat filter, which is frame accurate and independent of key frames. Clips taken from different files are scaled and padded to the frame of the first one, and silent clips get a silent track, so files of different size, frame rate or audio layout can be mixed. Everything after the join (filters, overlays, fades, split) works on the joined sequence. Subtitles are timed against the main file, so SubtitleRetimer rewrites them to follow their footage through cuts, moves and reversal before they are burnt in. It also drops what a SubStation Alpha script carries embedded (its [Fonts] and [Graphics] sections): FFmpeg's subtitles filter has libass extract such fonts, and FreeType would parse them inside the app's process, where a crafted one could reach a bug of its (CVE-2025-27363 affects the FreeType 2.13.0 of FFmpegKitNext 6.1.1). Fonts are picked in the app instead, or come from the phone.

Placed sounds

Music, voice-overs and effects are not tied to clips: like in CapCut they are placed freely on the exported video's clock (AudioClip in EditorModels.kt, its SoundSpan in core/timeline/SoundSpan.kt). A span says which part of the file plays (trimStartMs..trimEndMs) and from when (startMs); sounds may overlap. Cutting, moving or deleting clips does not move them, exactly as in the usual editors. SoundSpan keeps every edit valid (inside the file, never before the start, at least 100 ms), cuts a sound in two and packs overlapping sounds onto rows (SoundSpan.rows), all unit tested.

Each sound has its own tools: trim, volume (with mute and loudness levelling), speed (keeping the pitch or not), voice (a pitch shift in semitones, with presets), fades, a three-band equaliser (with presets), repeat (the trimmed part plays N times) and replace (another file, same place and settings). Speed and repeats change the sound's length on the video; SoundSpan takes them into account in every edit, so a dragged edge still follows the finger. A repeated sound is not split.

RequestMapping.kt turns them into ConversionRequest.sounds (PlacedSound); a muted sound is left out. The export gives each one an input of its own: atrim for its part of the file; aresample=48000 and aloop for repeats; asetrate + aresample for the pitch and an atempo chain for the rest of the speed (pitch factor = 2^(semitones/12), times the speed when the pitch follows it); bass / equalizer / treble; its own effects (the same steps as the final sound's, see "Audio files", cut at its end when they ring on); loudnorm; its volume and fades; adelay=delays=<start>:all=1 to put it in place (timestamps renumbered after it); the whole video's fades; then padded / cut to the output length. Every input of the mix is made stereo with pan=stereo|FL=FL+FC|FR=FR+FC, which keeps stereo as it is and puts mono on both sides at full level (as a player plays it; FFmpeg's own conversion would lower it by 3 dB). The source sound (if kept) and all the sounds are mixed with amix ... normalize=0, so each keeps its own level. Whatever runs past the end of the video is cut.

The timeline draws the sounds through OutputClock (core/preview), which converts between the output clock and the timeline (clips at media length, before the trim) and extends both ways past the ends, so a sound sticking out is still drawn whole and can still be grabbed.

The editor side is described under "Editor layout" below.

Audio files

When the output is an audio file (OutputFormat.isAudioOnly: MP3, M4A, OGG, OPUS, FLAC, WAV) the editor works on the sound alone:

  • the preview pane shows AudioPreview.kt instead of the picture: the exported sound as rounded bars (preview/AudioEnvelope.kt builds the outline of the output from the clips' and placed sounds' waveforms, through the output clock, with volumes, fades and levelling, sounds heard together added as their powers, a clip heard from its stems counted once, and the stems of one file laid at one place counted as their sum but never above that file's own peak there at their loudest level (they are parts of it: as powers, five stems peaking at 0.6 made 1.34; a premix is read as it is); values above 1 would clip and are drawn in the error tone), the playhead, dashed lines where the file will be cut into parts, the tags and format on top and the effects in use below. The bars at the playhead and a morphing badge follow the level that is playing (LevelMeter, fed by the players). Touching or dragging the waveform seeks. The player does not decode the picture at all;
  • under the waveform, the spectrum of what plays (AudioSpectrum.kt, setting AppSettings.audioSpectrum, on by default; it is left out when the pane is too short for both, the waveform keeping at least 96 dp). The analysis is core's SpectrumBands and SpectrumBars (tested), ported from WiFi Audio Streaming's desktop spectrum (same author, same licence, EUPL-1.2): a Hann window of 2048 samples, an FFT, 48 bands evenly spaced in pitch from 35 Hz to 18 kHz (each its loudest bin), a tilt of 6 dB towards the highs, -66 to -9 dB shown, quick rise, slow fall, caps that fall under gravity; timed in seconds rather than frames, and the players of the preview added as their powers band by band, each at its rate and volume. It must show what is heard, not what goes through the effects: the speaker keeps up to two seconds in hand. So each player's AudioEffectsProcessor keeps its last five seconds of sound, mono, in a SpectrumTap, timed by the player's own sink (TimedSink, a ForwardingAudioSink: the time of each buffer handed to it, and the position it reports as heard, on the same clock), and the spectrum reads the samples that end where each speaker is (SpectrumHub.read). A seek flushes the sink and starts the run again; nothing is kept while no spectrum is shown (watch). Drawn on every frame without composing: pills from the primary to the tertiary colour with a faint reflection, caps as dots, a glow that follows where the sound is; the pills grow in one after the other, and while paused they breathe in a slow wave;
  • the timeline draws each clip's waveform instead of frames and shows only the sound lanes; the trim panels trim on the waveform too;
  • the tools are those of AudioPanels.kt: silence (moves the whole-sequence trim past the silent start and end, measured on the waveforms), audio (volume, levelling, placed sounds), equaliser, clean up, dynamics, space, effects, pitch, speed, fades, stereo, tags and parts; a clip has trim, speed, volume and fades.

The same tools exist in the video editor: its "Audio" tool opens a row of sound tools for the whole video (volume and sounds, equaliser, clean up, dynamics, space, effects, pitch, stereo), and a selected placed sound has clean up, dynamics, space, effects and stereo of its own (AudioClip.effects, next to its three-band equaliser and voice).

The footage's sound is one hierarchy, and the audio panel (AudioSectionContent) shows it as such. The whole video's track is the master: its switch (wholeEffects.muted) drops the footage's sound of every clip, and its volume multiplies each clip's (Effects.volume), exactly as the export and the preview apply them. Under the master the panel lists every playing clip with sound (ClipSound), recognisable at a glance: its first frame with its number on the timeline (rememberClipThumbnail, as the clip strips draw it), its file name and a tag on the clip under the playhead; then its switch, the level it is heard at (its own volume times the master's) and, on a tap, its own volume panel, with the clip selected and shown. A clip's panel says when the master silences it, with the level actually heard when the master's volume is not 100 %. Turning a clip on while the master is off lets only that clip be heard (ConversionViewModel.soloClipSound, one undo step: the master back on, every other clip muted, every volume kept). Turning the master off and on again leaves each clip's own settings as they were.

What is done to the final sound is AudioEffects (core/model/AudioEffects.kt): ten octave bands of equaliser, low and high cut, noise gate, compressor, lo-fi bits, one modulation (chorus, flanger, phaser, tremolo, vibrato), echo, reverb, stereo image (width, balance, swapped sides, mono), limiter and pitch. core/audio turns it into a list of AudioStages, each of which is both an FFmpeg filter (highpass, lowpass, equalizer, agate, acompressor, acrusher, chorus, flanger, aphaser, tremolo, vibrato, aecho, extrastereo, pan) and a sample-by-sample Kotlin port of that filter's C code (core/audio/Dsp.kt: same coefficients, tables, rounding and state). The reverb is a dense aecho, so it is exact too. The export runs the stages after the mix (AudioFilters.masterChain, on float samples), then the pitch shift; echoes and choruses would ring past the end, so the sound is cut at the output length. The preview runs the same stages in a Media3 AudioProcessor (preview/AudioEffectsProcessor.kt) on the footage's player and on every placed sound's player (where the sound's own three-band equaliser runs too, with the export's own filters), and the pitch is the players' pitch. Checked against FFmpeg 6.1 on a test signal, the ports match bit for bit except the biquads, which differ by float rounding (below -70 dB).

How an audio file is encoded is AudioEncoding (bitrate of the lossy formats, bit depth of WAV and FLAC, sample rate; Opus always runs at 48 kHz), set in the export sheet: each choice says what it is good for, a five-bar meter shows where the bitrate sits, and the file's size is estimated (lossy formats, and WAV at a chosen sample rate).

The audio panels lean on ValueSlider (ui/expressive/Panels.kt): with a ValueInput (unit, scale, step) it gets - and + buttons that repeat while held and a field opened from its value pill to type the exact value, with the accepted range and a sign button. Every slider ticks under the finger (rate limited), ticks harder at its default, and gives TalkBack its value as the state description. Parts show the result first (the number of files, the sound as a bar cut into them, the length of each and of the last one, read out by TalkBack as it changes) and set the length in minutes and seconds, on a slider that is finer at the short end, or with quick lengths; equal parts have a big count between repeating - and + buttons. Tags show the file as a player lists it, with icons, clear buttons, next-field keys, a year check and genre chips; silence draws the sound with its silent ends hatched and the button says how much it cuts.

Image files

When the output is a picture (OutputFormat.isImage: JPG, PNG, WEBP, BMP, TIFF) the editor is ImageEditorScreen.kt, a photo editor: the picture fills the top, the tool row and the open panel (ImagePanels.kt) sit below it, as in the video editor. The picture is the first clip's file: a photo (opened as a still), or the frame of a video at ImageEdit.frameMs (the Frame tool; the export seeks there with -ss, kept 100 ms before the end of the clip by pictureFrameMs).

What is done to it is ImageEdit (core/model/ImageEdit.kt) plus, for the turn, mirrors and crop, the whole video's Effects; texts and stickers are the project's own texts and pictures (whole video, no clock). In order:

  1. geometry: mirrors, quarter turn, keystone (perspective), straighten (rotate, keeping the largest part of the same shape that the turned picture fills, so nothing is scaled up and the picture gets a little smaller), crop;
  2. tone (exposure, brightness, contrast, highlights, shadows, warmth, tint, fade, with the chosen filter's tone added) as one lutrgb expression per channel;
  3. colour (saturation, hue, black and white, sepia) as one colorchannelmixer matrix;
  4. one effect (blur boxblur, pixels pixelize, glitch rgbashift, poster and invert and duotone as lutrgb, emboss and sketch as convolution), then sharpen (convolution), vignette and grain (noise);
  5. texts (drawtext), stickers (overlay), then the drawing;
  6. the border (pad) and the output size (scale, never larger).
  7. rounded corners (ImageEdit.corners, core/image/RoundCorners.kt): one geq over the whole picture at its output size, from square (0) to half the shorter side (1: a square picture becomes a circle), its edge smoothed over a pixel; the cut-off parts of the colour chosen (ImageEdit.cornerColor, cornerFillFor): by default transparent in PNG, WEBP and TIFF, black in JPG and BMP. A colour in a format with transparency is mixed in RGBA, so the rest of the picture keeps its own (FFmpeg 6.1's rgba to gbrap conversion may move a half-transparent pixel's alpha by one). i=nearest, since geq's bilinear reading takes the last row and column from the ones before. roundCorners is its exact port (checked against FFmpeg pixel for pixel); the editor shows it as a rounded clip of the stage (with white behind it for a format without transparency).

core/image turns it into ImageStages, each both an FFmpeg filter and a Kotlin port of that filter's C code (ImageStages.kt), so the editor shows the export's own result: imageStages builds them for the real picture's size in the export and for the preview's smaller copy (at most 1080 px, rendered off the main thread, preview/ImageRendering.kt) in the preview. PictureRenderer keeps the geometry's result while only light, colour or effects change. Two painters share the work: one follows the finger, making every look as soon as the last one is done, at full size when that takes 30 ms or less and otherwise on a smaller copy whose size adapts so that a picture takes about two frames; the other makes the full-size picture once the look has rested for 150 ms and drops it as soon as the look changes. A picture is shown only if nothing newer is on screen. The command shown in the editor is rebuilt off the main thread once the settings rest, so a slider never waits for it. The panels are built low (one slider over a row of round chips) and never take more than about a third of the screen. Checked against FFmpeg 6.1 on a test picture, tone, colour, filters, poster, invert, duotone, turn, mirrors, crop, straighten, sharpen, emboss, sketch, pixels and glitch are bit for bit equal; perspective, blur and vignette differ by at most one level on some pixels; grain is FFmpeg's random noise, so the preview shows noise of the same strength, not the same grains.

The Layers tool (LayerPanels.kt; the video editor has it too, for the texts, pictures, videos and shapes of the whole video) lists every text and sticker of the picture, the one in front at the top (the reverse of stackOrder), each with its picture (or its text in its own colour and font) in a MaterialShapes badge. A row dragged by its handle follows the finger while the others make way with the theme's spatial springs; dropped, placeLayer puts it at that place of the stack (movedLayersTo) and numbers every layer again, as one undo step. A tap selects the layer on the picture too, and the selected row opens its own actions (to the front, up, down, to the back, duplicate, delete); TalkBack gets the moves as custom actions of the row.

Each row also has a lock and an eye (issue #30), as the SVG editor's layers (SvgLayers.kt, where both were already): TextOverlay.hidden / locked and ImageOverlay.hidden / locked, kept in projects, each switch one undo step. A hidden layer stays in the project and the list (dimmed, "Hidden") but is left out everywhere else: StackedOverlays does not draw it (both editors), the video editor's preview neither draws nor plays it (PreviewContainer: its sound, its picture look), the timeline draws it faded, and RequestMapping leaves it out of the export (footage, toRequestOf, lookSpans). The picture itself on its canvas (PHOTO_LAYER_ID) has no eye. A locked layer cannot be picked nor moved on the picture: touches go through it (its tap and transform gestures are not installed), and selected from the list or the timeline it shows a small lock cookie where its resize grip would be; its panels still change it, and the timeline still moves it in time (the lock is for its place on the picture). Each switch is a round button that morphs into a cookie, in the tertiary tone, while it is on; TalkBack reads it as a switch.

The drawing (ImageEdit.strokes: free strokes, lines, arrows, boxes, ovals, eraser) is drawn by StrokePainter (:media) on the screen and, at export, into a transparent PNG (DrawingRenderer) that the export lays over the picture like a sticker, the whole picture wide.

A canvas of its own (ImageEdit.canvas, PictureCanvas: a size, a fill colour or none). The file is then the canvas and the picture one of its layers: in the editor it is the image overlay PHOTO_LAYER_ID, kept among the stickers so that it is dragged, pinched, turned, given a border, a shadow, an opacity or a blend and stacked in Layers exactly as they are, but drawn from the edited picture (StackedOverlays(pictures = ...)) and never a file laid over: footage() leaves it out of the pictures and the request carries it as ConversionRequest.photoPlacement. The export makes the canvas with color (transparent black@0, or white where the format holds no transparency), makes the picture as always into the link photo, and stacks it with the captions and stickers by its layer (overlayGraph, whose overlays are read from a stream or a link), mixed in RGB; border, output size and corners then apply to the canvas. editedSize is the canvas (the frame texts and the drawing lie on), pictureSize the picture itself (the crop works on it; while cropping the picture shows alone). Ultra HDR is not offered on a canvas.

Two fingers on the canvas zoom it (1x to 8x, around the fingers) and move it (StageZoom.kt): a graphicsLayer scales the whole canvas (picture, texts, stickers, drawing, crop frame), so every touch keeps landing in the canvas's own coordinates and drawing zoomed in is exact. The stage's detector only takes fingers nothing on the canvas took: once a layer consumed a move of the gesture (a sticker or text dragged, pinched or turned, the crop frame), the gesture is the layer's to the end, a second finger beside it included; and the zoom starts only past 1.6 times the layers' slop, so a layer under the fingers always claims them first (it sees every move before the stage). A stroke under way when a second finger lands is dropped (DrawingLayer). The zoom pill springs it back to fit. Zoom is a view of the canvas only: nothing of it reaches the project or the export.

While a picture is cropped, in the photo editor and in the video editor alike (issue #33), the view frames the crop frame each time it is let go: zoomed in until the frame fills the stage but for a margin left for its handles, and centred on it (core/image/CropFocus, tested; StageZoom.animateTo, the theme's default spatial spring). Never while a finger holds the frame (CropOverlay's onHeldChange): the picture does not move under the finger. A frame that nearly fills the picture leaves it as it fits; the crop tool closed, the picture fits again. In the video editor the same zoom is a layer of the frame (EditorPreview), which clips it.

Texts, stickers, shapes and the drawing are clipped to the picture on screen, as the export clips them: a layer moved partly off the picture is cut at its edge and never covers the editor (the file name, the tools).

The bundled FFmpeg has no WebP encoder: for WEBP the engine has FFmpeg write a PNG in its working directory and ImageRecoder (AndroidImageRecoder, Bitmap.compress) writes the WebP from it (lossless at quality 100 on Android 11 and later). JPG uses mjpeg (full-range 4:4:4), PNG, BMP and TIFF (deflate) their native encoders.

SVG

Opening one. FFmpeg cannot read an SVG: when the probe fails, the file is tried as a picture the phone draws (openPictureAsMain); AndroidStillMaker recognises an SVG (its type, name or first bytes) and draws it with AndroidSVG (Apache 2.0) at 2048 px on its longer side, transparency kept, into the usual PNG still. From then on it is a picture like any other.

Saving one (OutputFormat.SVG). FFmpeg makes the picture as a PNG, as for a WEBP; the engine reads it (ImageRecoder.read) and traces it (core/vector/VectorTracer.kt, ImageEdit.trace):

  1. at most maxSide px on its longer side (box filtered, colours weighed by their opacity); pixels below half opacity stay empty;
  2. a few colours: k-means (k-means++ with a fixed seed, so the same picture always traces the same) on a sample, the green weighed more; or, in black and white, the dark side of Otsu's threshold;
  3. each colour followed along the pixel edges into closed outlines, with their holes (filled even-odd), turning right where two pixels touch by a corner so that they stay apart; specks below speckle px dropped;
  4. each outline simplified (Ramer-Douglas-Peucker, 2.2 to 0.8 px by the detail) and, smooth, drawn as quadratic curves through the middles of its sides, turns sharper than about 70 degrees kept as corners;
  5. colours laid largest first, each stroked with itself by half a pixel so that no hairline shows between neighbours;
  6. with gradients asked for (TraceSettings.gradients, colours only), each colour's area split into its shapes (4-connected, as the outlines keep them apart); every shape of at least 120 px (and 1/4000 of the picture), the 400 largest, measured on the picture under it: a least-squares plane per channel gives the direction its colours change most, a least-squares bowl (c = a + bx x + by y + q (x^2 + y^2), one centre for the three channels) the centre they fade from; the pixels along each are cut into five bins from the 2nd to the 98th percentile, each bin's mean colour a stop. The radial one wins when it leaves at most 80 % of the linear one's error and its centre is near the shape. The gradient is kept when it leaves at most 55 % of a flat colour's error and its ends differ by about 12 levels a channel; the shape is then a layer of its own, just above the flat rest of its colour, filled and stroked with a userSpaceOnUse gradient (ffg-trace-N) in the traced pixels. Flat pictures trace exactly as without; a 1200 px photo takes up to about two thirds more time.

The shapes tool previews it over the stage (VectorPreview, the same path steps) while it is open. Shared copies of an SVG go to Download (not every gallery takes one).

The SVG editor

A screen of its own (app/.../svgedit/), for what is inside an SVG rather than the picture it makes. It opens from home (an SVG picked, or an empty drawing) and when SVG is picked as the format of a picture: an SVG opens as it is, any other picture is traced into it (SvgEditorViewModel.trace, the same VectorTracer, one path per colour named after it) with a card to change how until its shapes are edited. The photo editor's Trace tool opens its shapes there too, as they are traced at that moment ("Go on drawing it in the SVG editor", SvgEditorViewModel.openTraced: the same layers and names). That card and the picture editor's shapes tool share TraceControls (TraceStudio.kt): the kind of picture first (logo, drawing, text, illustration, photo, pixel art, each a ready TraceSettings; a photo's shapes are shaded with gradients), then what the last trace made (its palette, its outlines, how many are shaded and how heavy the file is), then its colours (with the gradients switch) and its shapes, every slider naming what its two ends do. A shaded shape is named after the colour it shades (gradient-rrggbb). A video's frame saved as an SVG still goes through the picture editor. The toolbar adds shapes (a gallery of basic ones, lines and every Material shape as an exact path), texts, and curves drawn with the pen (smoothStroke: simplified, then Catmull-Rom cubics); a sheet changes the selection's look; a tour shows the gestures the first time.

Finer tools (SvgShaping.kt). The side handles stretch a node one way in its own units (resizeOwn): a shape's geometry takes it (a circle becomes an ellipse, a path is moved point by point), so its outline keeps its width. A moved box snaps to the edges and middles of the page and of the other nodes (snapTargets, snapBox); turning rests on every 15 degrees. A connector is a line with data-connect-from / data-connect-to naming the shapes it holds (ids are made up for shapes that had none); after every change reroute puts its ends where the line between the shapes' middles leaves their boxes, or on the anchor it was dropped on (data-connect-from-at: fractions of the shape's own box, so it turns and sizes with the shape); a connector may run straight, in square turns or as a curve, leaving and reaching the shapes square to their sides, with chevron arrows (data-connect-style, data-connect-arrows; a path connector keeps its free ends in data-connect-ends).

Points (SvgPathEdit.kt). Any outline is edited as its points and the pieces joining them (EditPath): straight, a cubic curve (its controls dragged, or bent symmetrically), a zig-zag or a wave (how many, how tall), written into the path as the lines and curves they make, how they were made kept in data-ffg-path (dropped when another program changed the path). A tap on a piece adds a point there (a curve split where it is); the points of a new line are placed one tap each.

Several at once (SvgArrange.kt). "Pick several" makes a tap add a node to the selection or take it out (SvgEditorViewModel.also, next to the one picked first); they move, size and turn together by their bodies (transformAll on the outermost of them, snapping to everything else), and the look sheet changes them all. The "More tools" sheet (SvgActions.kt) lines them up on the box around them (one alone: on the page), spaces them out, mirrors and turns them, groups them (where the top one was, each keeping its place on the drawing) or takes a group apart, makes a basic shape a path, copies a look and pastes it, and places them by numbers.

Booleans (SvgBooleans.kt). Union, subtract (the top ones out of the bottom one), intersect, exclude and divide, as in Blender or Illustrator: each node's area on the drawing (its fill; for an outline with nothing inside, the band its stroke paints, Paint.getFillPath; a text as its letters, Paint.getTextPath; a group as everything in it) goes through Skia's Path.op, and the result is read back as lines and curves (PathIterator, conics as quadratics) into a path that takes the place of what was picked, in the look of the bottom one (replaceWithOutlines: where the top one was, in its units). Divide gives every piece the outlines cut, each in the look of the top one over it, bits apart as shapes of their own. The same reading back makes a stroke a shape ("Stroke to shape") and a text its letters ("Text to shapes").

Professional tools (SvgPro.kt, tested; SvgSymmetry.kt, SvgActions.kt).

  • Live symmetry. A group marked data-ffg-symmetry="kind;count;cx;cy" holds a source group and copies of it (data-ffg-copy), mirrored left and right, top and bottom, in four quarters, turned 2 to 24 times, or turned and mirrored (a kaleidoscope), about the page's centre. Whatever the editor adds goes into the source; after every change (live gestures too) the view model makes the copies again from it, only when the source changed, each copy keeping its keys where its tree has the same shape (the renderer's cache is by key). Copies are real SVG content (the file looks right in any viewer) but are skipped by hits, so only the source is picked; the layers leave them out. The stage draws the axes and the centre, and the pen's stroke on every side as it is drawn. Turned off, the copies are kept as ordinary groups or taken away.
  • Grid. A number of squares across the shorter side; drawn on the page (every fourth line stronger) and, when asked, its lines added to the snap targets of a move (SnapTargets.withGrid).
  • Magnetic points (SvgPointSnap.kt, tested). A point dragged in the points mode, or placed with a tap while building a line, lands on what is near it (snapPoint): another point of the drawing first (pathPoints, and the symmetry's centre); else the page's edges and middle and, when moves snap to the grid, its lines and the middles of its squares (gridSnapLines), each way on its own; else the symmetry's axes and, for a turn, the lines half way between them (Symmetry.snapRays), or where one crosses a line the point already rests on. The stage draws the guides and a ring that pops on the landing, with a haptic tick.
  • Locking. data-ffg-locked="1": skipped by hits and by select all; unlocked from the layers' padlock.
  • Repeat. In a row (across or down, a gap between the boxes), in a grid, or turned round the page's centre: each copy duplicated from the last and placed from the original's place, so the stack stays in order.
  • Mirrored copy: a copy flipped and laid beside the original.
  • Paths. Simplify (each piece flattened, thinned within 0.4 % of the page by Ramer-Douglas-Peucker, smooth curves laid through), smooth (every corner a Catmull-Rom curve), reverse (each piece end to start, a curve's controls swapped), break apart (each piece a shape), join (several shapes one even-odd path, in the first one's units and style).
  • Align on the page, for several, as well as on their box.

Gradients (core/vector/svg/SvgGradients.kt, tested; SvgGradientEditor.kt). In the look sheet a fill or an outline is none, a colour or a gradient (a line has no box for one: colours only). The editor shows a gradient as a GradientSpec: linear or radial, an angle, stops (offset, colour with its opacity). On the sheet: a tile painted as the shape is, a bar of the stops (a tap adds one in the colour the gradient has there, up to eight; a stop dragged along stays between its neighbours), the kind, the angle (resting on every 15 degrees), and the chosen stop's colour, opacity and place; reverse and remove. paintWithGradient writes it into the definitions at the top (into the first defs, or a new one): in place when the gradient is the node's alone and written plainly there, else as a new ffg-gradient-N, so that a duplicate, a symmetry copy or a file's shared gradient never changes with it. When only the stops change, the gradient keeps where it lies as written (its units, coordinates and transform, a traced one's pixels too); a new kind or angle lays it across the node's box (objectBoundingBox, long enough to reach the far corners). When the drawing is written, the app's own gradients (ffg- ids) that nothing names any more are left out (withoutUnusedGradients); a file's own are always kept, as some programs keep unused ones as swatches.

Continue editing (SvgDrawingStore). A moment after each change (and when the editor is left) the drawing is kept in files/drawings/<id>/ (drawing.svg, name.txt, thumb.png) when the app keeps projects; its cards (ProjectCard.drawing) join the recent projects on the home screen, newest first, and open, copy and delete as they do.

Paint bucket (core/vector/RegionFill.kt). The drawing is drawn into a picture (at most 1400 px), the space under the finger flood-filled up to any other colour, gaps closed by growing the borders and growing the space back, then traced (mono) into an even-odd path laid just above what is filled there.

The drawing (core/vector/svg/). Xml.kt reads and writes the file by hand (a doctype is skipped, its entities never expanded: nothing outside the file is read). SvgDocument holds it as the editor sees it: groups, basic shapes and paths, texts, pictures, each with its transform and its style; definitions, styles, metadata and every element the editor does not know (SvgOther: a use, a clip path...) are kept and written back as they were. Styles are read as CSS orders them (presentation attributes, then the file's simple class, tag and id rules, then the inline style) and written back inline, so that what the editor changes wins over the file's rules. SvgGeometry.kt has the transforms (Affine, SVG's matrix) and the whole path grammar (relative commands, shorthand curves, arcs made into cubics).

Editing (SvgEdits.kt). Nothing changes in place: every change (move, size, turn, hide, duplicate, delete, reorder) gives a new drawing, and undo keeps the old one (SvgEditorViewModel, a step per gesture). A transform of the drawing's pixels is laid on a node's own transform through what its groups and the view box do (transformInWorld), so a node moves under the finger whatever it sits in. A tap finds the node on top there (hits: filled by the non-zero or even-odd rule, or near its outline); a tap on the selection reaches the one under it.

Drawing it (SvgRenderer). From the editor's own model, onto the stage's canvas, so that a change shows at once: outlines, solid colours, gradients, dashes, opacity, texts and embedded pictures are drawn there; SvgOther nodes and texts placed piece by piece are drawn by AndroidSVG within the same transforms. Clip paths, masks and filters are not drawn while editing (they stay in the file).

A text's outline is drawn from its letters' outline (Paint.getTextPath) with the overlapping pieces of each letter merged first (Skia's path ops, Path.op UNION; kept for the lines drawn lately): the phone's sans and other fonts made of overlapping pieces would otherwise show their edges inside the letters (issue #23). Text made into letters is merged the same way.

The editor's panels (PeekThroughSheet) fade, with their scrim, while a control that shows its change live is used, so the drawing under them shows it: the controls marked Modifier.peeksSheet() (ui/expressive/SheetPeek; the sliders, ValueSlider and CompactSlider, and a gradient's stops), held still or dragged sideways. A hold or a sideways swipe anywhere else (a tile, the layers being reordered) leaves the panel as it is (issue #25).

ASCII and ANSI art

Saving one (OutputFormat.ASCII, a .txt; OutputFormat.ANSI, a .ans). FFmpeg makes the picture as a PNG, as for an SVG, transparency kept (a cut-out's background is left empty in the art); the engine reads it (ImageRecoder.read) and writes it as text (core/textart/, ImageEdit.textArt). Only what the text-art editor shows goes into the PNG: ConversionSettings.forTextArt and ImageEdit.forTextArt leave out the captions, stickers, drawing, border, corners, canvas, HDR look and output size, and the command is always the one the app makes. The characters are chosen with the system's monospaced font as AndroidGlyphAtlas (:media, behind GlyphAtlasSource) measures it: each character the editor offers drawn at 96 px and its ink averaged in 4 x 8 sub-cells, the characters it has no glyph for and those drawn wider than a cell noted. The editor and the export share that one measure, so they choose alike.

TextArtRenderer.render (deterministic, no randomness anywhere):

  1. the grid: columns across, the rows from the picture's shape and a cell's width over its height (the font's, or cellAspect), or rows when set (the picture then stretched or squeezed to fill the grid);
  2. the picture read once into samples, every pixel into exactly one, its colour weighed by its opacity: one a cell, or a part of one where the mode looks closer (two halves, two by two quadrants, two by four Braille dots, the atlas's sub-cells for shapes);
  3. the tone, on OKLab's lightness: histogram equalisation, brightness, contrast, gamma, a five-point curve, negative, the paper (light ink on a dark page or dark on light), edges sharpened (unsharp mask);
  4. what is left out (ArtHide): below the opacity cut, a band of lightness, arcs of hue (only where the chroma is above the band's minimum), colours within a tolerance in OKLab, all of it turned round on asking, then the brush's strokes in order (hiding or showing again);
  5. the characters, by mode: the ramp (the alphabet sorted by measured ink, near duplicates merged), shapes (least squares against each glyph's sub-cells), lines along Sobel edges (- / | \, the rest from the ramp), shades, half blocks, quadrants, Braille, or the letters of a phrase; with the alphabet's limits (a bias a character, the largest share of the cells one may take, the longest run) and ordered (Bayer 4 and 8) or error diffusion (Floyd-Steinberg, Atkinson) dithering;
  6. for an ANSI file the colours: saturation, the light left to the characters if asked, k-means down to a number of colours, then matched to 16 (a terminal theme's), 256 (xterm's) or true colour by OKLab, CIEDE2000 or RGB, entries banned or painted as others, dithered if asked (the error kept small). With the cells coloured too, a cell's two colours are the 2-means of its samples, and the glyph is matched to where each sample lies between them; a flat cell is a space on its colour; of two halves or quadrants the lighter is the character, the darker the background (a 16 colour background may have only the dark eight).

TextArtLayout.compose lays out the frame, margins, title and signature; TextArtWriter writes UTF-8 text, or ANSI: SGR codes only where they change (UTF-8), or a reset and every attribute at each change, bold for the bright foregrounds and blink for the bright backgrounds, then the end-of-file mark and a 128-byte SAUCE record without a date (code page 437, 16 colours, Braille drawn as quadrants). TextArtReport is the editor's check: size, characters and colours used, and what to look out for where the text is going. TextArtPresets sets what each place (terminal, editor, Markdown, chat, BBS) asks for.

The format picker shows the two among the text formats (shownCategory): pictures to the engine, text to whoever picks them, offered for a file with a picture and no sound (a still, or a silent video: no sound track, or a track of silence, as a recording made muted has, found by listening to it with soundIsAudible in Waveforms.kt: its first ten minutes, a peak above -54 dBFS is sound, the reading stopping at the first one); a file with sound gets the transcripts in Text instead. Then its banner plays the landscape of the image film written in characters behind a scan line, coloured as ANSI art (TextArtShowcase in FormatShowcase.kt), and says so; with sound it keeps the transcript's film.

The text-art editor (app/.../textart/) opens instead of the photo editor when ASCII or ANSI is picked. Its picture is the photo editor's (PictureRenderer: the frame of a video, turned, cropped, cut out); the art is made off the main thread from it at each change of the settings, laid out, written and drawn (TextArtPainter: the characters in the monospaced font, block elements and Braille dots as their shapes). The stage shows it in a terminal window, alone, over the picture or beside it (a divider), typing itself in row after row when its kind changes (not when animations are off); a tap there picks a colour to leave out, a drag paints with the brush. The tools: the picture (frame, turn, crop), the style (where it is going, the mode with each drawn small, the size: columns and lines chained to keep the picture's shape or set apart, a frame of the grid's shape and common widths; the page, the cell's shape), the characters (presets, groups, a key for each character with its measured ink, switched off with a tap, preferred or avoided with a long press, the safety filters and the limits), the light (tone, curve, edges, dithering), the colours (ANSI: depth, theme, layers, matching, saturation, reduction, the palette with banned and remapped entries), what is left out (the cut-out panel of the photo editor, the picture's own colours, the hue ring, the lightness band, the brush), the layout and the check (with copying the text, as a Markdown code block, or with its colour codes).

GIF

A GIF is pictures only. With GIF chosen the editor has no sound tools (no Audio row, no sound lane, no volume of clips or of videos laid over the picture), no frame extraction and no remux, and the preview plays without sound; the request drops placed sounds, stream copy and frame extraction left over from another format. Its own settings are GifSettings (core/model/GifSettings.kt), in the GIF tool and in the export sheet in place of resolution, frame rate and quality: width (scale, never larger than the video), frame rate (fps), colours (palettegen=max_colors), dithering (paletteuse=dither: Sierra-2-4A, Floyd-Steinberg, Bayer at scale 3, none), how many times it plays (-loop) and a lighter file (stats_mode=diff and diff_mode=rectangle).

The preview plays the plain video, smooth and silent: it does not imitate the GIF's size, colours, dithering or frame rate (those show in the exported file).

Settings

settings/ holds them: AppSettings (theme, wallpaper colours, pure black, vibration, copies in the phone's folders, recent projects and how many to keep, Live Updates) kept with Preferences DataStore by SettingsRepository (a StateFlow that is null until read; the splash screen waits for it so that the first frame has the chosen colours). The splash screen itself (its animated shapes, their timing and the hand-over in MainActivity) comes from WiFi Audio Streaming for Android (https://github.com/marcomorosi06/WiFiAudioStreaming-Android), by the same author and under the same licence (its copy of the EUPL in assets/licenses/wifi-audio-streaming/, pinned in scripts/licence-copies.lock and checked by scripts/check-licence-copies.py before every release); the glyph and the colours are this app's. Where each one acts:

  • theme, wallpaper colours, monochrome, pure black: FFmpegGUITheme and the editors' EditorTheme (ui/theme/Theme.kt: appColorScheme, onBlack). The Appearance page picks the palette on three tiles, each drawn with its own accents: the wallpaper's (Android 12+), the icon's teal, or monochrome (MonochromeLightColorScheme / MonochromeDarkColorScheme in ui/theme/Color.kt: greys only, built like Material's monochrome scheme, the tool families apart by tone, errors still red). Monochrome changes the app's colours, not its launcher icon. Without wallpaper colours (or before Android 12) the app wears the tonal-spot scheme of the icon's teal #00897B, generated with material-color-utilities (ui/theme/Color.kt). The icon follows too (ui/theme/AppIcon.kt): in the app (home title, About) through appIconBackground/appIconGlyph, and on the launcher through two activity aliases of MainActivity, LauncherDynamic (wallpaper colours) and LauncherTeal (ic_launcher_teal); AppIcon.sync enables the matching one in onStop, enabling before disabling so that the app always has a launcher entry;
  • vibration: off, MainActivity provides a LocalHapticFeedback that does nothing, so every haptic of the app goes quiet at once;
  • copies: after a successful export the view model copies the file through MediaStore (export/PublicCopies.kt, Android 10+, no permission) to Movies, Music, Pictures or Download, in an FFmpegGUI folder; the done screen says where. export/ExportedFiles.kt measures and empties the app's own exports;
  • phone's video chip: the view model sets EncodingSettings.hardwareVideo on every request (the export and the command shown), see "Hardware video encoding" below;
  • Live Updates: AndroidExportNotifier asks for promotion only when on;
  • recent projects: see "Recent projects" below. Turning them off asks first, then deletes every project kept.
  • with the screen on its side (Editor): LandscapeLayout. As it fits (the default) the editors lay themselves out by the window's size, as before; with the player on the left or on the right, a window wider than it is tall always puts the player beside the tools, half the width each, whatever the video's shape (the largest preview for a vertical one). SideBySide (LayoutChoice.kt) lays the two out without building the player again when it changes side (it would start over): the halves slide across with the theme's spatial spring. Turning the phone changes the layout itself (stacked, side by side, studio): the video editor's preview is movable content (movableContentOf), moved to its new place rather than built again, so the video goes on from the same instant, playing or paused. Should a preview be made again all the same, its new player starts where the playhead was (EditorPreviewPlayer.startAt), not at the beginning. On a landscape screen the editors' top bar has the switch (ConnectedSwapSidesButton) that moves the player to the other side and keeps the choice, joined to Export as one connected group. While the settings keep the player on a side, the group is led by the way back to the usual layout (ConnectedUsualLayoutButton, the setting back to "As it fits"), but only where that changes something (offersUsualLayout: not on a window so low that it would be side by side anyway, a phone on its side). Beside the player the tools have the whole height, so the video editor's timeline grows into it (EditorTimeline's maxHeight, up to what its rows need) and shows more rows of lanes before they scroll, instead of leaving the space empty above the tool row;
  • the editors' top bar (EditorTopBar in ConfigScreen.kt, shared by the video, audio, GIF and photo editors): a floating pill on surfaceContainer, Close a tonal button that squares up under the finger, the file's name cut in the middle (its extension stays in view) over its target format, Export the one filled button. Every target is 48 dp; the bar is at least EDITOR_TOP_BAR_HEIGHT tall and grows with large text instead of cutting it;
  • the sound of the clips (Editor): the video editor's timeline draws each clip's own sound along the bottom of its frames (drawClipSoundBand in EditorTimeline.kt, from the same WaveformCache the placed sounds use: the peaks of the clip's part of its file, growing up from the bottom over a dark band, faint when the clip is muted). Not for a GIF, a picture or a file without sound; an audio file's clips are their waveform already.

The pages (SettingsScreens.kt, SettingsUi.kt) are grouped lists in the Material 3 Expressive style: the rows of a group touch, very round at the group's ends; each row's badge is a MaterialShapes shape that morphs into a circle under the finger, with a spring squish and a haptic (toggle on/off for switches, confirm and reject in dialogs). A list of categories (each with its shape and colours), then the project (About, the source on GitHub, Ko-fi) and a reset. Page titles are set in the display face (Google Sans Flex, rounded and wide). AboutScreen.kt has no title, its hero says it: the logo grows in with a spin and keeps morphing through full shapes (no deep pinches, the glyph drawn smaller than on the launcher so it always fits) on a slow turn (a touch skips ahead with a bounce and a tick); the name in the display face; the motto ("Why would a video editing app need internet permission?"); FFmpeg's version from FFmpegKit (media/FfmpegInfo.kt); the licences (the typeface's OFL text from the assets). Links open in the browser: the app itself never goes online.

Settings pictures (SettingsArt.kt). Every other category, and the About page, opens on an ExpressiveStage: a very round card in a gradient of two containers with three big library shapes behind, turning each its own way and changing shape in turn. In front, what the category changes, made of shapes and moved by the theme's springs (overshoot and all), following its settings: the app in the current theme (the welcome's PhonePreview), clips as pills under a gem playhead that hops between them (their sound bars only when the waves are on), a phone in a blooming badge that kicks with each buzz while shapes fly out (asleep with the taps off), a file flying in an arc into one folder or two, the projects fanned or closed (ProjectsArt) and the export's notification (NotificationArt). Hidden from screen readers, still when the system's animations are off. The appearance and vibration pages use the welcome's own controls (ThemeTile, HapticPad); About groups its licences in two folding sections (in the app, models you download).

Smart features (SmartHub.kt, SmartShowcase.kt, SmartSettings.kt). Not one long page but a hub and a page per feature. The hub: a hero (the features' badges going round a turning shape, a ring of how many are ready, the privacy promise, the speed switch for every model with a chip of nine cores lit in turn while it is on), then a tile per feature (speech, translation, cut-out, eraser, voice apart, dubbing), two by two, each playing its own small film drawn by the app in the theme's colours (the welcome's art where there is one: the spoken words, the cut-out), with a line and a status pill (ready and its size, not set up, importing, needs a look, needs the AI marks); last, the models' licences. Above them a banner says whether the AI marks are here and leads to their page; while they are not, the dub's and the eraser's tiles are grey (see "Marked as made with AI"). A tile opens the feature's page: its film large, its badge blooming in, a line, its status, and only its own cards. Pages come in sliding and growing, and back goes to the hub (BackHandler). Screens that send the user to set something up name the page (SmartPageRequest), so "Get the eraser model" lands on the eraser's page.

Editors in a stack

The editors open on top of one another (core/stack/EditorStack.kt, ConversionViewModel "Editors in a stack", EditorStackUi.kt): a part of what one edits opens in the editor made for it, edited as a project of its own; Apply renders it into the cache (NestedRenderer, implemented by media/FfmpegNestedRenderer with an engine of its own writing to nested-work/, the result moved into nested/, owned by CacheJanitor) and it takes the part's place below, keeping its nested project (NestedEdit on EditorSource, ImageOverlay and AudioClip, written recursively by ProjectJson; Json.MAX_DEPTH 1024). Nothing changed since it opened (no step in its history): nothing is rendered, the editor below simply comes back.

The links (NestedKind, tested: its part, NestedPart, its editor, NestedEditor, and the format it is edited and rendered as):

From Part Opens in Applied as
video, GIF editor a clip video editor (MP4) the clip rendered, in place of the clip
video, audio editor a clip's sound audio editor (FLAC) under the clip's picture (SoundSwap); an audio file's clip, the clip itself
any editor a picture of the timeline (a photo, a freeze frame); the photo editor's own picture photo editor (PNG) the picture, in place of the still, as long as it was
video, GIF, photo editor a picture or sticker laid over photo editor (PNG) the picture laid there (its shape read again; a subject key dropped)
video editor a video laid over video editor (MP4) the whole video made, laid there
video, GIF editor a GIF laid over GIF editor the whole GIF made, laid there
video editor a video laid over: its sound audio editor (FLAC) under the video's picture (SoundSwap)
video, audio editor a sound of the timeline audio editor (FLAC) the whole sound made, played there (measured again when levelled)

A shape and an effect of the app's open in no editor: their own panels edit them. The text-art editor has no part to open, and no link opens it; the SVG editor is a tool of its own (its own view model), outside the stack.

SoundSwap (tested): the picture copied, the sound delayed to where the part starts in its file and padded to the file's length, FLAC in Matroska. The view model keeps the editors below as frames (settings, history, facts learnt about files, project id) and brings them back exactly; background work is stopped when the stack moves. A nested editor is never saved apart: the project is saved with every editor open on top of it (ConversionSettings.openAbove, an OpenEditor holding the part, the editor's project and the one on top of it in turn, only in what is saved), and when the editors have to be left without being applied (another project opening, another window taking this one, the app going) they are kept so (keptStack); opened again, the project opens them again, edits and all (reopenStack; one whose part is gone stops there). The editors show a first, different tool on a selected part (OpenInEditorButton; in the photo editor the picture itself too, with nothing selected), which opens a panel like any tool (EditorTool.OPEN_IN, ImageTool.OPEN_IN, OpenInPanel); Apply in place of Export; the way through the stack as badges in the top bar under the file's name (StackPath, the close button then going back down), which opens the map of the editors, a tidy tree drawn top down, editors only, the part on each link with what it is, a clip, a layer or a sound (StackGraphSheet on core/stack/TreeLayout, built by ConversionViewModel.stackGraph, a node tapped leading to its editor by navigateStack and StackGraph.route); each editor a shape and a colour (the project a four-sided cookie, video nine-sided, audio a soft burst, photo a flower, GIF a sun); the render's progress (NestedApplyOverlay) and a question before changes are lost (LeaveStackDialog); each level keeps its own UI state (SaveableStateHolder keyed by level, in MainActivity's editor route, around the video, audio, GIF and photo editors alike).

Recent projects

Every opened file is a project. Once something is changed (or exported), ConversionViewModel hands it to ProjectStore (app/.../projects/) 0.8 s after the edits rest, and once more when the project is closed (another file opened, the app left). The home screen shows them as a "Continue editing" carousel (RecentProjects.kt: a Material 3 multi-browse carousel, cards that round into pills as they shrink; a tap opens the editor where it was left, a long press opens a sheet to open, duplicate, share the last export or delete). openProject restores the files, edits and format; the undo history is not kept.

Privacy first: everything is in files/projects/<id>/, private to the app, never backed up (allowBackup="false") and never sent anywhere (the app has no network permission). A project folder holds:

  • project.json: the editor's state as plain, indented, ASCII JSON with the code's own field names and a header saying what the file is (ProjectJson; the generic JSON reader and writer and the :core models' mapping are core/project/Json.kt and ModelJson.kt, no library). Readers fall back on each model's defaults, so files of other versions still open;
  • thumb.jpg: the first frame as it shows (mirrored, turned, cropped), made again only when that changes;
  • media/: copies of what could not be opened again later: camera recordings, stills and covers (made in the cache, which is emptied) and files another app only lent (a file opened from a share). Files picked with the system picker are not copied: the store keeps their persistable read permission and gives it back when no project (and not the open editor) uses them any more. Files no longer used are removed when the project is closed (until then an undo may bring them back).

All the store's work runs in order on one background worker, so a save, a deletion and the cache clean-up after a project is closed never overlap (the cache files of a closed project are deleted only after its last save copied them). Only the newest projectsLimit projects are kept; a project the user deleted is never brought back by a late save. The settings page lists every project with its size, missing files and copies, and deletes one or all.

Frame by frame

Beside play / pause two arrows move the playhead one frame back or forward (FrameStepButtons.kt), for cutting exactly where a finger on the timeline would glide past; held, they step again and again, faster the longer they are. FrameStep (core/preview, unit tested) counts the frames in each clip's own file at its own rate (MediaInfo.frameRate, 30 a second when not known): a step lands in the middle of the part of the next (or previous) frame the clip shows, so that the preview's exact seek shows that very frame; past a clip's end it goes on to the first frame of the next clip, before its start to the last frame of the one before, and stops at either end of the video. A tap is an exact seek; the steps of a hold are scrubs (see "Live preview"), so the picture follows them frame by frame, and the frame it stops on is then shown exactly.

The row under the preview never lets two controls overlap (ControlRow in ConfigScreen.kt): the right group (the curve button of a keyframe, which grows out of the diamond, the diamond, undo, redo) is laid first, play / pause with its arrows stays in the middle while it fits and moves left when it does not, and the clock takes what is left: on one line, on two (the time above the length) or the time alone (AdaptiveClock).

Live preview

The editor shows every setting before anything is rendered. No FFmpeg run is involved: EditorPreview.kt nests the same layers as the export filter graph (the clip on screen with its own geometry, colour, captions, fade and pictures, fitted into the project frame; then the whole video's geometry, captions, subtitles, fade, pictures and canvas). preview/EditorPreviewPlayer.kt drives a player for the picture and source audio and one per placed sound (preview/SoundVoices.kt; a sound's player exists only from a few seconds before it starts to shortly after it ends, so only overlapping sounds hold a decoder at the same time). A sound's speed and pitch are the player's PlaybackParameters (set apart, as in the export), its repeats the player's repeat mode, its equaliser the export's own filters run in an AudioProcessor of its player (see "Audio files"), and its levelling a gain from the file's measured loudness; while playing, the output clock is the master and each clip is driven to where it should be, at its own rate or stepped backwards. The math that makes the preview agree with the export lives in :core, package preview, and is unit tested:

Setting How it is previewed
clips, cuts, whole-video trim clipped playlist; one output clock across all clips (PreviewTimeline)
stills (pictures, freeze frames) timed image items keep the clock; the picture is drawn over the video surface with the clip's colour and geometry
rotation, flip, crop, canvas nested view transforms and layout, clip inside whole (PreviewGeometry); on a canvas of another shape the picture laid whole, covering it or stretched to it (fittedOnCanvas, the export's CanvasFit)
brightness/contrast/saturation colour matrix equal to FFmpeg eq; clip and whole matrices combined
speed (clip and whole) player rate changes from clip to clip, pitch preserved like atempo
reverse (clip and whole) the clip's part reversed by FFmpeg, picture and sound, played forwards in its place (see below); until it is made the paused player is stepped backwards, silent
fades (clip and whole) black layers + audio volume, same ramps as fade/afade (fadeLevel)
text, image overlays Compose layers using the export's position and size rules, shown during their windows of the output clock with the same opacity and fade ramps; an overlay's own colour matrix, turn (a layout that reports the turned box, as the export places it) and mirrors
video overlays a silent player per video on a TextureView, driven to the output clock every 50 ms (loaded 1 s before it shows); its sound plays as a placed sound
still motion (Ken Burns) the still is scaled and shifted with the clip's progress, the same curve as zoompan
subtitles cues parsed by SubtitleParser, moved with their footage like the export; styled ones (spoken, or .srt) drawn as libass will (see "Smart subtitles")
volume above 100 % LoudnessEnhancer on the player's audio session
loudness normalisation source loudness is measured once (ebur128), then a gain to the loudnorm target
placed sounds a player per sound, kept in sync with the output clock (place, trim, volume, fades, speed, pitch, repeats, equaliser, levelling)
sound effects (final sound, each placed sound) sample-exact ports of the FFmpeg filters in an AudioProcessor of every player; pitch as the players' pitch; the waveform view drawn from the output's outline
stream copy, frame extraction filters are ignored, exactly as the export ignores them
denoise (clip and whole) not live, as it works over time: a badge on the picture and a mark on the clip say it is there, and the Details panel shows it close up, made by FFmpeg with the export's own filter (see "Denoise")

Video in a frame of another shape (issue #34; CanvasFit in ConversionRequest.canvasFit, ConversionSettings.projectFit, saved with the project; VideoFilters.canvasFilter, tested in CanvasFitTest). A canvas ratio grows the frame to its shape (one side kept, the other made larger, even sides) and the picture meets it in one of three ways: whole, with black bars (pad, as before); covering it, enlarged and its edges cut (scale with force_original_aspect_ratio=increase, then a centred crop); or stretched to it, nothing cut and nothing around it (scale to both sides, setsar=1). The frame is the same size each way, so the pictures and captions on it land alike (wholeFrameSize). Placed by hand (VideoPlacement), the picture lies on the black canvas as the fit lays it, times its scale. The preview lays it out the same way (fittedOnCanvas, fittedSize; a covering one goes past the canvas, which clips it), and the snapping guides use that box. The Canvas tool offers the three as tiles drawn as they look (CanvasFitChooser).

Stretched pictures (ImagePlacement.stretch, ImageOverlay.stretch). Any picture laid over (a sticker, a picture on the video, the photo on its own canvas) can be pulled out of its shape: laid widthFraction wide and stretch times less high than its shape (overlayScale: ow/dar/stretch, then setsar=1), still or moving; its look is drawn on it once stretched (the preview's made look too, decoratedPicture(stretch)), and a turned one is placed by its stretched box (laidPictureAspect). The ways to fit it have the same names and drawings everywhere: Whole, Cover and Stretch, as tiles drawn by CanvasFitPreview (the video in its frame, the photo on its canvas, a sticker or picture in its Arrange card), the one it lies as lit (photoFit, ImageOverlay.fitIn); the Arrange card adds Full width and Centre, which only place it. Whole, Cover and Stretch put a picture back upright in the middle (fittedTo: FitMode.STRETCH pulls it to the frame's shape, Whole and Cover give it its own shape back; Full width and Centre keep its shape and turn). The photo editor's canvas puts the photo back the same three ways (fitPictureOnCanvas(CanvasFit), stretchedPhoto); a photo stretched to its canvas is stretched again to the canvas's new shape when it changes. By hand, a selected picture (the photo too) has a bar in the middle of its right side and of its bottom (StretchHandle): the side one makes it wider or narrower and keeps its height (width and stretch grow by the same factor), the bottom one taller or shorter and keeps its width (only the stretch), both as the export lays them (its look's room counted, within MIN_STRETCH..MAX_STRETCH and the widths laid) and snapping to its own shape within 4%, with a tick (core/model/PictureStretch, tested). The finger is followed in the root's coordinates and projected on the turned box's axes (PictureStretch.along), since the bar moves with the edge it pulls; the corner grip and two fingers keep its shape. While it is out of its shape (laidStretch != 1), its Arrange card and the photo's canvas panel say so, with "Own shape" (StretchedNotice: the stretch back to 1, where it is and as wide).

Backwards clips (core/preview/ReversedSound, tested; media/FfmpegSoundReverser; ReverseSoundWindow.kt). A player only goes forwards: stepped back frame by frame the picture showed about ten frames a second, and sound cannot be stepped at all. So FFmpeg writes the clip's part of its file reversed into a file of the cache named after that part (the same part is found again without FFmpeg; files not played for three days go at start-up). reverse and areverse keep all they reverse in memory: the picture is made small (480 on its short side, 30 frames a second, tone mapped when HDR) and reversed in pieces of three seconds, the last first, joined as they are; the sound is reversed in one go (no seam) and laid under it. Every piece is cut before the input, so that only it is decoded. The preview asks for the parts its backwards clips need (reversedSoundRequests: a clip's own reversal and the whole video's cancel out), the view model makes them one after the other (wantReversedSounds), and the player plays each forwards in its clip's place (EditorPreviewPlayer.reversePlayer, driveReversed: at the clip's rate, gain and fades; its picture over the clip's own with the same look, PreviewSnapshot.reversedPicture; while it plays in step the output clock follows it and the footage is not stepped). Nothing is added to the project, the opened file is never touched and the export reverses the clip itself. Meanwhile a window shows the work done, with a stop (closing it stops too); a part stopped or failed is not asked again until its clip stops going backwards, is stepped back in silence meanwhile, and its sound is drawn faint on the timeline (quietClips). When only the picture fails (its steps are logged under SoundReverser), the reversed sound is still kept, as a sound-only file: the player hears it and steps the picture back itself (reverseShowsPicture holds only for an .mp4).

Clip and whole speed together (ClipPace, ClipPaces in EditorPanels.kt). As the volume tool lists the clips under the original track, the speed tool of the whole video lists the clips that play: each at its own speed times the whole video's (a curve's average), backwards when only one of the two is reversed, with a switch that sets the clip's own direction so that it ends up the one asked; a tap opens the clip's own speed. There, a hint says how the clip plays in the end when the whole video changes it.

Sped-up sound (issue #35). The preview's players have Android change the speed and pitch (setEnableAudioOutputPlaybackParameters, in processingPlayer: AudioTrack's playback parameters, done in the audio server) instead of Media3's Sonic, which works on the player's own thread: on a Pixel 9 Pro a 4x MP3 came out at 0.6x with Sonic, crackling, even on a bare Media3 player, and at full speed without underruns with Android's, the decoder alone managing 11.6x (measured on the debug dashboard). Android changes the speed up to 6x (faster clips are stepped, as backwards ones) and the pitch within 0.25 to 4 (androidPitch). Media3 makes the speaker's buffer long enough for 8x, two seconds at normal speed: a change of the sound effects is heard that much later while playing. Made for 3x instead, a 4x preview still ran dry. The editor's sound effects (AudioEffectsProcessor) read and write their samples in bulk, and let the sound through untouched when there is no effect: one sample at a time through the buffer, sped up four times, was too slow on the player's thread.

Debug dashboard (debug/DebugDashboard.kt, debug/DebugBench.kt). Only a build that can be debugged has it: a bubble floating over every screen, dragged anywhere, that opens into a small card. It picks the preview's sound path for issue #35 (the usual one, or Media3's Sonic as before the fix), made again with the editor, and counts what the main player went through (speaker underruns, dropped frames, seeks asked by the output clock, its rate). The same build logs Media3's events under "PreviewPlayer". Release builds take the usual path and count nothing. Its tests play the editor's first file alone for ten seconds at 4x (on a bare Media3 player, with Android's speed, or as the preview builds its players) or decode its sound as fast as the decoder goes, and report the speed reached.

Dragging the timeline asks the preview for a frame about 20 times a second. Plain exact seeks would each start over before the one before had shown anything, so the picture stood still until the finger stopped; instead the timeline's seeks are scrubs (PreviewSeek.scrubbing, EditorPreviewPlayer.scrubMediaFrame): the main player goes into Media3's scrubbing mode (ExoPlayer.setScrubbingModeEnabled, Media3 1.8 and later), which shows each frame as soon as it can, and leaves it once no frame has been asked for 180 ms, showing the last one exactly. A tap, a frame step or a seek from the media notification stays a plain exact seek.

From one clip to the next the main player goes on along its playlist without a seek. A clip rarely starts on a key frame, so its first frames have to be decoded from the key frame before it; on one decoder that held the picture for a moment at every cut. The main player is built with Media3's video renderer prewarming (processingPlayer(..., prewarm = true), DefaultRenderersFactory.experimentalSetEnableMediaCodecVideoRendererPrewarming): a second decoder gets the next item ready while the current one plays, and the two take turns. The other players (placed sounds, the clip coming in through a transition) play one item and do without.

At the end the video starts over, like a player on repeat, but the main player does not repeat on its own (REPEAT_MODE_OFF): the output clock wraps and seeks it to the start at once. On repeat, the prewarming decoder got the first clip ready as the "next" item and the two decoders swapped on the surface at every loop; after a few loops the picture stayed on the last frame with no error (issue #32: the decoder reconnected to the surface, showed a few frames and stopped). Stopped at the end of its last item, perhaps a little before the clock (whose lengths are the editor's), the player is not followed: the clock runs on to the end, the last frame held, and wraps. A player that fails (onPlayerError) would stop on its last frame for good, since seeking does nothing until it is prepared again: it is prepared again where the clock is, up to three times within ten seconds, and the error goes to Logcat (tag PreviewPlayer).

The editor's timeline is one Canvas drawn again on every frame while the preview plays (the playhead stays in the middle and everything moves under it), so its drawing keeps the work per frame small: the bars of every waveform (the clips' own sound, an audio file's clips, the placed sounds) are collected in a BarBatch and drawn with a single drawLines call instead of a rounded rectangle each, and stay anchored to their clip as it scrolls (one every few pixels from the clip's left end, not from the screen's edge, so they no longer shimmer); the dark band under a clip's sound reuses one gradient (BandShade); the ruler writes its times without String.format.

What a live preview cannot be: reversed audio is not audible (muted), loudness normalisation is a constant gain rather than loudnorm's dynamic processing, and the styling of .ass files is not reproduced (libass does that at export; the app's own styles, for spoken subtitles and .srt files, are). The final sound's effects run on each player apart, so where sounds overlap the gate, compressor, limiter and lo-fi (which are not linear) act on each sound rather than on their mix; with the speed changed, echoes and modulation run at the footage's own pace. The audio encoding (bitrate, bit depth, sample rate) is not heard, like the picture's encoder settings. Resolution, frame rate and compression are encoder settings chosen in the export dialog; they do not change the picture shown in the editor.

Text

All user-visible text is in app/src/main/res/values/strings.xml (English). The :core enums carry no labels; app/.../Labels.kt maps them to string resources, and conversion/ConversionMessages.kt does the same for engine status and errors.

Colour key (green screen)

A colour can be cut out of a clip, of the whole video, of a picture or video laid over the video, of a photo and of its stickers: ColorKey (core/model/ColorKey.kt) in Effects.key, ImagePlacement.key and ImageEdit.key. It cuts by hue (chromakey on yuva444p: a green or blue screen whatever its light) or by the exact colour (colorkey on rgba), with a tolerance, a soft edge and spill removal (despill, on green and blue screens only); see core/command/KeyFilters.kt. Where it goes:

  • a clip or the whole video: after its colours, before captions; what is cut shows the key's background (black without one: a video file holds no transparency). In a filter chain that is a little graph of its own, the keyed picture split and laid over a copy filled with that colour (drawbox ... t=fill), its links named after the clip so they stay unique;
  • a picture or video laid over the video: after its own colours and mirrors, before it is sized and turned; what is cut lets the video show through. Its own transparent parts (a PNG sticker) stay so: FFmpeg's keys write the alpha plane, so keyKeepingAlpha keys a copy and multiplies the two alpha planes (alphaextract, blend=all_mode=multiply, alphamerge), and the preview multiplies them the same way (ColorKeying.apply(keepAlpha = true), the integer a * b / 255 of blend);
  • a photo: last of its steps (ImageStage.Key), before texts and stickers. The photo's steps run on rgb24, so they read a transparent picture opaque (Raster.opaque(), keepAlpha = false). A picture with an alpha channel of its own (MediaInfo.hasAlpha, told by the probe from a small decoded copy) saved as PNG, WEBP or TIFF keeps its transparency all the same (ConversionRequest.keepsPictureAlpha): the export splits its alpha off at the start (alphaextract), puts it through the same geometry as the picture (mirrors, turn, keystone, straighten, crop) and multiplies it back into the alpha the steps left (a key's, 255 without one) at the end (blend=all_mode=multiply, alphamerge); the editor does the same in Kotlin (alphaOf, shapeAlpha, mergeAlpha in core/image/ImageEditing.kt, in PictureRenderer) and shows the checkerboard through it. Saved as JPG or BMP the picture stays opaque, as before. The stickers laid on it keep their transparency as overlays do. PNG, WEBP and TIFF keep the cut transparent (-pix_fmt rgba, overlay with format=auto so the stickers keep it), or show a fill colour at an opacity (ColorKey.backgroundOpacity, keyOverTranslucentBackground: the keyed picture laid with overlay on packed RGB over a copy turned into the fill by lutrgb with constants, rgba); JPG and BMP show white (ImageEdit.keyFor) or the chosen colour, always opaque, in full colour (yuv444p).

ColorKeying (core/preview) does the same in Kotlin: the key colour's chroma with chromakey's own fixed-point formula, each pixel's in TV range, the 3 x 3 average (neighbours outside the picture keep the value their slot last had, as FFmpeg's loop does), the opacity ramp truncated as FFmpeg does, and despill's float round trip; a see-through fill with overlay's own integer math (overlayStraight: the overlay's alpha un-premultiplied against the main picture's, FAST_DIV255, the alphas composited). Against FFmpeg 6.1 on a noisy green screen the alpha is exact on 99.5 % of the pixels (at most 3 off), the colours within 2 (FFmpeg goes through YUV and back); over a see-through fill the premultiplied colours are within 3, and exact for an exact-colour key. The previews use it:

  • the photo editor runs it as one of the steps of its picture, and shows a checkerboard where the picture is transparent;
  • pictures (stills, stickers, pictures laid over the video) are keyed once on the CPU, after their own colours;
  • video, on Android 13 and later, is keyed on the GPU: a RuntimeShader (AGSL) with the same math as a RenderEffect of the layer (preview/KeyEffect.kt, ColorKeyLayer.kt), on the clip's layer, on the joined video's (whole video), on a video laid over the picture (then the colours of what it lies on as a colour filter after it). The shader keeps to the plainest AGSL (no loops, no ternaries, no branches on uniforms; checked with Skia against the Kotlin port) and is compiled once behind a guard (KeyShader.compiles): a device whose compiler refuses it gets the CPU path below instead of a crash. The uniforms are named by constants used both in the shader's text and where they are set: setting a uniform the shader lacks throws, and Android 13 builds that exception's message from garbage, which aborts the app under CheckJNI ("input is not valid Modified UTF-8") instead of throwing - the crash a misspelt name once caused as soon as a video was keyed. Before Android 13, with a refused shader, and under a GIF's frames, the frames are grabbed from the surface and keyed on the CPU, at up to 480 px and 24 frames a second (ProcessedFrames, which also makes the GIF frames; the pixels are read and keyed off the main thread).

The key panel (KeyPanels.kt) shows a sample of what is keyed as it looks before the key (the frame under the playhead with its colours, or the picture laid over the video), with the result on a checkerboard:

  • switched on, the key starts from the colour the sample suggests (ColorKeying.suggestedKey: the commonest colour of the sample's border band, in 4-bit bins; a clearly green or blue one is cut by hue, anything else exactly); the Detect button does it again;
  • touching or dragging on the sample picks the colour under the finger, live, with a loupe above the finger that magnifies the pixels there (sharp, the one taken framed) and is ringed in the colour taken; a held compare button shows the original; another takes the frame under the playhead again;
  • the colour taken is always shown with its code and mode; quick colours (green screen, blue screen, white, black) set the mode that suits them;
  • then the mode, the edges (tolerance, soft edge, spill where it applies) and the fill: transparent (the default where the output keeps transparency) or a colour, with its opacity where the result can hold it (a photo saved as PNG, WEBP or TIFF); a line says what the output does with it.

Blend modes

A picture, a video or a shape laid over the picture or the video can mix with what lies under it instead of covering it (ImagePlacement.blend, LayerBlend in core/model): multiply, screen, overlay, darken, lighten, colour dodge and burn, hard light, difference, exclusion, add. They are the separable modes of the W3C compositing spec, which Android draws too.

Export. layOver (CommandBuilder.kt) lays a NORMAL picture with overlay as before. A blended one is first laid on a see-through copy of the frame (colorchannelmixer=aa=0, overlay=...:format=rgb with its own place, time window, turn and fades, so it moves exactly as it would); the two are mixed by blend=all_mode=... in planar RGB (gbrap), the picture as the top input, and the mix is kept only where the picture is (alphaextract, alphamerge, then overlay on the frame): the frame times (1 - alpha) plus the mix times alpha, which is how Android composites a layer with a blend mode. FFmpeg's hardlight tests its bottom input, so it is the spec's overlay, and its overlay is the spec's hard light (LayerBlend.ffmpegMode). Soft light is left out: FFmpeg's formula is not the spec's.

Preview. The picture's layer in ImageOverlaysLayer gets the mode as its blendMode (composeMode(), BlendPanels.kt), drawn over the photo or the video's TextureView.

Captions blend too, turn and stretch (TextOverlay.blend, rotationDegrees, stretch). A caption turned, blended or stretched (TextOverlay.layered) is drawn as a layer of its own (VideoFilters.layeredText): on a see-through canvas twice the frame's size, in the letters' colour (so the turn's and the stretch's smoothing bring in no other), its anchor in the middle (the point of the text at (x, y) of its size, which lies at (x, y) of the frame: drawtext places it so, whatever its width); stretched there (laidStretch, as a picture's, within 0.25..4: wider, the canvas is made that many times narrower first, twice the frame's height, and widened back with the letters on it, so that it is never wider than twice the frame; narrower, it is narrowed after them; setsar=1 each time), then rotate about that middle; laid back with the anchor at its place, plainly or by the same blend as a picture. The preview stretches and turns it about the same point (scaleX, rotationZ, transformOrigin: scale first, as the export) and mixes it with its blendMode (offscreen); its grips are drawn back round in the stretched layer (squash). Upright, normal and in its own shape, a caption is drawn straight on the frame as before. By hand a selected caption has the bars of a picture (StretchHandle): the side one makes it wider at its size (PictureStretch.textSideways), the bottom one taller at its width (textUpDown: the size grows, the stretch shrinks as much), snapping to its own shape within 4%; its Layout panel offers "Own shape" while it is stretched. Kept in the project only when not 1 (ModelJson, "stretch").

Panel. ImageBlendPanel: the chosen mode large, in a badge that morphs into the mode's own shape, and every mode on a tile that shows itself (a warm-to-cold backdrop with a disc mixed onto it by that mode). In the picture editor it is a tool of a sticker, a shape or a caption, in the video editor of a picture, a video, a shape or a caption laid over it.

Magnetic guides (SnapGuides.kt; the math is core/preview/Snapping, tested). On one frame (a clip's, or the whole video's canvas) everything that can be moved by hand registers its box in a SnapScene: captions (as drawn, stretched), pictures, stickers, shapes and videos laid over (turned, their look's room included), the video itself on its canvas, and the styled subtitles (their lines as drawn). Moved, a thing's edges and middle stick to the frame's and to the others' (snapMove, within 8 dp); resized or stretched (two fingers, the corner grip, the side bars, the subtitles' corner), its moving edges do (snapResize: the box is affine in the one number the gesture changes, a size, a scale or how far a bar was pulled, so the number that lays an edge on a line is solved for exactly; its middle does not count, a move snaps that). A pulled bar near the thing's own shape takes that first. The lines show while it sticks, a tick each time it catches, and go when the finger lifts. The subtitles once only held to the middle; with no scene they still do.

Turning by hand. What is selected on the picture (a caption, a picture, a video) has a knob beyond its top (RotateHandle): dragged, it turns the thing to face the finger, about its middle (a caption: its anchor), caught by whole steps of 45 degrees with a tick (SnapScene.turn), the angle in a pill meanwhile; a double tap stands it upright. Where the knob would leave the frame it sits below. Two fingers turn a caption as they do a picture, and the Rotate tool has the angle on a slider (a caption has no flips). A turned (or stretched) thing feels its touches in its own coordinates, turned with it: a drag moves it by the finger's move turned and stretched back onto the canvas (detectTotalTransform's toCanvas, turnedMove), so it follows the finger whatever its turn, in the video and photo editors alike.

Video effects

Sparks, fireworks, flames, smoke, snow, rain, confetti, dust in the light, bokeh, hearts and math loop over the video for as long as they are laid; the fairy's wand, an explosion and a flash are transitions laid over the cut between two clips. None comes from elsewhere: scripts/generate-effects.py draws every frame (numpy, scipy, fixed seeds) and encodes them with FFmpeg into app/src/main/assets/effects/ (H.264, 1280x720, 30 fps), with index.json saying what each is (kind loop or transition, frames, fps, for a transition the second of the cut, under its brightest frame, and its sheet: a JPEG of small frames, 160x90 at 15 fps, eight a row). The script is only run to make them again (--index-only makes the sheets and the index from the videos there); the build uses the files.

Math. scripts/math_effect.py draws the formulas, question marks and little diagrams (a right triangle, a sine wave, a circle and its radius, a parabola, a turning wire cube) of the confused-maths meme with its own stroke font (each glyph a few lines and arcs, no typeface): each item is written on stroke by stroke with a bright chalk tip, drifts, turns and comes closer, and fades; two share each cell of a 6x4 grid, one showing while the other is away, so the frame never clumps. Points along the strokes are splatted into Canvas (sharp near, softer far, with a glow), as the other effects.

Light on black. Each effect is drawn as light on black and laid with the screen blend (LayerBlend.SCREEN): black disappears and the light adds up, so no transparency is needed and every phone decodes the files. A looping effect loops seamlessly: every particle has a phase in the loop, falling things cross a whole number of frame heights per loop, and the noise of flames and smoke is periodic in time.

Library. EffectLibrary (app/.../effects/) reads the index when the app starts and copies each file into files/effects/ (players and FFmpeg read a file, not an asset), again when a new version brings a different one. Projects keep these copies' URIs as they are (ProjectStore.keep, EffectLibrary.owns). The gallery, the effect's panels, the layers list and the transition tiles never decode a video: they draw the effect's sheet (EffectLibrary.sheet, read once, RGB 565), one frame after the other at the effect's own speed, in the drawing phase only (EffectFrames, drawSheetFrame), so they show at once.

An effect laid over the video is an ImageOverlay with effect (its name) and hue (degrees), full frame, screened, on the effects lane of the timeline (LaneKind.EFFECT, the first lane, as effects lie on top) and above everything placed later (effectLayer leaves a thousand layers under it). Its tools are which effect (EffectChoiceContent), its look (strength and colour, EffectLookContent), fades and timing; it is added from the gallery (EffectPickerSheet), every tile playing its own frames, opened from the tool row ("Effects") or from the effects lane.

Export. toPlacement makes it an ImagePlacement with cover, hueDegrees and OverlayVideo(loop = true): the input is read with -stream_loop -1 (and -t for as long as it shows), moved to its start (setpts), turned round the colour wheel (hue=h=), scaled to fill the frame and cut (coverScale, from wholeFrameSize), then laid by layOver with its opacity and fades.

Preview. EffectsLayer (EffectOverlays.kt), inside StackedOverlays in the stacking order: a silent ExoPlayer per effect on screen, repeating, kept at (now - start) modulo the loop's length; its TextureView fills the frame as the export's cover does, with the hue as a colour matrix (hueColorMatrix in core/preview, the chroma of BT.601 YUV turned as hue turns it) and BlendMode.Screen.

Transitions. A Transition with an effect joins its clips with a cut (transitionLengths gives it no overlap, so the preview and the export cut there). transitionEffects (app/.../effects/TransitionEffects.kt) makes, for each, an overlay of the whole video playing the effect once, its cut on the cut (OutputClock.cutOutputMs): footage() in RequestMapping adds them to the pictures the export lays, and the preview adds them to the pictures of the whole video, so the two always agree and the editor never keeps them. The transitions panel shows them as a "Magic" family, each tile cutting between its two pictures under the effect's own frames.

Punches: an effect with its own opacity. The 3D fists of TransitionKind.PUNCH and PUNCH_BACK are not light on black: a fist hides what is behind it. scripts/punch3d.py draws them (a right fist built of rounded shapes blended into one signed distance field, ray marched in numpy, lit by a key, a fill and a rim light, with soft shadows, ambient occlusion, skin that lets light through near its edges, ACES tone mapping, six-sample motion blur and the fist's soft shadow on the picture), as colour plus opacity. generate-effects.py packs each frame as its colour over its opacity (grey) in a plain H.264 MP4 twice as tall (1280x1440), so every phone and FFmpeg read it; index.json marks them "kind": "companion" and "alpha": true, with a sheet of 512x288 frames with transparency (WebP, six a row, 30 fps). They are not offered among the magic transitions: each belongs to its kind (TransitionKind.companionEffect). A punch is a recipe like the others (the clip leans into the blow, the new clip springs back from it, or the clip shakes as if struck) and lasts PUNCH_MS whatever its length was (Transition.requestedMs, TransitionKind.fixedMs); transitionEffects lays its fist over the stretch the clips overlap (OutputClock.transitionOutputRange), the fist's cut on the moment the recipe swaps the clips (companionSwitch), as large as Transition.size (30 to 100 percent, the panel's "Fist size", kept in the project). The export reads the two halves apart (ImagePlacement.stackedAlpha: split, crop each half, the lower half's luma taken out as it is with extractplanes and stretched from 16..235 by lut, alphamerge; a format=gray there would turn the whole split grey) and lays the fist with the normal blend, scaled to coverSize of the cover frame, in the middle; a smaller fist fades out towards the edges of its frame (COVER_FEATHER of its shorter side, a geq on its opacity), so that its arm and the flash are never cut sharp. The preview draws the fist from its sheet, the frame of the file at the output clock (AlphaEffectFrames), so it never drifts from the clips; the tiles draw it over their two pictures (TileCompanion). Below Android 13 the clips cross-fade under the fist.

Denoise

ColorSettings.denoise (0..DENOISE_MAX, 10) is FFmpeg's hqdn3d with that as its spatial luma strength; the other three strengths follow from it as FFmpeg works them out. A clip has its own, first in its chain, on untouched frames; the whole video has one too, once the clips are joined (see VideoFilters.kt). denoiseFilter in VideoFilters.kt writes the filter for both, and for the comparison below, so all three run the very same filter.

The live preview does not show it: hqdn3d also works over time, each frame blended with the ones before it, so no frame alone shows what it does. The Details tool (app/.../DenoisePanels.kt, a clip's or the whole video's, named as the photo editor's, where sharpen, blur, vignette and grain will join it) shows it where it matters, close up: a square of the frame under the playhead, at the footage's own pixels, before and after, side by side.

core/preview/DenoiseSample.kt builds the command, run by media/.../FfmpegDenoiseSampler.kt (the DenoiseSampler port) into the cache's denoise-samples/, and read and deleted by the panel:

  • one decode and one split: the two halves come from the same frame and leave as one PNG (hstack), so they cannot be two different frames (two outputs of one run, each with -frames:v 1, were tried first: one of them did not match the export);
  • over time: hqdn3d starts again at each clip's start and forgets older frames quickly, so the run seeks to WARM_UP_MS (500 ms) before the frame, or to the clip's start if nearer, and trim drops those frames only after the denoise has seen them;
  • across the frame: its smoothing runs along rows and columns and fades within a few pixels, so the square (SIDE_PX, 512) is cut out with MARGIN_PX (32) more on every side, denoised, and the margin cut away; at the frame's own edge the edge is the frame's, as in the export;
  • HDR footage that the export makes SDR is tone mapped first (toneMapFilters), as there; footage kept HDR is denoised as it is and tone mapped only to be shown, each half before hstack (which drops the colour tags zscale reads).

Checked with FFmpeg 6.1 on a noisy test video: the "after" half of a clip's denoise matched, pixel for pixel, the same square of a run over the whole video. The whole video's denoise is compared on the clip's own pixels, while the export runs it on the joined (and sized) video, so it is close rather than exact; the panel says so.

The panel, in the PICTURE family's tone: a connected button group of levels (off, light 2, medium 4, strong 7) and a slider for the values in between (tenths, as the command writes them), a warning from 6 on (detail goes too). The comparison sits in a nine-sided cookie that swells into a soft burst while FFmpeg works and settles back on a spring with the new comparison, which a divider splits: dragged or tapped, read out by TalkBack as a value, it sweeps from the right to the middle the first time. Under it, the whole frame, a frame on the part compared that springs to wherever it is tapped, or is dragged after a long press, so that a plain swipe still scrolls the panel (DenoiseSample.shownArea and focusFor, the same math as the command). It asks FFmpeg again 350 ms after the strength, the part compared or the frame stop changing, and only while the playhead is still. Undo keeps the denoise apart from the colours (its own step key, videoDenoise).

Since the preview does not show it, two signs say where it runs:

  • on the picture, among the preview's status badges (StatusBadges in EditorPreview.kt), DenoiseBadge: a 24 dp cookie of the same dark glass with the grain in it, while the clip under the playhead (or the whole video) has a denoise and filters run (not with stream copy or frames taken out). It springs in from a circle, says "Denoise 4.0, in the export" for a moment (both strengths, "2.0 + 3.0", when the clip and the whole video each have one), then folds back to the cookie; it fades while the video plays. A tap opens the Details tool for that clip, or for the whole video when the denoise is its. What it shows is a derivedStateOf of the clock: it changes with the clip, not on every frame;
  • on the timeline, the clip's corner carries the same cookie, small (16 dp, in the picture's tone), in place of the dot of clips with effects; the dot moves beside it when the clip has other effects too. A clip narrower than 40 dp keeps the dot alone.

Sharpness, blur, vignette and grain

The Details tool holds five settings, each a round chip in a row (VideoDetailsPanels.kt, ControlChip: the chosen one a cookie, a dot on those in use): the denoise above, then the finishing touches of ColorSettings, which the export writes after the colours, in this order (VideoFilters): sharpness (-1..1, unsharp=5:5:amount, amount x1.5; below zero it softens), gaussianBlur (gblur, its sigma in the footage's pixels, up to BLUR_MAX 20), vignette (0..1, vignette=angle=, the strength times VIGNETTE_MAX_ANGLE, PI/2.5, as the photo editor's; 0.5 is FFmpeg's default, and a project of an older version that kept it as on or off reads on as 0.5, ModelJson) and filmGrain (noise=alls=N:allf=t+u, up to GRAIN_MAX 60). Each has levels in a connected button group, a slider for the values in between (each its own undo grouping, effectsStepKey), and a swatch: a small scene in the theme's colours (sky, sun, hills) with the setting laid on it, beside a badge whose shape says it too (puffy to spiky for the sharpness).

The preview draws them live (FinishingTouchesPreview.kt), on a clip's picture before its key (ClipPicture) and on the whole video's frame over its clips (EditorPreview), from the same numbers (core/preview/ FinishingTouches, tested): on Android 13 and later two AGSL shaders, one for unsharp's 5x5 luma kernel (one footage pixel a tap apart on screen), one after the blur for vignette's falloff (cos(angle x d / dmax)^4) and noise's uniform draw on each component, a new draw for each frame; the blur between them by the platform's (RenderEffect.createBlurEffect, a gaussian, the sigma scaled to the screen). On Android 12 only the blur is a render effect, and the vignette is a radial gradient drawn over the picture from the same falloff (FinishingTouches.vignetteStops); before, the vignette alone. The panel says when the preview cannot show a setting.

Stabilize

EditorTool.STABILIZE (StabilizePanels.kt) steadies the whole video with vid.stab in two passes of the export (ConversionSettings.stabilize, stabilizeShakiness 1..10 for vidstabdetect, stabilizeSmoothing 0..100 frames for vidstabtransform, 0 holding the camera still as on a tripod; CommandBuilder.buildStabilizationAnalysis, needsStabilizationAnalysis). The panel: a hero card whose badge shakes (random nudges) while the video is left shaky and settles on the slow spatial spring, swelling from a soft burst into a cookie, once it is on; a switch; levels (Gentle 3/8, Steady 5/15, Firm 8/30, Tripod 10/0) and the two sliders; what to expect (the preview cannot show it, two passes take longer, the frame is zoomed a little to hide the borders). Not with stream copy or frame extraction. The preview has a badge for it among its status badges (StabilizeBadge, as the denoise's), which opens the tool.

The sound's denoise

Effects.audioDenoise (0..1, FFmpeg's anlmdn at up to strength 10, audioDenoiseFilter in AudioFilters.kt) has a tool of its own, Noise (EditorTool.AUDIO_DENOISE, AudioDenoisePanels.kt), in both editors, for a clip or for the whole video: levels and a slider, and a listening card. The live sound cannot run anlmdn (it compares every short patch of the sound with those around it), so "Listen" makes a comparison where asked: eight seconds of the sound under the playhead (moved back to fit in the clip), decoded once with a quarter-second lead-in, split, one half as it is and the other through the export's filters (the clip's, then the whole video's), the lead-in cut away, both written as WAV (core/preview/ AudioDenoiseSample, tested; AudioDenoiseSampler, implemented by media/FfmpegAudioDenoiseSampler, into the denoise comparisons' cache folder, emptied at start-up). Both halves play together in a loop in two players, only the chosen one heard, so a connected button group switches "Before" and "After" at the same instant; they are put back in step when they drift. The card's badge swells into a soft burst with the loading indicator while FFmpeg works; a change of strength makes the comparison again once the slider rests. The editor's own playback pauses while it plays.

The TV break

TransitionKind.TECHNICAL_DIFFICULTIES is an old TV losing its signal: the channel changes (a click, the picture tears, snow and "CH 03"), a test card ("TECHNICAL DIFFICULTIES / PLEASE STAND BY") plays a little lounge tune, then the channel changes again (snow, "CH 04", the screen coming on from a bright line onto the next clip). It lasts as long as asked, within bounds of its own (minMs 1.6 s, maxMs 15 s, defaultMs 3 s; Transition.effectiveMs takes the kind), and the transitions panel's slider follows them.

Its parts. scripts/tv_break.py draws and composes everything: the card (a test-card field, a plate with the title in its own stroke-font capitals, a little TV set, an old screen's hum bar, flicker, grain, scan lines and curved glass), the channel changes (with opacity, packed as the fists are) and the sound. The tune is composed there (eight bars at 120 beats a minute in F: vibraphone melody, electric piano, walking upright bass, brushes) and played by small synthesizers; every note ringing past the loop's end, and the room's echo, is wrapped round to its beginning, so it loops seamlessly. The library has three companion effects with their own sound ("audio": true): tv_intro, tv_card (16 s, the card's loop and the tune) and tv_outro.

On the video. The recipe is a plain cut halfway, always under the card. transitionEffects lays the three parts over the stretch the clips overlap (tvBreak): the card from TV_BREAK_CARD_IN_MS in to as far before the end (its tune fading in and out under the snow), the channel changes over the two ends (TV_BREAK_PART_MS, shorter on a sped-up video). Each is a video laid over the whole video with hasAudio, so its sound is mixed in the export (overlaySounds) and played in the preview (previewSounds, which now takes the cut effects too). The clips' own sound gives way (TransitionKind.mutesClips): in the export the joined sound fades out over CLIP_SOUND_FADE_MS at the break's start, the next clip's fades in over its end, and acrossfade adds them with nofade curves; the preview's player follows clipSoundShares. In the preview the opaque card plays through its own player with the normal blend (EffectSurface screens only light on black), the channel changes from their frames (AlphaEffectFrames).

Picture looks

Black and white, old film, VHS, glitch, TV static, prism, shake, pulse, rainbow, strobe, pixelate, invert, mirror and glow (PictureLook, core/model/PictureLook.kt) are effects of the library too, offered under their own heading in the gallery: an ImageOverlay whose effect is the look's name, on the effects lane, with its strength (opacity), fades and hue, and two settings of its own: its intensity (lookAmount, 25 to 200 percent: how far a glitch tears, how hard a strobe flashes, how big the pixels...) and its speed (lookSpeed, 0.25x to 4x, for the looks that move: the time it reads runs that much faster, so its loop still closes on a whole turn). They have no file (EffectLibrary.looks; ensure has nothing to copy, the uri is only a name the library owns): instead of being laid over the video they change the picture itself, with its captions and pictures, while they last.

One description, two drawings. core/look/Looks.kt tells each look as FFmpeg filters (lookSpanGraph) and as an AGSL shader (lookShader), from the same numbers. Whatever moves is a function of the time since the look started, modulo its period (PictureLook.periodSeconds), so a look loops seamlessly for as long as it lasts; noise is new on every frame. Random choices (a glitch's bands and bursts, an old film's scratches) come from one hash, fract(sin(0.37 x + 0.3) * 43.758), whose numbers stay small enough for a phone's single-precision floats to agree with FFmpeg's doubles. A time that falls right on a step of the loop gets 0.1 ms added, so that every turn of the loop counts it in the same step.

Export. ConversionRequest.looks (from lookSpans in RequestMapping) are drawn after the captions and pictures and before the whole video's frame rate and fades (videoGraph). Every filter runs on the whole video and is switched on only while the look lasts (timeline enable): nothing is cut out and put back, so no frame waits in a queue for the moment a look starts. The moving parts use expressions of t (eq, hue, overlay, crop); shake and pulse use perspective, which only counts frames, on a branch made steady with fps and laid back with overlay. Bands and scratches are small color sources laid with shortest=1 (they never end). A look below full strength, fading or turned is drawn on a copy, given an alpha (lut, fade) and laid back over the picture.

Preview. pictureLooks (app/.../PictureLooks.kt) puts the looks on screen now on the box that holds the picture and what lies on it, as one RenderEffect: each look's shader from Android 13 (uniforms uSize, uTime, uStrength, uHue, uAmount; uTime already times the speed), chained in order, each shader kept by its overlay and its look (a look swapped for another gets the other's shader); on Android 12 the looks that only change colours (black and white, old film's sepia, invert, rainbow, strobe) as colour matrices; before, nothing. The gallery's tiles and the look panel draw a look on a little sunset of their own (LookFrames). Tests: LookGraphTest (labels, windows, loops, the hash in single precision), LookExportTest.

Transitions told as layers

The first 25 TransitionKinds are FFmpeg's xfade transitions, drawn in the preview by transitionLook (core/preview/TransitionLook.kt). The others are told as layers: a TransitionRecipe (core/transition/) says what shows behind, how each clip's picture is moved, bent, blurred, faded and tinted at each point and moment, and what light is laid over them, as small expressions (Ex, core/transition/Ex.kt) of the point (X, Y), the progress (T) and the frame's size. One recipe gives everything:

  • The preview (TransitionLayers in app/.../TransitionPanels.kt, app/.../preview/TransitionShaders.kt): each clip's layer runs an AGSL shader written from its part (partShader), the light one more (overlayShader), the progress a uniform, so a running transition never recomposes. Android 13 and later; before, or where a device refuses a shader, the layer is cross-faded (transitionLook gives a fade for these kinds) and the panel says so.
  • The export (nativeTransitionJoin, core/transition/TransitionGraph.kt): the kinds whose xfade name is a transition of FFmpeg's own (dissolve, pixelize, the corner and diagonal wipes, cover and reveal up and down, squeezes, closing doors and shutters, soft wipes up and down) are exported by xfade, their recipe drawing the same for the preview. The rest (TransitionKind.isCustom) cannot be one xfade custom expression: worked out pixel by pixel at full size it takes minutes for a second of video. So the joined video is cut where the transition starts and the clip coming in where it ends (split, trim by frame), the two pieces that overlap make a region, and the three parts are joined again by concat (then settb, the clips' clock, which xfade after it needs). In the region each clip is drawn with FFmpeg's own filters, frame by frame: perspective where the recipe gives where the picture's corners go (Part.corners: moved, enlarged, turned, a face of a cube), with a transparent edge when it does not cover the frame; remap through maps of where each pixel comes from, worked out by geq at 192 pixels wide and enlarged (swirl, wave, spin with mirrored edges...); opacity and light as masks worked out by geq at 256 pixels wide (a tiny one when they do not change across the frame), enlarged and merged (alphamerge); eq for saturation and dips toward black or white; gblur, dblur and chromashift with their strength sent for each frame (sendcmd, the values worked out here); mix of copies for the zoom and spin blurs (NativeBlur.Echo); noise for TV static; overlay lays the layers. The region's frames run from T = 0 to T = 1, so it starts and ends on one clip. HDR videos (10 bits, which perspective does not take) get a cross-fade instead.

TransitionRecipeTest checks that every recipe starts on one clip and ends on the other, that its corners agree with where it reads its picture and that its shaders are whole; TransitionGraphTest that every graph makes and uses each of its labels once.

The transitions panel groups the kinds in twelve families (TransitionFamily, as CapCut does), two rows of tiles that scroll sideways, each tile playing its transition through the same layers; the duration slider reaches as far as the two clips allow (half of the shorter), and a button plays the transition in the preview. On the timeline the edges of two clips joined by a transition lean alike across the join (transitionSlants in EditorTimeline.kt), as wide as the footage the transition takes from each, the join's button on the middle of the slant.

Shape layers

A shape (rectangle, ellipse, triangle, diamond, pentagon, hexagon, star, heart, arrow, and Material's sun, cookie, clover, flower and burst) is a sticker whose picture the app draws: ImageOverlay.shape (ShapeLayer, ShapeLayers.kt) says what it is, how it is filled (a colour, a gradient at an angle, or none), its outline, its proportions and how round its corners are, and its uri is a transparent PNG ShapeRenderer draws with the stills (CacheJanitor.newStill, 1600 px on its longer side). So the export, the preview and every tool of a picture (place, size, turn, opacity, blend, the outline and shadow of its look, keyframes, fades, timing, layers) work on it as they are; it has no crop, colours, key or Replace, since it is drawn as asked.

The view model draws the picture again whenever the shape changes (updateShape, syncShapes): one drawing at a time per layer, the newest shape asked for when it is done, so a slider dragged on every frame draws only a few. The picture follows the shape as a change of its own (mutate(record = false)), so undo always finds a shape and the picture drawn for it; a step undone or redone before its picture was drawn has it drawn then. Projects keep the PNGs like the stills, and their shapes trust them when opened. The panel (ShapeLayerPanel) draws the shape live with the very path the PNG is made from (ShapePaths), so it changes under the finger while the picture follows.

Both editors add a shape from the tool row ("Shape") and list it among the layers; in the video editor it is laid at the playhead, like a picture.

Starting from nothing

"Create from scratch" on the home screen, under the recent projects (BlankPanels.kt): a new blank, then the user's own starting points. A blank (BlankSpec, core/model/Blank.kt) is a size (a shape among square, post, story, wide, photo, 4:3, A4, banner, this screen, or any size typed; its longer side 1080, 2160 or 3840 px, sides even, at most 3840 as a still) and a fill (BlankFill: transparent, a colour at any opacity, a linear gradient at an angle across the picture's extent along it, or a radial one from the middle to the corners, each colour with its own opacity). AndroidBlankMaker draws it into a PNG still, row by row as BlankFill.row works it out (the sheet's preview draws the same with Compose brushes), and it opens as a picture named after the blank; its file type is picked afterwards, as for any file. The presets are neither settings nor projects: BlankPresetStore keeps them in a JSON file of their own (BlankPresetsJson), newest first, one per name; a tap makes one at once, a long press edits or forgets it. Beside the new blank, ready ones (transparent, a few colours, a gradient; a square of 1080 px) open at once, or held, in the sheet to change them first.

Every row of colour swatches (ColorSwatchRow) ends with a colour wheel that takes any colour by its hex code (HexColorDialog, parseHexColor); a colour chosen that way shows chosen at the end of the row.

Compressor

A tool of its own from home (CompressScreen): the home's tile asks for the file first (no screen between), and the screen opens on it; it can pick another. The file is opened as usual (setInputUri) and exported at once with ConversionSettings.compression set for that export only (never saved). Compression (core/model) is a level (same quality, balanced, smallest) or a size to fit:

  • a video becomes an MP4 with HEVC by x265 (-tag:v hvc1, +faststart), never the phone's encoder: at CRF 24 / 28 / 32, or, to fit a size, in two passes (buildFirstPass: the same command with -x265-params pass=1:stats=... into -f null, then pass=2 at the bitrate that fills the size, 3 % kept for the container, the AAC sound at most a fifth of it, 64 to 128 kbit/s); a lower resolution is the usual Resolution;
  • a JPG at the level's quality, or fitted by the engine (fitted): made again at half the quality range [FIT_STEPS] times, the best that fits kept;
  • a PNG with -compression_level 9 -pred mixed, or brought to 256 colours (palettegen / paletteuse dithered, transparency kept, pal8, unfiltered rows); a WEBP at the level's quality; a longer side as ImageOutput.maxSide;
  • a sound in its own lossy format (a lossless one becomes M4A) at a lower bitrate (Compression.audioKbps), or the one that fills the size;
  • with "without place and device", the tags go through withoutSensitive.

All of it was checked against FFmpeg 6.1 (x265's two passes, the palette).

Background playback

The preview keeps playing when the app goes to the background or the screen turns off, with the system's media notification (title, picture, play / pause, seek bar), the lock screen controls and headphone buttons:

  • the preview is not one player (see "Live preview"), so a stand-in faces the system: playback/PreviewSessionPlayer.kt, a Media3 SimpleBasePlayer with one item as long as the exported file. Its position is read from the preview's output clock whenever it is asked for; a jump (a seek in the editor) and a new length are reported as they happen. Play, pause and seeks from the notification go back to the editor's play state and to EditorPreviewPlayer.seekToOutput;
  • PreviewPlaybackService (a Media3 MediaSessionService, foreground type mediaPlayback) holds the session. PreviewSessions starts it when an editor opens (the editor is on screen then, as the system requires) and stops it when the editor closes or the app is swiped away; Media3 brings it to the foreground while the preview plays, which keeps the process alive;
  • the service is exported (the system binds to it), so its session only lets trusted controllers in (TrustedControllersOnly, ControllerInfo.isTrusted): the app and its notification, the system (System UI, lock screen, headphone and Bluetooth buttons), apps holding MEDIA_CONTENT_CONTROL and apps the user made notification listeners. Any other app is refused and cannot read the title, the artist or the frame the session shows;
  • the notification shows the tags (title and artist, else the file name) and a picture: the cover of an audio file, the first frame of a video's first clip, else the app's icon (PreviewMediaSession.kt);
  • while it can be heard the preview holds the audio focus like any media player: another app playing, a call or unplugged headphones pause it (EditorPreviewPlayer.onInterrupted); a silent preview (a GIF, a muted video) does not take the focus, and a GIF has no session at all;
  • the CPU is kept awake while it plays (WAKE_LOCK, ExoPlayer's local wake mode). Media notifications need no notification permission on Android 13+.

Export notification (Live Updates)

While an export runs, ExportService holds the foreground (type media processing from Android 15, data sync before; it only keeps the app alive, the export runs in the view model) with the notification of AndroidExportNotifier: the file, the stage, the percentage, the time left once it can be told, a cancel button, and the app opened when touched. On Android 16 it is a Live Update, following developer.android.com ("Live Updates" and "Progress-centric notifications"): Notification.ProgressStyle (indeterminate while preparing), android.requestPromotedOngoing in the extras with the POST_PROMOTED_NOTIFICATIONS permission, the percentage in the status bar chip (setShortCriticalText), ongoing, with a title, no custom view, not a group summary, not colorized, on a low-importance channel (not MIN). The system then shows it promoted (top of the shade, lock screen, status bar chip) unless the user turned Live Updates off for the app; it is then a normal notification with the same bar. Before Android 16 it is an ongoing notification with a progress bar. Updates come at most every 400 ms or on a new percentage (the system drops faster ones). Once over, a short notification says it is ready or failed, only when the app is not on screen, and goes when the app comes back. Before every export, while its notifications cannot show (the app's not allowed, or one of its two channels turned off), they are asked for first (export/NotificationsFirst, rememberNotificationsFirst, around startExportOf in MainActivity): Android 13 and later get Android's own question while it still asks (never asked by the app, or refused once: shouldShowRequestPermissionRationale); refused for good, turned off in Android's settings or a channel off, a dialog of the app's own leads to its notification settings (ACTION_APP_NOTIFICATION_SETTINGS) and the export starts on the way back (ON_RESUME), or at once with "Export without". The export never waits on the answer otherwise: it starts once it is given.

Where things live

I want to change... Look at
a picture filter or the filter order core/command/VideoFilters.kt
an audio filter core/command/AudioFilters.kt
an effect of an audio file (and its preview) core/audio/ (AudioMastering.kt, AudioStage.kt, Dsp.kt)
an edit of a picture (and its preview) core/image/ (ImageEditing.kt, ImageStages.kt)
the picture editor app/.../ImageEditorScreen.kt, app/.../ImagePanels.kt
the layers of a picture (texts, stickers) app/.../LayerPanels.kt, core/model/Stacking.kt
blend modes (export and preview) core/model/LayerBlend.kt, layOver in CommandBuilder.kt, app/.../BlendPanels.kt
shape layers app/.../ShapeLayers.kt, app/.../ShapePanels.kt, updateShape in the view model
transitions between clips core/model/Transition.kt, core/transition/ (recipes, shaders, export graph), transitionJoin in CommandBuilder.kt, app/.../TransitionPanels.kt
the video effects and effect transitions scripts/generate-effects.py, scripts/punch3d.py (the punches' 3D fist), scripts/math_effect.py (the math formulas), app/.../effects/, app/.../EffectPanels.kt, app/.../EffectOverlays.kt
the TV break (technical difficulties) scripts/tv_break.py, app/.../effects/TransitionEffects.kt (tvBreak), core/model/Transition.kt
the picture looks (black and white, glitch...) core/model/PictureLook.kt, core/look/Looks.kt, app/.../PictureLooks.kt
the SVG editor core/vector/svg/, app/.../svgedit/
ASCII and ANSI art (engine, files, editor) core/textart/, media/.../AndroidGlyphAtlas.kt, app/.../textart/
subtitle translation (models, text, UI) core/translate/, app/.../translate/
dubbing in the subtitles' language core/dub/, app/.../dub/
the marks of what AI made (AI Act Art. 50) core/provenance/, core/marking/, seal in ConversionEngine.kt, app/.../provenance/, app/.../marking/, scripts/export-ai-markers.py
the voice set apart from the rest core/voice/, app/.../voice/, app/.../VoicePanels.kt, app/.../VoiceScreen.kt
SVG gradients (editing, tracing) core/vector/svg/SvgGradients.kt, app/.../svgedit/SvgGradientEditor.kt, core/vector/VectorTracer.kt
SVG booleans, aligning, grouping app/.../svgedit/SvgBooleans.kt, core/vector/svg/SvgArrange.kt, app/.../svgedit/SvgActions.kt
the command studio (suggestions, checker) core/command/studio/, app/.../command/
codecs, quality mapping, containers core/command/EncoderArguments.kt
inputs, stream mapping, the filter graph core/command/CommandBuilder.kt
output file names and directories core/output/OutputPlanner.kt
packing parts or frames into a zip core/output/PartsArchive.kt
the conversion steps or error handling core/engine/ConversionEngine.kt
how a new setting reaches the backend app/.../conversion/RequestMapping.kt
the FFmpeg runtime itself media/.../FfmpegKitExecutor.kt + libs.versions.toml
user-facing error/status wording strings.xml via app/.../conversion/ConversionMessages.kt
what the editor shows before export app/.../EditorPreview.kt, app/.../preview/, core/preview/
clip operations (split, move, trim...) core/timeline/Timeline.kt
the cover of an audio file core/command/CoverArt.kt, media/.../AndroidCoverArtist.kt, AudioCoverContent in app/.../AudioPanels.kt
background playback, media notification app/.../playback/
the denoise and its comparison denoiseFilter in core/command/VideoFilters.kt, core/preview/DenoiseSample.kt, media/.../FfmpegDenoiseSampler.kt, app/.../DenoisePanels.kt
the colour key (green screen) core/command/KeyFilters.kt, core/preview/ColorKeying.kt, app/.../preview/KeyEffect.kt, app/.../KeyPanels.kt
the export notification (Live Update) app/.../export/
sound placement, trim, cut, rows core/timeline/SoundSpan.kt, core/preview/OutputClock.kt
the editor layout and tool row app/.../ConfigScreen.kt, app/.../EditorToolbar.kt
a tool panel or the export sheet app/.../EditorPanels.kt, app/.../ExportSheet.kt, app/.../ui/expressive/Panels.kt
how formats look (shape, colours, badge) app/.../FormatVisuals.kt
the timeline of the editor app/.../EditorTimeline.kt, app/.../TrimPanels.kt
undo / redo mutate in app/.../conversion/ConversionViewModel.kt
what a project file holds, recent projects app/.../projects/ (ProjectJson.kt, ProjectStore.kt), core/project/
how clips become FFmpeg inputs footage() in core/command/CommandBuilder.kt
spoken subtitles: words, lines, the script core/subtitles/ (SpokenWords.kt, SubtitleLayout.kt, AssSubtitles.kt, FontFacts.kt)
the speech model and its import app/.../speech/, app/.../settings/SmartSettings.kt
the subtitles panel and their preview app/.../SubtitlePanels.kt, app/.../StyledSubtitles.kt

Adding a setting end to end: field in ConversionSettings -> setter in ConversionViewModel -> field in ConversionRequest -> one line in RequestMapping.kt -> filter/argument in :core -> a test in core/src/test -> one line each way in ProjectJson (or ModelJson for a :core model), so recent projects keep it.

Tests

./gradlew :core:test runs on a plain JVM (no emulator). It covers command generation for every mode, command-line quoting, the manual command resolver, the command studio (help parsing, suggestions, checker, recipes), output naming, duration math, progress parsing, the timeline operations, the preview math (with the denoise comparison's command, DenoiseSampleTest), the text-art engine and its files (TextArtTest), subtitle re-timing, the project files (ProjectJsonTest: every field of every model, filled with random values, must come back from the JSON text) and the whole engine workflow with fake ports.

Storage

Nothing is kept in the cache longer than needed:

  • sources are copied into cache/staging/<session> for each conversion and the directory is deleted when the run ends, whatever the outcome (CacheSourceStager + ConversionEngine);
  • the denoise panel's comparisons go to denoise-samples/ in the cache and are deleted as soon as they are read (or when FFmpeg fails or is cancelled);
  • camera recordings go to recordings/, and stills (pictures, freeze frames) and covers of audio files to stills/ in the cache (CacheJanitor); they are deleted as soon as their project is closed: another file is opened, the recording is cancelled, or the app is left for good (ConversionViewModel). Files only the undo history still refers to go with the project too;
  • at start-up FfmpegGuiApp sweeps what a killed process or an older version left behind (orphaned working copies, old recordings, denoise comparisons, the time-stamped copies earlier versions wrote straight into the cache).

Exported files are results, not cache: they stay in the app's files directory (AppOutputDirectory) until the user removes them. Recent projects are not cache either: they live in files/projects/ (see "Recent projects").

Files opened with the app or shared to it (openedFile in MainActivity) must be content:// URIs of another app (isForeignContentUri): any app can start MainActivity, and a file:// URI or one of the app's own FileProvider would make it open its own private files (cache, exports, projects) on the sender's behalf. Such an intent opens nothing. Files picked inside the app go through the system picker and are not affected.

A shared file into a project under way (IntakeProjects.kt, core/project/ProjectIntake, tested). Above the formats, a file shared from another app is offered the recent projects in an uncontained carousel. ProjectIntake.ways says how a project of each format takes each kind of file (a sound, a picture, a video with or without sound), the most wanted way first: a video project takes a clip at the end, an overlay from the start or a sound from the start; an audio project a clip or a sound, and a picture as its cover when the format holds one; a picture project an overlay, or a frame of a video chosen first in the frame picker (placeInto); SVG, ASCII and ANSI art, transcripts and commands take nothing. Projects that cannot take the file are greyed, say why and shake when touched; one way is simply done, more are asked in a sheet. ConversionViewModel.addToProject opens the project, adds the file as one step of its history and saves at once: the file is only lent by the share, and the save keeps a copy in media/.

Each window of the app has an editor of its own (a share opens a second MainActivity in the sender's task), so a project must never be open in two at once: the later one's work would be undone by the first one's next save. ProjectStore.claim marks which editor has a project open; the one that had it before saves it at once and lets it go (ProjectStore.Holder), the new one reads it after that save (loadWhenSaved), and the first opens it again from what was saved when its window comes back to the front (reclaimProject, on resume, or when it gets the focus again beside another window). Each switch costs the undo history of the project, which is not saved.

Offline guarantee

The app requests no network permission:

  • nothing network-related is declared in app/src/main/AndroidManifest.xml;
  • Media3 declares ACCESS_NETWORK_STATE in its library manifest, which would be merged into the app; the app manifest strips it with tools:node="remove";
  • verify<Variant>NoNetworkPermissions (defined in app/build.gradle.kts) reads the merged manifest on every assemble/bundle and fails the build if any network, Wi-Fi, Bluetooth or NFC permission is present;
  • cloud backup is disabled (android:allowBackup="false").

The permissions of background playback and of the export notification (FOREGROUND_SERVICE, FOREGROUND_SERVICE_MEDIA_PLAYBACK, FOREGROUND_SERVICE_MEDIA_PROCESSING, FOREGROUND_SERVICE_DATA_SYNC, WAKE_LOCK, POST_NOTIFICATIONS, POST_PROMOTED_NOTIFICATIONS) have nothing to do with the network, and media3-session declares none of its own; the check above guards it like any other dependency.

Without INTERNET the process cannot open sockets, so FFmpeg's network protocols are inert as well.

The smart subtitles keep to this: the speech model is downloaded by the user's browser (the settings only open the link) and imported as a file; sherpa-onnx's AAR declares no permission (its manifest is empty), and the check above guards it like any other dependency.

Hardware video encoding

The bundled FFmpeg (n6.1, built with --enable-mediacodec --enable-jni) has the MediaCodec encoders. FFmpeg 6.1's h264_mediacodec copies each frame into the encoder's input buffer packed its own way; a chip that reads frames another way makes a video torn into coloured streaks, with no error anywhere. An emulation of FFmpeg's packing in Kotlin did not predict it on a real phone, so the chip is calibrated for real instead (media/.../HardwareCalibration.kt), from a test button in the settings: for each hardware H.264 encoder (at most three), each input FFmpeg can give it (NV12, I420) and each size step (16, 32, 64, 128, 256), FFmpeg encodes a one-second testsrc2 video with the chip, as an export does, then decodes it and compares it with the original (psnr). Each step is tried on a portrait and a landscape frame near a phone video's whose sides are ODD multiples of the step (16: 1072 x 1904, 32: 1056 x 1888, 64: 1088 x 1856, 128: 1152 x 1920, 256: 1280 x 1792): a side that is also a multiple of a larger step would pass for a chip that needs that larger step (the first version tested 1088 x 1920 only, 17 x 64 by 15 x 128, passed a chip at step 16, and exports at other widths came out torn). Every plane must reach MIN_PSNR_DB (28 dB): checked here with FFmpeg, a correct encode is 38 to 46 dB on each plane, rows read with the wrong stride about 10, colours read from the wrong rows about 20 on the colour planes only (an average would let that through). The first way that passes on both sizes becomes HardwareEncoder (codec name, pixel format, size step). The switch that turns the chip on can only be turned on once a test has passed; a test that passes turns it on, one that fails turns it off. The result is kept in the app's preferences, keyed by the system build, so a system update asks for a new test. A test cannot start while an export runs.

With a calibrated encoder and the setting on, CommandBuilder encodes the picture of an MP4 or MKV with h264_mediacodec -codec_name <the encoder checked>, its pixel format, -bitrate_mode vbr and -g 60 (the default gives a key frame every second), with the picture's sides scaled to the nearest multiples of the encoder's alignment (the pixel aspect ratio keeps the shape). A phone encoder has no CRF, so the quality slider becomes a bitrate (hardwareVideoKbps: 0.02 to 0.25 bits per pixel along a curve, for the first clip's frame at the chosen height and frame rate; 0.7 is about 8 Mbit/s at 1080p30). Decoding and filters stay on the processor, as before. When the processor encodes (the chip off, not calibrated, or after it failed), x264 runs at the preset chosen in the settings (CpuEncoderSpeed: ultrafast, veryfast, the default on the phone, medium, slow). Not used for AVI, GIF, pictures, sound, remuxes, frames, or parts (cutting into parts needs key frames exactly at the cuts, which only x264 places). If FFmpeg fails with the phone's encoder, ConversionEngine deletes the partial file and runs the same request again with x264; it says so with ConversionEvent.Encoder, the processing screen shows who encodes, and the done screen says which one made the file (VideoEncoderUse).

Models in memory

Every model of the app (speech, translation, voice, dubbing, cut-out, EdgeTAM, the magic eraser, and the check of an imported translation model) goes through one gate (speech/ModelGate.kt), so that their memory never adds up: each takes from a hundred megabytes to over a gigabyte, and two together would leave the phone short.

  • One model in memory. Each engine has a ModelSlot; before its model runs, whichever other model is in memory is let go of (it loads again when next needed).
  • One run at a time. A run waits for the one under way to end, whatever started it (an editor, a job in the background, a speed test), so two models never work side by side. A model is never let go of in the middle of a run: unload waits for it.
  • Let go of as soon as the work is done. There is no timer: once a run ends (done, cancelled or failed) its model goes. A piece of work made of many runs holds it (ModelSlot.hold) until it ends: the correction dialog, from the first stroke to its closing (confirmed or not), and a lift across its two calls. A dub holds the voice only while the lines are said (the file is written after), the voice model only while the voice is set apart (not while FFmpeg takes the sound out or packs it), and the way through English loads its second model only after letting the first go.
  • Settings shows a model in memory while it works, with Unload.

Subtitle translation

Subtitles translated on the phone with Opus-MT (University of Helsinki, CC BY 4.0), as converted to ONNX for transformers.js by Xenova: one model per language pair (or from many languages to English, or from English to many), four files each (vocab.json, source.spm, the quantised encoder and decoder; config.json is read when given, else the vocabulary's "" and "" say what it would, MarianConfig.fromVocab). The user's browser downloads them; any number of pairs can be on the phone (app/.../translate/).

Import (TranslationModelStore). The files are picked at once and known by their names (a browser's "vocab (1).json", or a .json saved as .txt); the import is put together apart and takes its place only once the settings read as a Marian model's, the vocabulary and the SentencePiece model parse and both ONNX files open. Its languages come from its name on Hugging Face (given in the sheet, or the settings' _name_or_path); a model that writes many languages lists them by the ">>ita<<" tokens of its vocabulary. Unlike the other models, whose files are pinned, any pair can be added, so each file's SHA-256 is taken at import and checked again on request.

Text (core/translate/, pure Kotlin, tested). SentencePiece.kt reads the .spm file (a protocol buffer, by hand) and cuts text as the unigram model does (NFKC, spaces as "\u2581", Viterbi over the pieces' scores); MarianTokenizer looks the pieces up in the vocabulary, puts the target's token first for a model that writes many and the end of the sentence last, and turns ids back into words. GreedyDecoder writes the likeliest id each time, never the padding, until the end of the sentence (or a word stuck four times).

Running (TranslationEngine). ONNX Runtime on the processor: each line is encoded once, then the decoder (the one without a cache: subtitles are short) is run for every id it writes, its last row of scores read. One model in memory (the way through English lets the first go before the second loads), let go of as soon as the lines are translated (see Models in memory).

Which models (TranslationRoute). A model that reads the subtitles' language and writes the one asked; else two, through English. The language is guessed (LanguageGuess): the alphabet first (Cyrillic, Arabic, CJK...), then which language's commonest words the lines are made of; the user can change it.

What changes (ConversionViewModel.translateSubtitles). Every line keeps its time. A file (SubtitleTranslation) keeps everything but its words: a SubRip file its numbers and times, a SubStation Alpha file its header, styles, every field of its dialogue lines and the override tags that start a line; long lines are broken near their middle; it becomes a new file in the app's files (the project copies it), one step of undo. Spoken subtitles get, line by line (as the subtitles show them), the translated words spread over the line's time.

Voice and the rest, and every part on its own track

The sound of a file split into stems, by two models this project converts to ONNX and serves from its releases (StemModels, kept like the speech model's files by SpeechModelStore; docs/MODELS.md has the record): Bandit v2 (Watcharasupat, Wu, Orife; weights CC BY-SA 4.0) for the dialogue, the music and the effects, and Spleeter's five stems (Deezer, MIT) for the music's singing, piano, drums, bass and the rest. They replace Spleeter's two stems, which RetiredModels deletes from the phone at the first start (RetiredModelsDialog says so, with a way to the Voice page). Bandit comes in two builds of the same weights (StemModels.BanditBuild), each in its own folder, the choice kept by SmartFeatures (chooseBandit): int8, the default (scripts/export-bandit-int8.py writes the float file's GRUs out as matrix products and scans, then quantises the products: a quarter of the size, about three times quicker to load, quicker where the processor has int8 instructions, its stems about 40 dB from the float one's), and the float file, kept below it for older phones and for stems that must match the authors' model to the last bit. A build whose file is not pinned yet cannot be chosen. VoiceSeparator asks for the chosen store each time it loads, and lets go of the other build's session.

Separation (core/stems/, pure Kotlin, tested). StemSeparation is one engine for both: a short-time Fourier transform of both channels at once (left as the real part, right as the imaginary; a periodic Hann window every quarter of it; Fft is the app's own radix-2), shown to the model a window of frames at a time, a complex mask per stem, channel, frame and bin back from it, each stem the sound's spectrum times its mask, back to sound by overlap-add (the squared windows add up to 1.5). Each window reaches some frames past both ends of what is kept of it, so that no mask is cut at its edge and a recurrent network hears around; however long the file, memory stays the same. One stem, the one of "everything else", is the sound less the others: the stems always add up to the sound.

  • BanditMasks: 48 kHz, 2048 points, hop 512, 1024 frames seen (about 10.9 s) of which 48 at each end only heard around. The network is mono: it reads the middle of the two channels, scaled as torchaudio's "normalized" transform, laid out [1][bin][frame][re, im], and its masks are laid on each channel's own spectrum, so the stems stay stereo. Effects take what the dialogue and the music leave. scripts/export-bandit.py checks these transforms against the authors' model end to end. The first file (bandit-onnx-1, retired) gave its masks as doubles (the authors' frequency weights are float64, and carried the sums after them) and had some 23,000 nodes; bandit-onnx-2 puts the masks together in one scatter-add per stem, in float32, and loads three times quicker. VoiceSeparator reads either. Its recurrent layers (GRU) take most of the time and do not run on XNNPACK, so its session stays on the plain kernels, threads spinning.
  • SpleeterMasks: 44.1 kHz, 4096 points, hop 1024, 512 frames (32 heard around), the magnitudes of the first 1024 bins of both channels; each instrument keeps e^2 / sum(e^2) of every bin. Above the 1024 bins nothing is told apart: that top end goes with "other" (Spleeter drops it).

The sound is padded with N - hop zeros in front, so that every sample lies under four frames.

Files (VoiceSeparator). FFmpeg takes the sound out (s16le, two channels, at the model's rate, sample 0 at the file's time 0); Bandit's run writes one stereo WAV per stem and the four-channel one the voice levels and the dub read (the dialogue's left and right, then the music's and the effects' together: VoiceStem), FFmpeg packs each into FLAC in files/voice-stems/, by the media file's address: a file is split once, for every project. The instruments are split from the music stem (read again at 44.1 kHz), only when asked; or, for a song (core/stems/StemSplit.SONG, the first choice for a sound without a picture), straight from the whole sound with Spleeter alone: Bandit, the far slower model, is not run, and a song has no dialogue or effects to keep apart. A .song mark next to those instruments says so: their singing is then the voice the subtitles listen to alone (ListenedSound.voiceOf), and asking for a video's instruments (the dialogue and the effects apart) splits them again from the music. One model in memory at a time, through the ModelGate. Four-channel files left by Spleeter's two stems keep playing; asking for tracks makes them again with Bandit. The settings say how many files are kept and remove them all.

Tracks (core/stems/StemTracks.kt, ConversionViewModel.placeStemTracks). For each clip that plays, the part of its file it plays, from where it starts on the output clock, at the speed it plays at: a stem file is on its media file's clock, so the same part of it, laid there as an AudioClip, plays in step with the picture. The clip goes silent; a track takes its clip's volume and the voice's or the rest's level. Clips played backwards or on a speed curve (or under a whole video so played) are left as they are and counted. One step of undo. Each track knows the clip it was laid for (AudioClip.stemOf, kept in the project; a copy laid elsewhere, or a track whose file is replaced, forgets it). Cut in two, the clip cuts its tracks where its halves meet, each part following its half (withStemTracksCut). On the audio file's row (EditorTimeline, splitClips) such a clip is an empty slot with a dashed outline: it still sets where the rest of the sound is, so it cannot simply go; when every clip is one, the row is made thin.

Put back together (core/stems/StemRejoin.kt, tested; ConversionViewModel.rejoinStems; StemRejoinCard). All of a clip's tracks: they go and the clip is heard again. Two or more of them: they go, and one track holding their stems takes their place, where the clip lays its stems now, at the clip's volume (and the voice's or the rest's level when it holds only one of them). Its file is the stems summed by FFmpeg (amix, normalize=0: the stems add up to the sound they were split from), kept beside them in voice-stems and named after what it holds (<file>.piano-drums.flac, StemRejoin.kindsPart), so it is put back with the others later, is part of the same premix in the preview, goes with the stems when they are removed and is made again when they are split again. What was done to the tracks put back goes with them; one step of undo. For the whole video one button puts every clip back.

Export (CommandBuilder.footage). A clip whose Effects.voiceLevel or backgroundLevel is not 1 takes its sound from its file's stems (SourceInfo.voiceStem, filled in by withVoiceStems), an input of its own seeked like the clip (-ss, -t), made stereo again at the two levels (pan=stereo|c0=v*c0+b*c2|c1=v*c1+b*c3) before the clip's own sound chain. Without stems the clip keeps its file's sound.

Preview (previewStemSounds). Such a clip's own sound is off and its part of the stems plays on a player of its own, where the clip is on the output clock (as the placed sounds do, SoundVoices); StemMixProcessor mixes the four channels down at the two levels for every buffer, so a slider is heard at once.

Where. The editor's Stems tool (video and audio editor), per clip or for every clip: on top the card of the split, with a strip of the parts it gave (dialogue, music, effects, instruments) lit as they are there; then "Quick mix" (the dialogue's level and the music's and effects' together, live, with presets) or "Tracks" (StemTracksCard: how far to split, which parts, one button). On the timeline a placed stem carries its kind's icon before its name: known by its file (VoiceSeparator.stemKindOf, its name in voice-stems) or by its own name ("Dialogue (film)", stemKindNamed). A sound's name is in its fill's "on" colour with a soft halo of the fill, so it reads over the waveform without covering it.

Dubbing

The subtitles said by a voice in their language, over the video: Kokoro-82M v1.0 (hexgrad, Apache 2.0) in sherpa-onnx's multi-language float32 build (on a phone's processor more than twice as fast as the int8 one, whose quantised convolutions ONNX Runtime runs slowly, and less metallic), run by sherpa-onnx's own text-to-speech (already in the app for speech recognition).

Model (DubModel, DubModelStore). One archive from sherpa-onnx's releases; unlike the other models it is unpacked whole, folders kept (eSpeak NG's data is a few hundred files), into files/dub/. The archive is hashed as it is read and the main files (the network, the voices, the token list, the lexicons, eSpeak NG's phoneme table) are pinned by their SHA-256. Its state reads like the other models' (SpeechModelStore.State), so the settings show it with the same card.

Languages and voices (core/dub/DubVoices.kt). The model's 54 voices in its own order (their speaker ids); a name says the language and whether the voice is a woman's or a man's. Each language is said through its eSpeak NG voice, English through its lexicons first, Chinese through its lexicons, jieba's dictionaries and the rules for numbers and dates. Japanese is left out (its kanji are read as Chinese).

Lines (ConversionViewModel.dubLines). The subtitles as they are shown on the edit: a file's cues moved onto the timeline (onTimeline), or the spoken words laid out in lines (SubtitleLayout), then on the output clock (divided by the whole video's speed). DubTiming.spoken keeps what can be said (no tags, no "[music]").

Timing (DubTiming, pure Kotlin, tested). A line may take the time until the next one starts (and never past the end of the video); a voice that needs more is said again faster (the voice's own speed, up to 1.5 times), and what still does not fit is made to fit (DubFit): its long pauses shortened to 140 ms, then the whole line faster without a higher voice (TimeStretch, WSOLA: 30 ms Hann frames, each moved up to 10 ms to where its waveform best goes on; up to 1.6 times), and only past that its end cut, faded. Nothing pushes the next lines later: every line starts with its subtitle, so the dub cannot drift from them nor outlast the video. How fast the voice speaks is learnt line after line (DubPace, characters a second), so that a long line is said fast enough the first time: a line is said a second time only when that was still not enough. Every line has its silence cut at both ends and its ends faded (DubAudio): it starts when its subtitle does, without a click.

In the preview a placed sound a little off its place (a long one, such as a dub, drifts) is brought back by playing it 4 % faster or slower until it is (SoundVoices.nudge), rather than by a seek, which is heard.

Stems heard together (core/preview/StemPremix, tested; media/FfmpegStemPremixer; preview/PreviewMapping.premixBundles, withPremixes). Players drift a few milliseconds apart, and nudged back they keep moving round their place: nobody hears that between unrelated sounds, but the stems of one song are parts of one waveform, and the same notes a few milliseconds apart cancel and add (a comb filter), heard as distortion. So the stems of one file (VoiceSeparator.stemOriginOf) laid at the same point, at least two of them heard, each with a known end, form a bundle: FFmpeg mixes them into one FLAC of the cache with the export's own filters (AudioFilters.soundShaping: trim, repeats, speed and pitch, equaliser, effects, levelling, volume and keyframes, own fades; the part of soundChain that does not depend on where the sound lies) and the export's mixer, and one player plays it in their place, at the bundle's start, with only the whole video's level, fades and effects on it, as on any sound. What is heard is what the export writes, and the level drawn is the premix's own. A muted stem stays in its bundle, left out of the mix as the export leaves it out. The bundle's start is not in the premix (each stem on its own clock), so moving the stems does not mix them again; any other change does, after the settings stay still for 400 ms (wantStemPremixes, one at a time, stale runs cancelled), and until it is ready the last premix made for the same tracks is heard (the stems one by one the first time). The file is named after the command and each stem's size and time, so stems split again are mixed again; premixes not heard for three days go at start-up. When a premix fails, its command and FFmpeg's log go to Logcat (StemPremixer) and the stems play one by one.

Result (DubEngine). Every line, at its place, in one mono WAV at 24 kHz in files/dubs/, laid on the video as a placed sound (AudioClip): it is moved, trimmed and levelled like any other (its "Level loudness" on from the start, the file measured right away: the voices come out of the models louder or softer than one another), and a project keeps its own copy. A new dub takes the place of the one made before. With "lower" or "take out", every clip's voiceLevel is set (see "Voice and the rest"): the files without stems are separated first, in the same run.

In one go (ConversionViewModel.autoDub, AutoDubSectionContent, the editor's "Dub" tool). The language, the voice and what becomes of the video's own voice (taken out or lowered) are picked; one button runs the editor's own steps one after the other, each awaited: listening (generateSubtitles, skipped when the video has subtitles), translating (translateSubtitles, from the language guessed in what was heard; skipped when it is the dub's already), then dubSubtitles (which sets the voice apart first where it is not yet, then dubs). Each step shows in its own place as it goes and keeps its own undo; the tool shows the four as a group of rows that tick off, with the running one's progress read from that step's own status. What can be known to be missing (the speech model, the voice model, a translation model once the language is known, the voice) is said before anything starts; a translation model for a language heard only then stops the run after the listening, which stays.

The two screens (DubCard in the subtitles panel, AutoDubSectionContent; their pieces in DubPickers.kt). Both are one group of joined blocks: the language as a single sideways strip of tokens (its code in a shape that blooms when chosen; the subtitles' own, found by itself, starred), the voice as a carousel of cards (a woman's, a man's or a quick Piper voice, its kind, a gavel when its licence has conditions) each with its own button to hear it (DubSampler: one voice at a time, the card shows it loading and playing), then how it sounds (speed and the video's own voice, a connected choice). However many voices there are, they take one line; the chosen one is scrolled into sight. While the card dubs, the group gives way to the voice saying it, its language and speed, the share done and the way to stop. The tool shows its plan before it starts as four nodes on a rail (struck off where not needed, a warning where something is missing, said in full under it); once it runs, the rows that tick off.

One breath (DubTiming.oneBreath). For eSpeak NG languages sherpa-onnx cuts a text at every ., !, ? (and ;), says each piece on its own and glues them: the seams are heard, the voice stops and starts again. So inside a line those marks become commas (CJK ones the full-width comma) before it is said, its own end kept: one line, one run of the network, one intonation. Numbers ("3.5") and abbreviations without a space after them are left.

Piper voices (core/dub/PiperVoice.kt, PiperVoiceStore, DubSpeaker). Any Piper voice (VITS, many times quicker than Kokoro on a phone's processor: about 0.05 of real time against 0.35 on two desktop cores) is imported as published: a sherpa-onnx vits-piper-*.tar.bz2 archive, or Piper's own .onnx and .onnx.json (a voice trained at home comes out so). sherpa-onnx wants facts Piper keeps in the settings inside the network (its metadata: model_type=vits, comment=piper, the eSpeak voice, the rate, the speakers) and the phoneme table as a token list: PiperVoiceConfig makes both from the settings, and OnnxMetadata appends the metadata to the file as more of the protocol buffer (ModelProto's field 14), without reading the network into memory. eSpeak NG's data is kept once for every voice in files/dub/ espeak-ng-data (from the first archive, or copied from Kokoro's), so the voices outlive the Kokoro model. Nothing is pinned (a voice can be any one): its SHA-256 and card are shown. DubSpeaker is one voice to dub with, Kokoro's or a Piper speaker; DubEngine keeps one network loaded (Kokoro for one language, or one Piper voice).

The list (core/dub/PiperCatalog.kt, PiperAddSheet). Adding a voice is picking it: the language (the phone's first), its voices the best first, with their quality, speakers and size, and a search across all of them. A voice opens into its two steps: its sherpa-onnx archive to download in the browser (or Piper's own three files on Hugging Face, once eSpeak NG's data is on the phone), then the import. The list is Piper's voices.json (123 voices, the one whose name is not ASCII left out) and OpenVoiceOS's two Italian "high" voices that sherpa-onnx packs; the app never goes online for it.

Licences (core/dub/VoiceLicence.kt, DubLicenceDialog). A Piper voice inherits the licence of its recordings, which its card states. On the phone the card is read for it (tested): research only (the Blizzard licence and the like), not for commercial use (CC BY-NC...), free (CC0, CC BY, public domain, Apache, MIT...), or not known, which is treated as not free. A voice that is not free shows a gavel on its chip and a tinted note under it, and before every dub with it a dialog says what its licence allows, what counts as earning from a video, that keeping to it is up to whoever publishes it, and offers its card, a voice free to use, or the dub. Kokoro is Apache 2.0.

Speed. The network runs on the processor's cores; a phone's GPU or NPU cannot be used: ONNX Runtime's Android build has no GPU provider, and NNAPI takes only fixed shapes, which these networks are not. ONNX Runtime's XNNPACK kernels are tried by default for every model but the invisible AI marks' (always the plain kernels: see "The invisible marks") (AiKernels: one switch at the top of the smart features, read by each engine when it loads; the ONNX Runtime sessions get the XNNPACK provider with its own threads and ONNX Runtime's left not spinning, the sherpa-onnx ones the "xnnpack" provider); where they cannot run, the plain kernels are used, and a change lets go of every model in memory. A dub says how long it took.

Marked as AI-generated (AI Act, Article 50(2)). The dub's WAV carries a LIST INFO comment (WavWriter) saying "AI-generated voice", and an XMP chunk saying it is all made by a model (IPTC trainedAlgorithmicMedia, RiffXmp); no dub is kept without it. The clip keeps the voice's name (AudioClip.aiVoice, in the project) and the request carries it (PlacedSound.aiVoice): every export where a dub is heard is marked (see "Marked as made with AI" below).

Magic eraser

What is painted over a picture goes, filled in with what should be behind it: in the picture editor (the picture itself, ImageTool.ERASE, and its stickers) and on pictures laid over the video (EditorTool.OVERLAY_ERASE).

Models (EraseModel, two SpeechModelStores, one chosen as the cut-out is). MI-GAN (Picsart, MIT), its ONNX pipeline taking bytes (the mask 255 where the picture is kept); LaMa big-lama (Apache 2.0, ONNX by Carve), a fixed 512 square of floats (the mask 1 where it is filled). MI-GAN's file is pinned; LaMa's is published without a hash, so FileCheck takes an empty pin as "there" and the card shows what was found.

Choosing what goes (RefineMaskDialog, MaskKind.ERASE). The cut-out's dialog with its two brushes: the magic one (EdgeTAM) finds the whole of what is tapped or scribbled on, the precise one takes exactly what it covers; painted in raspberry, kept in green. It starts from a blank mask (HdrSources.blankMask), or from the subject the cut-out model finds. The dialog (shared by the cut-out and the HDR light) keeps the picture as large as it can: its top bar has only Close, the title and Reset; what is going on is a pill over the top of the picture (aside while a finger paints); view, compare, fit, Undo and Redo float at its bottom, by the thumb; under it one tray holds the tool, the brush and its size, and a large button saying what it does ("Erase what is painted", or applying the correction). The panel puts the two ways in first and the model under them, as two compact rows (first, when the chosen model is not on the phone).

Filling in (EraseEngine, core/erase/InpaintArea, tested). The mask grown by the picture's side over 150 (an outline or a shadow goes too), a square around it of 2.2 times its size and at least 512 cut out and squeezed to the model's square, the model's picture squeezed back and laid only where the mask was, faded over the side over 220 outward; the picture's own alpha kept. A new PNG in files/erased/ takes the old picture's place (replacePicture for the picture itself, the overlay's uri otherwise): one step of undo. Through AiKernels, like every model. Beside it, in files/erased/unmarked/ under the same name, its twin before the invisible mark: never shared nor exported (what leaves the app is the marked PNG), it is what the next erase of that picture starts from, so that the result is marked once instead of a new mark laid over the old (they add up and show as a pattern; a mark does not survive another erase of more than about a twentieth of the picture, so it cannot simply be kept).

Memory (issue 28). The Java heap is a few hundred megabytes whatever the phone's memory, and a photo of 12 megapixels is 50 MB per copy as an int array. So the picture (decoded mutable, at most 4096 pixels) and its mask stay bitmaps; the mask's box is read a row at a time; only the crop is held as arrays (the mask in it, grown), the fade is weighed only three feathers past the erased part (its grow and blur reach two, so the weights are those of the whole crop: tested), and the result is laid row by row into the picture itself, which is then written. The app asks for the larger heap (largeHeap) as the margin for what is left.

Marked as made with AI

A file that holds what an AI model made says so, in a form programs can read (Regulation (EU) 2024/1689, Article 50(2); the Commission's guidelines on Article 50 count dubbing and the removal of things from a picture among what is marked, and transcription, translation, background removal and noise reduction among what is not). core.provenance, pure and tested:

  • What. AiDisclosure: whether a dub is heard (a clip with an aiVoice) and whether a picture a model changed is seen (ConversionRequest.aiEditedPictures), in words ("AI-generated voice", "AI-generated inpainting") and as IPTC's Digital Source Type (compositeWithTrainedAlgorithmicMedia; a dub's own file trainedAlgorithmicMedia). aiDisclosure() keeps what the format carries; for a command written by hand, aiSources() takes everything and AiSeal.disclosureFor what a file of its extension can hold (sound, pictures, both, or nothing for text).
  • Nothing else. No model, no voice (one the user imported may carry a person's name), no app, no version, no date, nothing of the phone or the person: as little as Article 50 asks for, so that the marks, alone or with other data, point to no one (GDPR, data minimisation). The app's own name is gone from the other files it writes too (the WAV's ISFT, the Ultra HDR XMP's xmptk, the subtitle script's header).
  • Where the pictures come from. The magic eraser writes its PNG with the same XMP iTXt (EraseEngine); AiMarksReader reads the XMP of the PNG and JPEG files of a project (its files, the pictures laid over it, the picture on its canvas), its own results wherever a project copied them and pictures other programs marked as made with a trained model (IPTC's source type, nothing more), and RequestMapping sets aiEditedPictures (AiEditLookup, set from AppContainer.aiMarks, one read per file).
  • Pictures in words too. FFmpeg writes no tags into a picture, so a picture a model changed gets its words after it, in its EXIF UserComment ("AI-generated inpainting", before a comment of the user's own; withAiNote), written whether or not the user set tags: what galleries and the app's own tag reader show, beside the XMP.
  • Two layers. The tags AI_GENERATED and DIGITAL_SOURCE_TYPE, given by CommandBuilder after the user's own (MP4 and M4A with use_metadata_tags; FFmpeg keeps only the last -movflags, so the command gives them once, all together, CommandBuilder.oneMovFlags), and an XMP packet (Xmp) that ConversionEngine.seal writes into the finished file, after FFmpeg and after the picture tags, each part of a zip included: MP4 and QuickTime in a top-level uuid box, WAV and AVI in a _PMX chunk, PNG in iTXt, JPEG in APP1 (an Ultra HDR picture keeps its own XMP, which names its gain map, with the marks merged in, and its MPF index is set right; UltraHdr.assemble does the same), WebP in an XMP chunk (a simple WebP made extended), GIF in an application extension, TIFF in tag 700, SVG in <metadata>. An XMP already there is put out of use. Matroska, WebM, MP3, FLAC and Ogg keep only the tags.
  • Not removable. The settings cannot leave them out (the tags come last, the XMP after FFmpeg). A command written by hand is followed by a stream copy that writes the tags again (AiSeal.remux, for the formats whose marks are their tags, and MP4), then the XMP; the command studio says so where a command drops or sets them (LintCode.AI_MARKS_KEPT).
  • Never without them. A format with no place for the marks (BMP, MPEG-TS and -PS, an extension AiSeal does not know) fails with AI_MARKS_NO_PLACE, a file they cannot be written into with AI_MARKS_FAILED; the unfinished file is deleted, as on any failure.

The invisible marks (core/marking/, pure and tested; app/.../marking/):

  • Voices: AudioSeal (MarkEngine.markSpeech, from DubEngine once the voice is let go of, and for the voice previews too). Each line at 16 kHz (Resampler), in 5 s windows that overlap by 0.5 s (the ONNX files take a fixed window: scripts/export-ai-markers.py), each window's watermark from the generator with the message of zeros, joined with crossfades, back at the line's own rate and added to it: the speech itself is never resampled, the silences between lines stay silent. Then read back by the detector over every line: at least 80% of the speech (its 20 ms frames above -50 dBFS) marked, the message zeros, or the dub is not kept.
  • Pictures: TrustMark Q (MarkEngine.markPicture, from EraseEngine before the PNG is written), as Adobe's encode(): the whole picture (its centre square when one side is more than twice the other) averaged down to 256 x 256, the encoder's difference with each colour's mean taken away, stretched back bilinearly and added, fading in over the last 1% of each edge; the transparency left as it was. The payload is TrustMark's BCH_SUPER schema with the fixed 40 data bits 0101...01, their 56 BCH bits and the version 0000 (Marks.PICTURE_CODEWORD): Adobe's own reader finds it as such. Not all zeros: the encoder takes the bits through a linear layer, and with no bit set the mark came out too weak (6 to 11 wrong bits right after marking on the export's test photos, often not found); with as many ones as zeros it read with none, also after JPEG 75 and at half the size. Read back at once: at most 3 wrong bits of 96 (BCH corrects 8), or the result is not kept. A picture that already reads with at most 3 wrong bits (erased again after a first erase) is not marked again: the marks would add up and start to show. Before the picture is touched the mark is checked twice: its mean strength (at most 0.08 in -1..1 units; TrustMark's own measured 0.001 to 0.04) and its bits on the marked 256 square; if either is wrong nothing is laid and nothing is kept. The marks' networks always run on ONNX Runtime's plain kernels, whatever the XNNPACK setting: with XNNPACK, TrustMark's encoder was seen on a phone to give a mark far too strong (burnt colours, bursts of blue). The log says the mark's strength, its wrong bits and the picture's size and colour space.
  • What was made before (a dub, an erased picture, from an older version or another program that marked it as made with a trained model): MarkGuard, right before an export, reads its mark and, when it has none, marks a copy that the export uses (a dub line by line between its silences: SpeechMarking.runs); an export that cannot have them fails.
  • Before the dub and the eraser (MarkingGate, MarkingSweep): the marks' four files (MarkerModel, one import) come before the models that make voices and pictures: the dub's (DubModelStore: Kokoro; PiperVoiceStore: the Piper voices, eSpeak NG's data) and the magic eraser's (EraseModel.MIGAN, EraseModel.LAMA, the only SpeechModels with generative = true). Those stores' imports ask the gate; the refusal says why (MarkersFirstDialog). The other models (speech, translation, cut-out and EdgeTAM, the stems) make nothing that is marked (the Commission's guidelines count transcription, translation and background removal among what is not): their stores do not ask, and they are imported and kept with or without the marks. At every start, before any store reads its files, the dub's and the eraser's models without the marks, or older than them, go and the user is told once (ModelsClearedDialog); the others are left alone (MarkingSweep.KINDS keeps them, generative = false, only so that what an older version deleted is still named). Removing the marks removes the dub's and the eraser's models with them (SmartFeatures.deleteMarkersAndGenerativeModels), not the others. In the settings, while the marks are missing, the dub's and the eraser's tiles on the hub are grey (Modifier.greyedOut: saturation 0, half faded) with a "Needs the AI marks" pill, their badges in the hero fainter, and their pages show the way to the marks (MarksFirstNotice) over their cards, grey and out of reach (LockedForMarks: a cover over them takes the taps and opens the marks' page, a drag still scrolls; screen readers hear one button that says why). The dub and the eraser refuse to start without the marks, and the engines throw rather than hand out something unmarked.
  • Nothing that tells anyone apart. Both payloads are constants (the speech's zeros, the pictures' fixed codeword), the same for every user: no identifier of the phone, the person, the file or the model, no date, nothing random (a random value per file would make its copies linkable). The metadata are signed by no one, by choice: a key per phone would link every file of a person, a key shared by every copy of an open-source app is public and proves nothing, a server that signs would need the network the app does not have and would make whoever runs it the controller of what it is sent; a trusted time-stamp would add the date. The Code of Practice on AI-generated content suggests signed, time-stamped metadata; it is voluntary, the Act asks for marks "as far as technically feasible", and the GDPR (data minimisation, Article 5(1)(c); the AI Act does not override it, its Article 2(7)) asks for as little as needed: the robustness signing would add is what the invisible marks give.
  • That they carry nothing is not certain, and the README says so to the user. The design leaves no identifier in them, but: the constant payloads and the XMP say "made with software that marks this way", and with few users that alone narrows down who made a file; the marks come from neural models that cannot be fully inspected, so nothing proves a mark carries no more than its bits, or that it does not tell which model files or app version wrote it (its residual depends on both, and on the content); and the rest of the file can identify its maker anyway: the place and device tags FFmpeg keeps unless "Remove sensitive" (withoutSensitive) is chosen, the encoder tags and x264/x265's settings SEI, the file's dates, the content itself.
  • Their files are made by .github/workflows/ai-markers.yml (on every push that changes scripts/ai-markers.lock, the export script or the tools, and by hand): AudioSeal converted from Meta's weights, TrustMark's ONNX files as Adobe serves them, every input pinned by hash, built twice and compared, checked against PyTorch and against compression, published as the pre-release ai-markers-1. Until its first run has learned the pins (scripts/pin-ai-markers.py) nothing can be imported and no app release passes scripts/check-model-pins.py. See docs/MODELS.md.

The whole sound of a dubbed file (SoundSeal, ConversionEngine.sealSound, marking/SoundSealer). A dub carries AudioSeal's mark from the moment it is said, but in a finished file it is mixed with music and effects, and there the mark sinks under them: tested with the real models, 4 s of 14 found with the wrong message under moderate music, under 1.5 s under louder music, so the AI check could not find it. So the export that holds a dub marks its whole sound once mixed, after FFmpeg and before the XMP: the first sound taken out as a 32-bit float WAV at its own rate and channels (SoundSeal.decode), marked 30 s at a time (mixed down to mono at 16 kHz, MarkEngine.watermarkOf over every part that is not digital silence, the watermark brought back to the file's rate and added to every channel), and put back in place of the first sound, every other stream, the tags and the chapters copied (SoundSeal.remux), encoded as the file's sound is (codecFor: a video's AAC at 192 kbit/s, being its second encoding; an audio file at the bitrate chosen). Found then on 99.7 % of the sound after AAC. Where the dub's own mark and this one lie together the message reads less surely, so the check goes by the share first. It is a step more for such an export (AudioSeal's generator: about half a second per 5 s of sound on two cores), shown as a stage of its own (ConversionStage.MARKING, "Marking the AI voice") with its own progress, 10 s of sound at a time, and its own time left, rather than a bar held at 99 %; and best effort: the dub keeps its own mark if it cannot be done (no sound, a format it does not know, a failure), which the log says.

The terms (README, About) ask not to remove the marks. Their limits: a sound or a picture changed enough (re-recorded, heavily cropped, painted over) can lose the invisible mark, as any mark today.

The AI check (AiCheckScreen, provenance/AiCheck): from the home, a picture, a sound or a video is picked and its marks are looked for, on the phone, nothing sent. Its metadata anywhere in the file (AiMarksReader.scan on the whole file, or its first 64 MB and last 32 MB: every format the app marks keeps its XMP as plain text, and Matroska, MP3, FLAC and Ogg their DIGITAL_SOURCE_TYPE tag; "made" for trainedAlgorithmicMedia, "changed" for the composite, "AI voice" when it says "AI-generated voice"). Then, when the AI marks are installed (else it says so and leads to them), its invisible mark: TrustMark's in a picture's pixels (MarkEngine.pictureMarked, decoded at most 24 megapixels), AudioSeal's in the speech of a sound or a video's first sound track (FfmpegAudioExtractor.extract at 16 kHz, its first 10 minutes; SpeechMarking.runs, then MarkEngine.speech), so a dub is found even with its metadata gone. A dub on its own is marked in most of its speech (SpeechVerdict.marked); in a finished video it is mixed with music and effects, which count as sound too and are not marked, so the share falls (a test mix: 13 %). There the mark is found by how much is marked, at least 2 s, with the app's message read where it is (SpeechMarking.inMix); stray detections in a sound that is not marked come to a fraction of a second, with another message. It reads only these marks: "no AI mark found" is said as such, never as "not made with AI".

Smart subtitles (speech)

Subtitles made from what is said in the video, on the phone. The idea, the model and the way to run it come from the anti-vocale app (github.com/RisorseArtificiali/anti-vocale, Apache 2.0); the code is this app's own.

The model. NVIDIA's Parakeet TDT 0.6B v3 (CC BY 4.0, 25 European languages, found by itself), as converted and quantised to int8 by sherpa-onnx (speech/SpeechModel.kt). It is not shipped: Settings > Smart features opens its three ONNX files in the browser, from this repository's release parakeet-onnx-1, which serves sherpa-onnx's files unchanged (scripts/parakeet-onnx.lock; docs/MODELS.md says why), and the user imports them; the token list is in the app's assets (kept byte for byte by .gitattributes). speech/SpeechModelStore.kt reads each file once, as a stream, hashing it as it is written into a temporary folder, and the folder takes the model's place only when every file matches its pinned SHA-256. The hashes are kept; "Check again" hashes the files anew, and a file that does not match shows the hash found next to the one expected. Only files that passed are ever handed to the native runtime: a damaged model makes ONNX Runtime abort the whole process. (An earlier build, SmoothQuant, the import of the sherpa-onnx archive and a speed test on its recordings are gone: cutout/RetiredModels.kt deletes what they left on a phone.)

Running it. sherpa-onnx 1.13.8 (Apache 2.0, with ONNX Runtime, MIT), the AAR of its GitHub release in app/libs (not on Maven Central), pinned by its SHA-256: the build checks it (verifySherpaOnnx) and scripts/fetch-sherpa-onnx.sh downloads it. Its C and C++ API libraries are left out of the APK (the app uses JNI only), and R8 keeps its classes (the native code reads them by name). speech/SpeechEngine.kt loads the OfflineRecognizer (greedy search, cores - 2 threads, between 2 and 4) when it is needed and lets it go as soon as the file is heard (see Models in memory).

Listening. speech/SpeechTranscriber.kt takes the sound out of a file with FFmpeg (media/.../FfmpegAudioExtractor.kt, the command from core/subtitles/ListenedSound.kt: 16 kHz mono floats, padded so that sample 0 is the file's time 0). When the voice of the file was set apart before (its dialogue and singing stems both kept by the voice separation), the two stems are mixed and heard instead of the whole sound: under music, singing above all, the model takes a voice for music and writes nothing, for most of a minute at times, while the same voice alone is heard (measured with the pinned model: a sung voice over music lost 10 to 24 sentences of 36, alone one). Nothing is separated for this; shorter pieces, a second pass shifted by half a piece and a blank penalty were tried and did not help (the penalty writes words over instrumental parts). The sound is fed to the model 30 seconds at a time (the model's memory grows with the square of what it hears at once). Each chunk ends at the quietest 20 ms of its last two seconds, so that no word is cut in two. The model returns word pieces with their start and duration (TDT); RecognizedWords (core/subtitles/SpokenWords.kt) joins them into words: a piece that starts with a space starts a word, punctuation joins the word before without keeping it on screen longer. The words of every chunk reach the preview as they are found (liveWords of the view model) and the settings when a file is done (one undo step); they are kept per file, on the file's own clock (ConversionSettings.spokenWords, saved in the project as a line of times and a string of words).

Lines and look. Words follow their footage through the edit like subtitle files (onTimeline), then SubtitleLayout makes the blocks shown at once: a block ends at the end of a sentence, at a pause, after six seconds or when its lines (one or two) are full; two lines are balanced. Lines break where their width, added up from the glyph advances of the very font file the export uses (FontFacts reads its cmap, hmtx, OS/2 and name tables), would leave 88 % of the frame; a single word too wide shrinks its block. SubtitleStyle holds the size (a fraction of the frame height), the place (the bottom middle of the text, dragged on the preview), colours, outline / box / shadow, bold, capitals and the highlight of the word being said (that word, or every word so far). A SubRip file is drawn with the same style; a SubStation Alpha file keeps its own.

Export. The engine writes the blocks as a SubStation Alpha script (AssSubtitles.kt): a canvas 1080 lines high and as wide as the frame's shape, every line placed with \pos from the bottom middle and broken with \N (wrapping off), the highlight as colour changes, one event per word where it moves. The font is copied alone into a folder handed to libass (fontsdir) and named by its family; FFmpegKit's fontconfig also knows /system/fonts (media/.../FfmpegFonts.kt) for files that name other fonts. The script is burnt in where subtitle files are, before the speed change.

Fonts. Captions and subtitles choose a typeface (TextFont: sans, serif, mono, typewriter, handwriting, casual, small caps), each one of the fonts AOSP ships in /system/fonts (SystemFontProvider; a phone without one falls back to the default font). drawtext, libass and the preview all draw from the same file (LocalExportFonts).

A caption's look and own font. A caption can have an outline, a shadow and a box behind it (TextLook, sizes as fractions of the font size, colours ARGB). drawtext takes them in whole pixels (borderw, shadowx/shadowy, box/boxborderw, colours as 0xRRGGBB@alpha), so VideoFilters.lookOptions works them out for the caption's starting size on the frame it lands on (the clip's frame, the whole video's at the output height, the picture or its canvas; 1080 when not known); the preview (TextOverlaysLayer) draws the box, then the shadow, the outline (a stroke twice as wide) and the letters at that same size. A font the user added (.ttf, .otf) is copied into the app (CaptionFonts, checked with Typeface.Builder) and kept as the caption's fontFile; FontProvider.customFontPath gives it to drawtext while it can be read, else the caption's family is used, and the preview does the same. Such a font is drawn as it is, without a made-up bold, in both.

As texts. "Make them texts" turns every line of every block into an ordinary caption (TextOverlay), timed on the output clock, in the subtitles' place, size, colour and font; the word highlight and the outline stay with subtitles only.

Transcript. The TXT and SRT formats (category TEXT) are not written by FFmpeg. The format picker greys them out until the model is set up; picking one opens TranscriptScreen: the file plays (ExoPlayer) above its lines, which are the blocks SubtitleLayout.ofWords makes, found live while the model listens. The line being said follows the player and its word is lit; a tap on a word seeks there, the pencil corrects a line in place (editSpokenLine, the same edit as the subtitles panel). Saving runs the export, which reuses the words already heard (and listens to any file not heard yet), moves them onto the edit and writes either paragraphs with their times (TranscriptText) or SubRip cues cut by the subtitle style (SrtSubtitles).

All the tags. Besides the usual fields, MetadataTags.custom holds every other tag the file had (AndroidMetadataReader: FFprobe's tags of the file and of its first sound stream by the names the file gives them, without those that only describe the container; a picture's EXIF from the list EXIF_TAGS, not its size, orientation or thumbnail) and those added by hand, grouped by tagKind (place, device, date, camera settings, other; place and device are sensitive). sourceKeys remembers which were read: one taken away is written empty, which makes FFmpeg drop it. "Remove sensitive" (withoutSensitive) drops those groups and also blanks the keys phones and cameras use even when they were not read (SENSITIVE_FFMPEG_KEYS); "Remove all" (cleared) adds -map_metadata -1 (with a cover in a tags file, only that file's tags are mapped). An MP4 or M4A keeps tags of its own only with -movflags +use_metadata_tags, added when there are any. FFmpeg writes no EXIF: a picture gets what is listed (AndroidPictureTagger, by EXIF name), so a picture's GPS and camera now travel with it unless taken away.

Lyrics. The tags of an audio file hold its album artist (album_artist: TPE2, aART, ALBUMARTIST) and its lyrics (lyrics: FFmpeg 6.1 writes it as an MP3's USLT frame, an M4A's lyrics atom, a LYRICS comment in FLAC, Ogg and Opus). The lyrics are typed, or written from what is sung (ConversionViewModel.lyricsFromSpeech): the files not heard yet are listened to as for subtitles, the words follow the edit onto the exported file's clock, and LyricsText (core/subtitles) makes short lines (a pause, a sentence, a comma once half full, 42 characters) and verses (3 s of silence), plain or synced the LRC way ("[mm:ss.xx]line", an empty timed line where a verse stops), which players that know LRC follow as the song plays. Read back from a file, FFmpeg's "lyrics-eng" (an USLT frame) counts as the lyrics. A switch, off at first, sets the singing apart before listening: every file is split as a song by Spleeter alone (VoiceSeparator.separateInstruments(song = true), those already set apart and the stems laid as tracks left as they are), its progress shown as the transcriber's Step.Isolating, then every file is listened to again, from its singing (ListenedSound). Slower, but the words are heard where the music covers them; off, the song is heard as it is, quicker.

The lines are also timed by hand (issue #16, LyricsSync.kt): a page with two views, "Plain" (the words as text) and "Synced" (a line each: a stopwatch that stamps it with where the song plays now, held to take the time away; its words; a button that plays the song from it), over the song's own player (where it is, play, 5 s back and on). The song is the editor's preview (LyricsPlayback: the position and length on the exported file's clock, a seek through OutputClock.timelineMs), so the times hold for the file made. LrcLines (core/subtitles, tested) reads and writes the lines: an LRC header kept, a line timed twice kept once, words edited as text keeping the times of the lines that are still there (retimed), and the line sung now (lineAt). Done writes LRC when a line is timed, plain lines otherwise.

Preview. StyledSubtitles.kt draws the same blocks on the whole output frame (canvas included) with the same font file, as libass measures it: a line is as high as the font size and its baseline sits where the OS/2 ascent puts it, the outline is a stroke of twice its width under the letters, the box surrounds each line's advance width. With the subtitles panel open the text is dragged into place (it sticks to the middle) and resized by its corner.

Keyframes

A text, a picture (or video) laid over the video, a clip and a placed sound can change over time, CapCut style: core/model/Keyframes holds keyframes, each the values of some KeyProperty at one moment and the easing to the next (smooth by default). Between two keyframes a value goes from one to the other; before the first and after the last it holds; a property no keyframe holds keeps the thing's own value. valueAt (the preview) and expression (an FFmpeg expression of the time, nested if(lt(...)) pieces with clip and smoothstep) follow the same curve.

Clocks: a text, picture or sound counts from its own start on the exported video (its keyframes move with it; dragging only its start edge, or cutting it, keeps them where they were on the video); a clip counts in its file, so its keyframes stay with the footage when it is trimmed, split or the whole video is trimmed.

What animates:

  • texts: place, size, opacity (drawtext with x, y, fontsize and alpha as expressions of t - start);
  • pictures: place, size, turn, opacity (CommandBuilder.animatedOverlay): read as a stream (-loop 1), sized once at the largest width (scale2ref), the opacity multiplied in from a 2 x 2 grey picture worked out on every frame (geq with T, stretched, blend=multiply on the alpha), turned on every frame in a square that holds it at any angle (rotate, transparent corners), shrunk on every frame (scale with eval=frame), and placed by the tight box of the turned picture, as a still one is (its shape is ImagePlacement.aspect);
  • clips: framing (Effects.zoom, panX, panY; an enlarged copy scaled frame by frame laid over the frame itself), colours (eq with eval=frame) and volume (volume with eval=frame); the picture's time is the file's (t + start, or end - t for a reversed clip, before the speed change), the sound's start + t * speed;
  • placed sounds: volume, from their start.

The editor (Keyframing.kt): the diamond next to undo (KeyframeButton) puts a keyframe at the playhead on what has the focus with its values there, or takes the one there away (the last one taken away leaves the values of that moment). Once something has keyframes the panels show its values at the playhead and a change of a value it animates goes into the keyframe there (keyedEdit), made if there is none; the preview's drags do the same. The selected item's keyframes are diamonds on the timeline; a tap on one brings the playhead there, a drag moves it (issue #31; KeyframeDrag in EditorTimeline.kt, Keyframes.moved). A drag is picked up within 12 dp of a diamond, on a clip only along the bottom of the track (higher up its edges always trim it), on a text, picture or sound on its row; it is seen before every other gesture of the timeline once the finger moves, so a tap still brings the playhead and a hold still lifts the clip. A keyframe at the start or the end sits where the edge is grabbed too: there the edge wins, unless the playhead is on the keyframe (drawn filled, within 6 dp). So a drag trims at once, and a tap on the keyframe first, which brings the playhead onto it, makes the next drag move the keyframe. With the playhead on such a keyframe the edge is still grabbed higher up the clip, or a little outside the text, picture or sound (past the keyframe's 12 dp, within the edge's 24 dp).

A trim can leave keyframes outside what they belong to (a clip's keyframes are on its file's clock, a text's, picture's or sound's from its start). They still count: the values head for them, in the preview and in the export. So they stay in view where they are, drawn after every clip and item (they may lie over the next one): hollow, their outline dashed and paler, tied to the edge they were cut off at by a dashed line. A tap brings the playhead onto one and a drag brings it back in (moveBounds always allows a keyframe its own time, so it never jumps when picked up). The keyframe takes its values, easing and curve with it, snaps to the playhead, the cuts and the edges as the other drags do, and stays inside what it belongs to and between its neighbours, MIN_GAP_MS (80 ms) from each (Keyframes.moveBounds): each keyframe's easing goes on to the next, so the animation keeps its order and two keyframes never share a moment. While it moves it is drawn larger with its time in a pill (from the start of the clip, text, picture or sound), the preview shows the frame there and a tick says when it snaps or meets a neighbour; on release it is one undo step and the playhead lands on it. The preview animates the texts and pictures (composed at their values on the clock), the clip's framing (a layer scaled and moved while drawing), its colours and the volumes (ClipLevels.gainAt, PreviewSound.volumeAt).

Subject cut-out

A key of KeyMode.SUBJECT cuts around the subject of a still picture instead of a colour: the main picture of the picture editor, its stickers, and pictures laid over the video (never a video, whose subject moves). Its mask is a PNG as large as the picture, white, whose alpha is how much of each pixel is the subject.

The models (CutoutModel, CutoutVariant; one chosen in the settings, all three can be installed): U2-NetP (fast, 320 x 320) and IS-Net (precise, 1024 x 1024) of Xuebin Qin et al. (Apache 2.0), as converted to ONNX by rembg (MIT); BiRefNet-lite (finest, 1024 x 1024) of Zheng Peng et al. (MIT), as converted by this project (below). Like the speech model they are downloaded by the browser (the app has no network access) and imported in Settings > Smart features: cutout/CutoutModelStore copies the one file, hashes it on the way and keeps it only when it matches the pinned SHA-256; it reports with the speech model's states, so SmartSettings shows them with the same card. cutout/CutoutEngine runs the chosen one through ONNX Runtime's Java API (loaded when needed, let go of after two idle minutes; each model says how much free memory it needs, and BiRefNet runs without ONNX Runtime's memory arena, which keeps its peak near 2.1 GB rather than 2.5). core/cutout/SubjectModelIo is the input and output as rembg has them (SubjectNet: the side, the mean and deviation per channel after dividing by the largest value; IS-Net's and U2-Net's output stretched to 0..255, BiRefNet's logits through a sigmoid and not stretched, so that a picture with no subject stays empty). cutout/SubjectCutter squeezes the picture (decoded as FFmpeg and the preview read it, EXIF turn ignored, at most 4096 pixels), runs the model, stretches the mask back to the picture's size and keeps it in files/cutouts, one per picture and model. Once a picture is cut the model leaves memory at once (see Models in memory).

Correcting it, and lifting a subject (cutout/RefineEngine, core/cutout/PromptSegmentation, tested). EdgeTAM (Meta, Apache 2.0), a Segment Anything 2 for phones, in two ONNX files: the image encoder (the picture stretched to 1024 x 1024 as SAM 2 does, without keeping its proportions; / 255, ImageNet mean and deviation), run once per picture, and the prompt encoder with the mask decoder (points in the square's pixels, each side stretched with the picture; labels 1 in, 0 out), which gives three masks of 256 x 256 logits over the whole picture and a score for each. MaskRefiner takes the mask that agrees with the strokes (choose), keeps what touches the stroke and brings its edge to the picture's own (GuidedFilter). SubjectLifter, behind the photo editor's long press (LiftSubject.kt, only on the picture itself, not on a canvas, nor while cropping or drawing): the picture made again at 2048 pixels with the same look (PictureRenderer), one point, chooseForTap (of the masks holding the point, the best scored that is not nearly the whole picture), the piece under the finger, the guided filter, then that piece alone on a transparent background, cut to its box, as a PNG in files/lifted/ (older than 14 days: deleted; a saved project keeps its own copy). addPlacedOverlay lays it as a sticker exactly where it was (an overlay's x and y are shares of the free space around it), one step of undo; it rises off the picture (a spring, a halo of its outline, sparks) and settles. The lift lasts as long as the finger stays down: let go before it is done and it is called off (the job cancelled, nothing laid, the PNG written meanwhile deleted), with a firm haptic. While the model looks, a badge sits under the finger (a nine-sided cookie turning slowly, a loading indicator in it) and ripples spread from it as on a pond, each born as the cookie and relaxing into a circle as it goes (MorphShape drawn as an outline); called off, the same badge springs to rest and morphs into a red four-leaf clover with a cross, the ripples turn red and are drawn back in, and it shrinks away; done, it pops away as the subject rises. Done or called off, the model and the picture it read leave memory at once (the lift holds the model only across its two calls, SubjectLifter.forget the picture); it is let go of after the step under way, never in the middle of one, and loads again for the next press. The correction dialog holds it from the first stroke to its closing, confirmed or not (see Models in memory).

What the long press leaves behind (Settings, Editor: AppSettings.liftMode, LiftMode). "Copy it" (the default) lays the sticker and leaves the photo as it was. "Cut it out" also cuts a hole where the subject was, in the photo itself, under its turns and crop: the press is taken back through the geometry steps (LiftHole.sourcePoint: each step undone, mirrors, quarter turns and the crop exactly, the keystone and the straighten by their own mappings) and the subject found there again (on the photo as it is, when it has no geometry step: the first lift is that one). Its shape covers the photo at its own size (LiftHoles.cover, LiftHole.scaled), and the hole is see-through or a colour (LiftHole.punch): see-through, its pixels keep the colour chosen for files that hold no transparency, which drop the alpha and show it (the picture is looked at anew once it takes the photo's place, replacePicture: its transparency is what lets a PNG keep the hole). The result is written by RgbaPng (Android's encoder works from premultiplied pixels, where a see-through one has lost its colour) to files/lift-holes/ and takes the photo's place (onReplacePicture). "Erase behind" gives the same shape to the magic eraser as its mask (LiftHoles.eraseMask, MagicEraseUi.onErase; the model chosen, or the other when only that one is there), and the filled-in photo takes its place, marked as any erased picture. It can be chosen only with one of the eraser's models on the phone (else its tile leads to them), and under it the model that fills in is chosen (MI-GAN or LaMa: the eraser's one choice, the same as in its panel and its settings page; one not on the phone leads to where it is set up); when none is there any more, MainActivity sets the long press back to "Copy it". This runs as a job of its own once the sticker is laid, so the finger may leave; if it fails, the sticker stays and a note says the photo behind it is unchanged. While the eraser works, waves show where (LiftFilling): over the subject's box a veil that wavy bands of the theme's colour sweep through, wavy rings spreading from its middle, and a badge with the expressive wavy progress indicator round the eraser's wand; they fade when the new photo is in, or when the eraser stops without one (then a note says so). The setting sits under its own "Photo editor" heading in Settings, Editor. The note under the picture rises above the selected sticker's actions while they show (noteBottom), so that it is not hidden under them.

A see-through hole keeps its colour only if nothing premultiplies it on the way: the photo editor reads its pictures as pixels with stillRaster (decoded with inPremultiplied = false, fitted by scaledRaster, channel by channel), so a picture shown without its transparency (a JPG export) shows the colour chosen for the hole, as the export does, not black.

The project's own ONNX files (docs/MODELS.md has the whole record). EdgeTAM is published by Meta for PyTorch only, and BiRefNet-lite's own ONNX needs more than 5.7 GB at 1024 x 1024 (its deformable convolutions written as gathers). scripts/export-edgetam.py and scripts/export-birefnet.py convert them from exactly what scripts/edgetam-onnx.lock and scripts/birefnet-onnx.lock pin (upstream commit, weights and licence, each by SHA-256), with the tools of scripts/model-export-requirements.txt (--require-hashes): nothing trained or changed, constant folding off (the same bytes on every computer), BiRefNet's deformable convolutions as ONNX's DeformConv (opset 19). Each run checks the files against PyTorch (random inputs and real photos) and writes, next to them, the upstream LICENSE, the NOTICE of the changes (Apache 2.0 section 4b) and their PROVENANCE; the licence and the notice also go inside each file (doc_string, metadata). .github/workflows/models-onnx.yml builds them twice (two runner images, the second on two cores) and publishes only byte-identical files, as the pre-releases named by the locks' release_tag; the app downloads them from there. scripts/check-model-pins.py (run by release.yml for every tag, with --online) checks that the Kotlin pins, the locks, the licences and notices bundled in assets/licenses/, the release files and docs/MODELS.md all agree.

The sticker gallery (stickers/StickerLibrary, stickers/StickerShelf). A picture saved as a sticker is decoded as the editors read it (at most 2048 pixels), cut by its key when it has one (the subject's mask, brush corrections included, or a colour; the fill made transparent) after its own colours (keyedBitmap), trimmed to what is left and written as a PNG in files/sticker-library/, with a record beside it (<id>.sig): the signature of what it was made from (core/stickers/StickerSignature, tested: the picture's SHA-256, the key's settings and its mask's bytes, the colours) and its own file's SHA-256. savedAs finds a picture cut the same way (or a sticker laid as it is) among them: the editors' bookmark is outlined until it is saved, filled once it is (and then takes it away), and a save of what is there already writes nothing. With AppSettings.stickerCopies (off by default) a copy goes to Pictures/FFmpegGUI/Stickers through MediaStore (PublicCopies, no permission). MainActivity provides the gallery as LocalStickerShelf (its list, save and delete); a save is confirmed by a haptic only, and a short message says when it could not be made. Saved from: the floating actions of a selected sticker or picture (a subject just lifted with a long press is selected, its actions in view), and the Cut out panel (ColorKeyContent's onSaveSticker). The photo editor's Sticker and the video editor's Overlay open StickerSheet first ("Add on top"): two connected tiles take it from the phone, the gallery (the system photo picker, PickMultipleVisualMedia: pictures and videos in the video editor, pictures in the photo editor) or any file (OpenMultipleDocuments); under them the saved stickers on soft tiles that morph under the finger (a tap lays one). "Edit" (or a long press) sets them all wobbling out of step, each with a button to take it away. A sticker deleted from the gallery stays in the projects that use it (they keep their own copy).

Retired models (cutout/RetiredModels). A model the app no longer uses (SlimSAM, before EdgeTAM; Parakeet's SmoothQuant build, no longer offered) is deleted at start when it is still on the phone, from its folder and its settings; what was deleted waits in retired_models until RetiredModelsDialog has said so once (which model, what replaced it or why it went, the space freed, a way to set up the new one). The list only grows; the pin check fails if a retired folder is ever one in use. Files and settings the app no longer reads inside models still in use (the speech model's test recordings, the speech build once chosen) go at the same time, without a word.

ONNX Runtime comes twice, inside the sherpa-onnx AAR (1.28.2) and with its Java API (1.28.0): the build keeps one libonnxruntime.so (pickFirsts), which both reach through the same C API version.

Export. subjectFilters (KeyFilters.kt) reads the mask with movie, takes its alpha with alphaextract, puts it through the steps that turned, mirrored and cut the picture before the key (the picture editor's geometry stages; an overlay's mirrors), sizes it to the picture with scale2ref and merges it in with alphamerge (which repeats its one frame for a looped picture). The fill, the see-through fill and an overlay's own transparency then work as for a colour key.

Preview. ImageStage.Key gets the mask as a raster as large as the picture it starts from (SubjectMasks), puts it through the same geometry stages (kept from the last time while they stay the same) and uses it as the key's alpha in ColorKeying.apply. Stickers and pictures laid over the video are cut on the CPU (keyedBitmap) with the mask read at their size; the GPU shader keys by colour only.

Panel. ColorKeyContent shows a card to cut the subject out where it gets the still picture's uri (subjectUri); LocalSubjectCut (provided by MainActivity) carries the model's state, where the view model's cutOutSubject is, and the actions. A subject key keeps only the fill.

Command studio

The full-screen editor for a hand-written command (app/command), opened from the command panel of every editor. Its rule: it never types for the user. Suggestions sit in a strip above the keyboard and only a tap, or Tab on a keyboard, takes one; whatever the studio changes on request (a recipe, a history entry, laying the command out, renaming a file) can be undone.

  • What FFmpeg knows (core/command/studio/FfmpegCatalog.kt): the bundled FFmpeg's own answers to -h full, -filters, -encoders, -muxers and -pix_fmts, read by FfmpegHelpParser into options (with their AVOption values, defaults and ranges), filters, encoders, formats and pixel formats. -h filter=x, -h encoder=x and -h muxer=x are asked when needed. The answers are kept in files/ffmpeg_help/<install time>/, so FFmpeg is asked once per installed version; nothing goes online. Before the first answer a built-in list of everyday options (FfmpegCatalog.COMMON) is used, which also ranks the suggestions.
  • Reading the command (CommandReading): spanCommandLine splits it as a shell would and keeps where every argument is; each one is the program, an option, an option's value or a file on its own. A backslash before a line break joins two lines, so a command laid out on several lines runs as is.
  • Suggestions (CommandAssist): for the word at the cursor, by context: options (the everyday ones first, then FFmpeg's), stream specifiers, the value an option takes (encoders of the right kind, formats, pixel formats, AVOption constants, flags with a +, usual sizes, rates and bitrates), inside a filtergraph the filters of the right kind, their options and the values of those, labels and inputs, then the project's file names. With nothing typed: what usually comes next. applySuggestion replaces exactly the word the suggestion names.
  • Checker (CommandLint): unclosed quotes and brackets, options missing their value, unknown options, filters and encoders (only once FFmpeg has been read), inputs that do not exist (-map 3, [2:v]), files that are not the project's, no input, no output. It marks; a fix (the missing output) is offered, never applied.
  • Recipes (CommandRecipes.kt): ready commands filled with the project's files; one needing an encoder this FFmpeg lacks says so.
  • Files: besides the project's own names, files added in the studio are copied into files/command_files/ and named for the command (CommandFile, kept with the project). CommandMode.Manual carries them and the engine stages them like any source before ManualCommandResolver swaps their names for paths (a name of the project's wins).
  • History and favourites: CommandStudioStore, in the app's preferences.

FFmpeg runtime

The original FFmpegKit was retired and its binaries removed from Maven Central. The project uses FFmpegKitNext 6.1.1 (arthenica/ffmpeg-kit-next), the official continuation by FFmpegKit's author: LGPL 3.0, GPL 3.0 with the GPL libraries enabled, same com.arthenica.ffmpegkit API (now written in Kotlin), FFmpeg n6.1.6. It publishes source only, so the AAR is built here, from pinned sources, and never downloaded:

  • scripts/ffmpeg-kit-next.lock pins FFmpegKitNext and every source its Android build may download (FFmpeg and each external library) to a commit. The commits of FFmpeg, libass, dav1d, freetype, harfbuzz, fribidi, opus, vorbis, ogg, zimg, vid.stab, expat, libpng, cpu_features and x264 were checked against the upstream projects' own repositories.
  • scripts/build-ffmpeg-kit-next.sh clones FFmpegKitNext at the locked commit, checks that the lock says what its scripts/source.sh asks for, fetches the sources the build needs and checks each commit before anything is compiled, runs nix-android.sh -p android-r27d (the toolchain, Android SDK, NDK r27d and CMake, pinned by FFmpegKitNext's flake.lock), checks every source the build used again and only then copies the AAR to media/libs/ffmpeg-kit-next-6.1.1.aar, with a .provenance.txt beside it (its SHA-256, the commits, the options). media/libs/ is ignored by git.
  • What could slip past those checks is closed off too:
    • the script runs again with a clean environment (KEEP_ENV): FFmpegKitNext reads version_<lib>, REDOWNLOAD_<lib>, SKIP_ffmpeg, OK_<lib>, CFLAGS and more from the environment, nix develop passes it through, and git reads GIT_DIR; none of them reaches git or the build;
    • FFmpegKitNext reuses any library left in its prebuilt/ without asking which source built it, so prebuilt/ is emptied whenever a source is fetched again, the lock, the options or the script change, or a check after the build fails (prepare_prebuilt, stamp in prebuilt/.build-ffmpeg-kit-next-inputs);
    • every source is put back exactly as its commit before each build (git reset --hard, git clean -ffdx), so a file changed by hand never gets built;
    • FFmpegKitNext's nix.conf sets accept-flake-config = true; the build runs with NIX_CONFIG="accept-flake-config = false", so a nixConfig added to its flake.nix one day (extra binary caches) is not taken;
    • the work folder must be new or empty the first time; --clean only deletes a folder that holds the script's marker (.build-ffmpeg-kit-next), or one that holds FFmpegKitNext's clone alone (made by an earlier version);
    • git runs without the user's and the system's configuration (GIT_CONFIG_GLOBAL=/dev/null, GIT_CONFIG_NOSYSTEM=1): no hook or core.fsmonitor command of theirs runs on the sources;
    • FFmpegKitNext's own tracked files are put back as its commit at each run (its build edits build.gradle, the script adds a line to gradle-wrapper.properties);
    • the AAR is packed by FFmpegKitNext's Android project with Gradle, which neither Nix nor the commits pin. The Gradle distribution is pinned by the gradle line of the lock: the script adds its SHA-256 as distributionSha256Sum, so the wrapper checks the zip it downloads, and drops a distribution it did not see installed with that check (the wrapper checks downloads only, and deletes the zip once unpacked). The Android Gradle plugin, Kotlin, Dokka and every library they bring are pinned by scripts/ffmpeg-kit-next.verification-metadata.xml (Gradle's dependency verification), copied into the project before each build; it is made with --write-gradle-verification (trusted on first use: read it before committing). The project's gradle-wrapper.jar is Gradle 8.13's official one, and Gradle's home is inside the work folder, so ~/.gradle does not take part;
    • that Gradle home is set up anew each run: its gradle.properties only sets org.gradle.dependency.verification=strict (nothing can make the verification lenient or turn it off), and its only init script keeps the project's unit tests from running. They do not change the AAR, and a test framework can download what it needs outside Gradle's verification (later FFmpegKitNext releases use Robolectric, which fetches Android jars by itself and trusts any copy in ~/.m2). What the tests compile is still resolved and checked by Gradle.
  • settings.gradle.kts serves that AAR as com.arthenica:ffmpeg-kit-next, from media/libs only (exclusiveContent). verifyFfmpegKitNext (in media/build.gradle.kts, before every build) fails if the AAR is missing, lacks one of the two ABIs, or if FFmpeg's libraries are not aligned for 16 KB pages. Its only dependency, smart-exception-java, comes from Maven Central.
  • The build enables only what the app uses: x264, x265, xvidcore (GPL), lame, vorbis, opus, libass with fontconfig, freetype, fribidi and harfbuzz (subtitles and drawtext), vid.stab, zimg (zscale), dav1d (AV1 sources: FFmpeg 6.1 has no AV1 software decoder of its own), Android MediaCodec (h264_mediacodec) and the system zlib. Nothing network-related (no gnutls, openssl or srt). A hand-written command can use any of FFmpeg's built-in codecs and filters, but not libraries left out (libvpx, libaom, ...): add one with scripts/build-ffmpeg-kit-next.sh -- --enable-libvpx; its source is checked against the lock like the others.

The first build can take an hour or more and needs Nix on Linux (not NixOS) or macOS; later builds reuse the checked sources in .ffmpeg-kit-next/.

The AAR the app is built with is pinned, and reproducible.

  • .github/workflows/ffmpeg-aar.yml runs after a push to master that changes the AAR's inputs (scripts/ffmpeg-kit-next-inputs.sh: the lock, the build script, the Gradle verification file), and stops at once if the pin is already current. It builds the AAR twice, on ubuntu-24.04 and on ubuntu-26.04 limited to two CPU cores, and publishes it as an asset of a pre-release named after its SHA-256 only when both are byte-identical (otherwise it uploads a diffoscope report). It then commits the pin, unless master has moved on to other inputs meanwhile (that push's own run pins those).
  • scripts/ffmpeg-kit-next-aar.txt pins that asset (URL, SHA-256 and the hash of its inputs); scripts/fetch-ffmpeg-kit-next.sh downloads and checks it. verifyFfmpegKitNext refuses any other AAR unless -Pffmpeg.unpinned=true (local experiments only).
  • Nothing in the AAR depends on when it was built: FFmpegKitNext compiles the build day into libffmpegkit.so (FFMPEG_KIT_BUILD_DATE from date), so the script writes the day of FFmpegKitNext's commit instead (pin_build_date) and exports SOURCE_DATE_EPOCH as that commit's time.
  • FFmpeg keeps its configure line, with the work folder's path, in its libraries: the published AAR is built in /var/tmp/ffmpeg-kit-next (CANONICAL_WORK), and a rebuild there gives the same file. The script says at the end whether its AAR is the pinned one.

Releases publish only what IzzyOnDroid's reproducible builds will accept, and the release key stays on the maintainer's machine:

  1. .github/workflows/release.yml (every v* tag) refuses the tag if its pinned AAR was built from other inputs, builds the unsigned APK in the environment IzzyOnDroid's recipes use (debian:bookworm-slim, OpenJDK 17, command line tools 12.0), has IzzyOnDroid's own builder, rbtlog (pinned commit, docker, default limits: 10 + 10 minutes of provisioning, 20 minutes of build), rebuild it from .github/izzyondroid/, and only if both are byte-identical puts it in a draft release whose notes give its SHA-256 (unsigned-sha256:).
  2. The maintainer downloads that APK, signs it on their machine with apksigner sign --alignment-preserved true --v1-signing-enabled false (the signed APK is the unsigned one plus the signature), attaches it to the draft as ffmpeg-gui-<tag>.apk in place of the unsigned APK and publishes the release.
  3. .github/workflows/verify-release.yml (release published) runs rbtlog on the recipe as it is: it downloads the published APK, rebuilds the tag and compares, as IzzyOnDroid's server will.
  • It ships arm64-v8a and x86_64 only, so app/build.gradle.kts limits the APK to those ABIs (no 32-bit devices).
  • It has no WebP encoder (libwebp is not built in): WEBP pictures are written by Android from a PNG that FFmpeg makes (see "Image files").
  • FFmpegKitNext's classes are Kotlin: from Kotlin, their values are read with their getters (session.getReturnCode(), info.getTags()), not as properties.
  • An earlier choice, io.github.jamaismagic.ffmpeg:ffmpeg-kit-main-full-gpl-16kb, cannot be loaded: its libavdevice.so references SDL's PLATFORM_hid_* and OpenSL ES symbols that no library provides (dlopen fails, FFmpegKit throws "failed to start"). Before changing runtime, check every .so for undefined symbols that neither its dependencies nor the Android system libraries define.

The version number selects the FFmpeg line (6.1.x = FFmpeg 6.1, 7.1.x, 8.1.x, 9.0.x): the filter graphs here were verified against FFmpeg 6.1 (scale2ref in particular is deprecated in later majors), so moving line means re-running the command checks. To move: change the kit line of the lock, regenerate its lib lines from that release's scripts/source.sh (the build script refuses a lock that disagrees with it), check them, and change ffmpegKitNext in gradle/libs.versions.toml.

Only :media references it (FfmpegKitExecutor, AndroidMetadataReader, FfmpegFonts, FfmpegInfo). Switching runtime means implementing FfmpegExecutor again and changing one dependency.

Upgrading the toolchain

The build uses AGP 9.1 / Gradle 9.3.1 / compileSdk 37 (targetSdk 36), which Compose 1.12 and material3 1.5 require; it opens in Android Studio Panda 2 (2025.3.2) or newer. AGP 9 compiles Kotlin itself, so the modules apply no kotlin-android plugin; :core stays a plain kotlin-jvm module and :app applies the Compose compiler plugin.

Next steps, which have to move together:

  1. Compose 1.13+ needs compileSdk 37.1: raise compileSdk and AGP to a version that supports it (and the Gradle wrapper to the version that AGP requires).
  2. Then raise composeBom, lifecycle (2.11+), navigationCompose (2.10+), coreKtx, media3, and material3 to 1.5.0 stable once it is published.

Hidden corners

A few things for whoever pokes where they need not. None ever starts by chance while working, none touches a project or an export, none needs the network, and each keeps to the vibration setting and to the system's animations (off: they stand still, or do not show).

  • Life, by FFmpeg (settings/AboutLife.kt, core/eggs/LifeGame, media/FfmpegLife, tested). Seven taps on FFmpeg's version in About, as the build number unlocks Android's developer options (from the fourth, the row counts down). FFmpeg's own life source computes Conway's Game of Life on a 32 x 20 grid with its edges joined, 48 generations a run as raw grey bytes, each run going on from the last grid (written as text for the source's filename); a game that dies or only blinks is shuffled again. A card under the row shows it (cells born grow from a dot into a rounded square, those dying shrink away, the badge changing shape every generation), the page's background plays it faintly, tiled; a tap on the grid drops a glider, from which FFmpeg goes on. The first generations tick softly, then it plays silently; leaving the page ends it, and the run's files go at once (and with the cache janitor).
  • The DJ's scratch (DjScratch.kt, core/eggs/ScratchDetector, tested). Dragging the video editor's timeline back and forth quickly and widely (four turns within 1.2 seconds, each swing at least 56 dp) turns the play button into a vinyl for a moment: it follows the finger, ticks at every turn, spins on by itself and springs back into the button once the finger has rested. The playhead moves as any scrub moves it; nothing else changes.
  • The infinite zoom (EditorTimeline). Pinching on past the widest zoom squeezes the ruler and has it count days, then years, centuries, ice ages, eras and eons (a tick for each new span); it springs back to the video's own time when the fingers leave.
  • The command studio's small talk (command/StudioReply.kt, core/command/studio/CommandJokes, tested). What is typed at terminals gets a reply in a bubble that springs out under the header, read out by TalkBack; such a command can never be run (Run is off), and a real FFmpeg command is never taken for one. A few classics match as the whole command (rm -rf of the root, make me a sandwich, :wq, git status, xyzzy, coffee, 42...); a few programs get their own reply whatever follows them (cat, pwd, cd, python, apt, dnf, pacman, rm, ls, sl, clear, top, exit, sudo and sudo please, vim, kill, whoami, man, ping, history, echo, reboot, nmap, grep, ssh and su); any other program a terminal would run (a lowercase word first, not FFmpeg's nor on its way to being typed as one) gets the general one ("bruh you're not on termux my guy"). The cat's reply is a kaomoji, written in strings.xml as escaped code points so the file stays ASCII.

A foreground service that must take the foreground. An export that ends at once (a command that fails straight away) used to stop ExportService before it had called startForeground, which Android answers by closing the app (ForegroundServiceDidNotStartInTimeException). The service now always takes the foreground when it starts (a quiet notification when the export is over already) and then lets go; ExportService.stop before that only asks it to let go as soon as it has.

Known limitations (backend)

  • Transitions between clips (xfade and acrossfade, see core/model/Transition.kt and transitionJoin in CommandBuilder.kt) need the clips re-encoded: stream copy ignores them, as it ignores every filter. The transitions told as layers are cross-faded in an HDR video (10 bits) and, in the preview, before Android 13.
  • reverse keeps a whole clip in memory while it is reversed: very long reversed clips can exhaust it on a phone.
  • Stream copy can only cut and split on key frames, and with several clips it needs them to share codec and resolution (concat demuxer).
  • The preview switches clips through the player's playlist, so a cut can show a short pause that the exported file does not have.
  • Stabilization, and the video's sharpen, blur, vignette and film grain are implemented and tested in :core and have view model setters, but the video editor has no controls for them yet (the denoise has its own tool, see "Denoise") (the photo editor has its own sharpen, blur, vignette and grain, see "Image files"; splitting and tags have their panels, see "Audio files").

Keep Android Open

The home shows, under the tagline, one card for Keep Android Open (keepandroidopen.org), the campaign against Google's registration of every Android developer (KeepAndroidOpen.kt). Folded, it is one row as small as the home's other tiles (a little phone whose apps a padlock greys out, then springs open, three times, then still; the name; one line), so that the app stays the first thing on the screen. A tap unfolds what the registration means for an app like this one (its key words in bold: installing it gets harder, and one day may not be possible at all), what it means for this one (it will not register with Google: once the lockdown reaches a country it installs only through the extra steps Google allows, while it allows them, or on an Android without Google such as GrapheneOS), and a button to the campaign's site, opened in the browser (nothing is fetched by the app). Its cross hides it for good (AppSettings.keepAndroidOpenBanner); Settings, Appearance, "Home" brings it back.

Links

Every link of the app says where it goes before it is followed: under it, LinkDestination (ui/expressive) writes "Opens :

", the site named as people call it (GitHub, Hugging Face, Zenodo...). A settings row that opens a page passes it as its link; a button has the line under it. The pages open in the browser (openLink): the app itself never goes online.