Migrate the Built-In Emoji Catalog From JSON to FlatBuffers for the Next Major Version

[wax911]

Problem

The built-in emoji catalog is currently shipped as emoticons/emoji.json. EmojiManager.create(context, serializer) opens that asset and deserializes the complete dataset into runtime objects before the manager can be used.

The emoji dataset is immutable, generated by this repository, and known before the library is published. Runtime JSON parsing and reconstruction of lookup structures therefore perform work on every consumer device that can instead be completed once during library generation.

This migration is intentionally a major-version change because the default initialization API, serializer dependency model, packaged asset format, and internal catalog architecture may change.

Prerequisite

This issue is blocked by Establish AndroidX Benchmark Baselines for Emoji Initialization and Parsing.

Do not claim performance improvements until the baseline benchmark suite and baseline report from that issue exist and can be rerun unchanged against this migration.

Goal

Replace the built-in runtime JSON catalog with a generated FlatBuffers catalog so that the default Emojify path:

• does not parse JSON at runtime
• does not require Gson, Moshi, or kotlinx.serialization to load the built-in catalog
• does not eagerly materialize the complete emoji dataset as a Kotlin object graph
• does not rebuild deterministic lookup structures from the same immutable catalog on every device
• remains lazy so applications that never use Emojify perform no catalog-loading work during startup
• preserves existing public parsing and lookup behaviour unless a breaking change is explicitly documented as part of the major-version migration

Architectural Target

The intended data flow is:

Upstream emoji sources
        |
        v
Repository generator
        |
        +--> normalized emoji records
        +--> shortcode index
        +--> tag index
        +--> Unicode lookup structure
        |
        v
Generated FlatBuffers catalog
        |
        v
Packaged library asset
        |
        v
Lazy read-only ByteBuffer backed catalog
        |
        v
EmojiManager

The runtime representation must be read-only. Do not introduce a database, cache database, or writable copy of the built-in catalog.

Required Design Constraints

1. Preserve behaviour before optimising fields

The first FlatBuffers implementation must preserve the current observable values and parser behaviour for:

• emoji
• description
• supportsFitzpatrick
• tags
• unicode
• htmlDec
• htmlHex
• shortCodes

Do not remove, derive, or change the semantics of htmlDec or htmlHex in this migration. That work is intentionally isolated in a separate optimisation issue so storage migration and behavioural optimisation can be measured independently.

2. Separate manager behaviour from catalog storage

Introduce a catalog abstraction so EmojiManager is not responsible for decoding a storage format.

The exact visibility can be internal unless a public extension point is justified, but responsibilities must be separated conceptually:

EmojiManager
    |
    v
EmojiCatalog
    |
    +--> current JSON implementation during migration
    |
    +--> FlatBuffers implementation

Use the JSON implementation only as a temporary parity/reference path during migration. It must not remain the default built-in runtime path in the completed major version.

3. Remove mandatory serializer coupling from the built-in path

The next major version must provide a first-class built-in initialization API that does not require consumers to choose a JSON serializer.

The expected consumer shape is equivalent to:

val manager = EmojiManager.create(context)

or:

val manager = EmojiManager.default(context)

The final public name must follow existing API naming conventions and Dokka style.

Before removing or repurposing IEmojiDeserializer and the :serializer:gson, :serializer:moshi, and :serializer:kotlinx artifacts, audit repository and published API usage. Record the disposition of each artifact in the migration document. Since this is a major version, breaking removal is allowed, but it must be intentional and documented rather than incidental.

Do not make :emojify depend on one of the existing JSON serializer modules.

4. Generate the FlatBuffers artifact in this repository

Extend the existing scripts/emoji_generator pipeline. Consumers must not run code generation for the built-in catalog.

Generation must be deterministic for identical normalized source input.

The generator must produce:

• FlatBuffers emoji records
• shortcode lookup data
• tag lookup data
• Unicode lookup data required by the parser
• schema/version metadata needed to validate compatibility at runtime

The generated binary must never be edited manually.

5. Precompute deterministic lookup structures

The current implementation lazily constructs lookup maps and an EmojiTrie from the full emoji collection. The FlatBuffers migration must evaluate and implement precomputed equivalents rather than rebuilding the same deterministic structures on-device.

At minimum precompute:

• shortcode to emoji identifier mappings
• tag to emoji identifier mappings
• Unicode sequence lookup needed for longest-match parsing

The runtime lookup path should operate on compact identifiers or buffer offsets and create public-facing IEmoji views only when an API actually returns an emoji.

Do not eagerly allocate one Kotlin wrapper object per catalog entry during manager creation.

6. Asset access

Package the FlatBuffers runtime asset so it can be accessed efficiently from Android resources.

Prefer a read-only ByteBuffer backed path. If an uncompressed asset and file mapping are used, configure packaging explicitly and validate release APK/AAB behaviour.

If memory mapping is not portable across all supported Android versions or packaging modes, provide a documented read-only buffer fallback. Do not silently fall back to JSON.

7. Lazy initialization

The default built-in manager must not require AndroidX Startup or any manifest ContentProvider initialization.

Creating the application process without using Emojify must not parse, map, or materialize the catalog.

The first operation that actually requires catalog data may trigger lazy catalog opening. Thread safety must be deterministic and tested.

Implementation Phases

Phase 1: Baseline and API inventory

• Confirm issue 539 is complete and record the baseline report commit.
• Inventory current public APIs related to EmojiManager, IEmoji, and IEmojiDeserializer.
• Inventory published serializer artifacts and repository consumers.
• Record which APIs and artifacts are retained, replaced, deprecated, or removed for the major version.
• Add a migration document under docs/ before breaking API work begins.

Gate: Do not modify the storage implementation until the API and artifact disposition is written down.

Phase 2: Catalog abstraction with JSON parity

• Introduce the catalog abstraction.
• Adapt the current JSON implementation behind that abstraction.
• Keep existing tests green without changing parser semantics.
• Add parity tests that can execute the same lookup and parser assertions against multiple catalog implementations.

Gate: Existing behaviour must be reproducible through the abstraction before FlatBuffers becomes involved.

Phase 3: FlatBuffers schema and deterministic generation

• Add the FlatBuffers schema to a clearly owned repository location.
• Extend scripts/emoji_generator to emit the binary catalog.
• Add generation validation for required fields and index integrity.
• Add a deterministic-generation test or checksum-based verification for identical input.
• Add schema/version metadata and reject unsupported catalog versions with an actionable failure.

Gate: Generated output must be deterministic and independently testable before Android runtime integration.

Phase 4: Read-only runtime catalog

• Implement the buffer-backed EmojiCatalog.
• Avoid eager creation of one IEmoji object per record.
• Implement shortcode lookup using generated index data.
• Implement tag lookup using generated index data.
• Implement Unicode lookup using generated data.
• Preserve longest-match behaviour for multi-code-point and ZWJ emoji sequences.
• Preserve Fitzpatrick behaviour.
• Add malformed/version-mismatch handling.

Gate: The complete existing parser and manager test suite must pass against both JSON reference and FlatBuffers implementations.

Phase 5: Make FlatBuffers the built-in default

• Add the new serializer-free built-in initialization API.
• Switch the sample application to the new default path.
• Remove AndroidX Startup from the required/default integration path.
• Remove the bundled JSON asset from the runtime artifact once parity is proven.
• Apply the Phase 1 decision for legacy serializer APIs and artifacts.
• Update README, Dokka, samples, migration notes, and dependency instructions.

Gate: A new consumer must be able to use the built-in catalog without adding a JSON serialization library.

Phase 6: Benchmark and release validation

• Rerun the unchanged benchmark suite from issue 539 against the FlatBuffers implementation.
• Compare startup control, eager legacy baseline, lazy first use, warm parsing, lookup, memory, and allocation results.
• Record binary asset size and resulting AAR/APK contribution.
• Record whether the asset is compressed or uncompressed in the final package.
• Verify minified release behaviour.
• Verify the supported Android API range.
• Publish before/after results under docs/benchmarks/.

Do not claim the migration is faster if the benchmark data does not support that conclusion. Report regressions explicitly.

Long-Horizon Execution Protocol

This issue is expected to span several implementation sessions and may be handled by different agents.

At the start of every session:

1. Read this issue from the beginning.
2. Read the latest progress comment.
3. Identify the current phase and the first unchecked item in that phase.
4. Verify the branch and HEAD commit before editing.
5. Do not begin a later phase while an earlier phase gate is unresolved.

At the end of every session, add or update a progress comment containing exactly these state categories:

• Current phase: phase number and name
• Completed: checklist items completed in this session
• Changed: files and modules changed
• Evidence: commands, tests, benchmark results, or generated checksums
• Decisions: architecture decisions made and where they are documented
• Pending: next unchecked item
• Blockers: unresolved failure or decision

When handing work to another agent, the latest progress comment is the handoff contract. Do not rely on unstated chat context.

Required Test Coverage

At minimum preserve or add tests for:

• catalog record count parity
• every generated record maps to the expected IEmoji fields
• shortcode lookup parity
• tag lookup parity
• Unicode lookup parity
• longest-match Unicode sequences
• ZWJ sequences
• Fitzpatrick modifiers
• aliases and shortcodes
• decimal HTML parsing and conversion
• hexadecimal HTML parsing and conversion
• extraction and removal APIs
• empty and invalid input
• concurrent first access
• unsupported FlatBuffers schema/catalog version
• corrupted or truncated catalog failure behaviour

Prefer table-driven parity tests that run against both the legacy JSON reference catalog and FlatBuffers catalog during migration.

Acceptance Criteria

This issue is complete only when:

1. The default built-in catalog no longer reads or deserializes JSON at runtime.
2. The default built-in path requires no Gson, Moshi, or kotlinx.serialization dependency.
3. No AndroidX Startup initializer is required for correct library operation.
4. Applications that never use Emojify perform no catalog initialization work during startup.
5. The FlatBuffers catalog is generated deterministically by repository tooling.
6. Shortcode, tag, and Unicode lookup structures are generated ahead of runtime rather than reconstructed from the full catalog on-device.
7. Existing parser behaviour passes parity tests, including multi-code-point, ZWJ, and Fitzpatrick cases.
8. The runtime does not eagerly create one Kotlin emoji object per catalog entry.
9. The benchmark suite from issue 539 is rerun without changing its methodology and before/after results are committed.
10. Consumer documentation and Dokka describe the new major-version initialization path and migration steps.
11. The disposition of legacy serializer APIs and published serializer artifacts is explicitly documented.
12. Release/minified builds pass the project validation suite.

Non-Goals

• Do not remove or derive htmlDec or htmlHex as part of this issue.
• Do not design a proprietary binary format before FlatBuffers has been implemented and benchmarked.
• Do not add Room, SQLite, or a writable cache for the built-in catalog.
• Do not require consumer-side KSP, Gradle code generation, or FlatBuffers generation for the built-in dataset.
• Do not mix unrelated parser algorithm rewrites into the storage migration unless required for generated Unicode lookup parity.

Release Requirement

Ship this work only as the next major version of the library. The release notes must contain a dedicated migration section covering dependency changes, initialization changes, removed/deprecated APIs, and any published artifact changes.