Evaluate Deriving htmlDec and htmlHex Instead of Storing Them in the Emoji Catalog

[wax911]

Problem

The generated emoji catalog currently stores htmlDec and htmlHex for every emoji record.

The repository generator already computes both values deterministically from the NFC-normalized emoji sequence:

htmlDec = "".join(f"&#{ord(char)};" for char in emoji)
htmlHex = "".join(f"&#x{ord(char):x};" for char in emoji)

These fields may therefore duplicate information already represented by the emoji/code-point sequence. With the FlatBuffers migration, retaining both strings in every record may increase binary size without providing enough runtime benefit to justify the duplication.

This issue is an evaluation first. Do not assume the fields should be removed until full-catalog equivalence and benchmark evidence support that decision.

Dependencies

• Baseline measurements come from Establish AndroidX Benchmark Baselines for Emoji Initialization and Parsing.
• The storage candidate comes from Migrate the Built-In Emoji Catalog From JSON to FlatBuffers for the Next Major Version.

This issue may begin once issue 540 has a parity-complete FlatBuffers catalog implementation. Do not change the production FlatBuffers schema before the evaluation phase is complete.

Goal

Determine whether htmlDec and htmlHex should remain persisted catalog fields or should instead be derived from the canonical emoji/code-point representation when requested.

If derivation is adopted, preserve the existing observable API and conversion output unless the next-major migration explicitly documents a separate breaking API decision.

Questions This Issue Must Answer

1. Are the stored htmlDec and htmlHex values fully derivable for every catalog entry?
2. Which canonical source should drive derivation: the stored emoji string, a normalized code-point sequence, or another already-required FlatBuffers field?
3. Does derivation preserve exact existing formatting for decimal and hexadecimal entities?
4. How many bytes are saved in the FlatBuffers catalog, AAR, and representative APK when the strings are not stored?
5. What CPU and allocation cost is added to parseToHtmlDecimal(...) and parseToHtmlHexadecimal(...)?
6. Is caching derived values necessary, and if so, does the cache erase the memory benefit?
7. Can IEmoji.htmlDec and IEmoji.htmlHex remain computed properties without forcing eager materialization?

Required Investigation

1. Establish current semantics

Audit all repository usage of:

• IEmoji.htmlDec
• IEmoji.htmlHex
• parseToHtmlDecimal(...)
• parseToHtmlHexadecimal(...)
• generator functions that create these values

Document whether any code depends on formatting details such as:

• lowercase versus uppercase hexadecimal digits
• &#x...; prefix and suffix formatting
• entity ordering for multi-code-point emoji
• variation selectors
• zero-width joiners
• keycap sequences
• regional indicators
• Fitzpatrick modifiers

Do not infer expected formatting. Derive it from the existing generated catalog and tests.

2. Prove full-catalog equivalence

Create a test or analysis tool that processes every emoji record in the current catalog.

For every record:

1. derive decimal HTML entities from the proposed canonical runtime representation
2. derive hexadecimal HTML entities from the same representation
3. compare both results byte-for-byte with the currently generated htmlDec and htmlHex

The evaluation must report:

• total records checked
• decimal mismatches
• hexadecimal mismatches
• representative mismatch details if any exist

A single unexplained mismatch blocks removal of the persisted fields.

Be careful with Unicode representation. Do not derive entities by iterating UTF-16 Char values as if each Char were a Unicode code point. Supplementary emoji must remain single Unicode code points when producing HTML entities.

3. Evaluate canonical source data

The current generator normalizes the emoji sequence with NFC before computing unicode, htmlDec, and htmlHex, while the public emoji field originates from the source record.

Verify full-catalog normalization equivalence before assuming emoji alone is a safe derivation source.

Prefer deriving from a canonical code-point representation that is already required by the FlatBuffers Unicode lookup implementation if that avoids duplicated normalized strings.

Do not add a new persisted field merely to remove htmlDec and htmlHex unless the resulting representation is demonstrably smaller or simpler.

4. Measure storage impact

Build two otherwise-equivalent FlatBuffers catalogs:

• Stored: includes htmlDec and htmlHex
• Derived: omits their payload values and derives them at runtime

Record for both variants:

• raw FlatBuffers asset size
• compressed size if applicable
• published AAR size contribution
• representative release APK or APK split contribution
• any effect on string-table or duplicated-data size

Use the same normalized emoji input and the same FlatBuffers schema generation toolchain for both comparisons.

5. Measure runtime impact

Reuse the benchmark methodology from issue 539.

At minimum compare stored versus derived implementations for:

• warm parseToHtmlDecimal(...)
• warm parseToHtmlHexadecimal(...)
• short input with one emoji
• mixed input with several emoji
• ZWJ and multi-code-point sequences
• longer mixed-content input
• repeated occurrences of the same emoji

Record timing and allocation differences.

Do not add a global cache solely to improve benchmark results. If caching is evaluated, benchmark and report it as a separate variant with its steady-state memory cost.

Decision Gate

After equivalence, size, and runtime evidence are complete, produce a short decision record under docs/ containing this table:

Variant	Catalog Size	Packaged Size	HTML Parse Cost	Allocation Cost	API Impact	Complexity
Stored
Derived
Derived + cache, only if evaluated

Then state one recommendation:

• KEEP STORED
• DERIVE ON ACCESS
• DERIVE WITH BOUNDED CACHE

The recommendation must be justified by measured evidence, not by the assumption that fewer fields are automatically better.

If equivalence fails, stop and recommend KEEP STORED unless the mismatch is proven to be a bug in the current generated data and changing it is explicitly approved.

Implementation, Only If Derivation Is Accepted

If the evidence supports derivation:

• Stop emitting htmlDec and htmlHex payloads from the runtime FlatBuffers catalog.
• Preserve schema compatibility rules required by the chosen FlatBuffers evolution strategy.
• Implement code-point-correct decimal entity generation.
• Implement code-point-correct lowercase hexadecimal entity generation matching current output.
• Keep IEmoji.htmlDec and IEmoji.htmlHex as computed properties unless their public removal was separately approved for the major version.
• Update parseToHtmlDecimal(...) and parseToHtmlHexadecimal(...) without changing their external behaviour.
• Update generator validation so future emoji data must pass derivation parity checks.
• Rerun the complete parity and benchmark suites.
• Update Dokka and migration documentation if implementation details affect consumers.

Long-Horizon Execution Protocol

At the start of each session:

1. Read this issue and the latest progress comment.
2. Confirm the state of issues 539 and 540.
3. Identify whether the task is in investigation, measurement, decision, or implementation.
4. Resume from the first incomplete checklist item rather than restarting the analysis.

At the end of each session, update the progress comment with:

• Stage: investigation, measurement, decision, or implementation
• Completed: exact completed items
• Evidence: tests, file sizes, benchmark results, or mismatch counts
• Changed: files modified
• Decision status: pending, keep stored, derive on access, or derive with bounded cache
• Pending: next action
• Blockers: unresolved issue requiring maintainer input

Do not move from measurement to implementation without a written decision record.

Evaluation Checklist

• Audit current HTML field usage and formatting semantics.
• Verify how NFC normalization affects the entire current catalog.
• Implement full-catalog decimal derivation parity check.
• Implement full-catalog hexadecimal derivation parity check.
• Report all mismatches, if any.
• Measure FlatBuffers raw asset size with stored fields.
• Measure FlatBuffers raw asset size without stored field payloads.
• Measure AAR and representative APK impact.
• Benchmark stored decimal conversion.
• Benchmark derived decimal conversion.
• Benchmark stored hexadecimal conversion.
• Benchmark derived hexadecimal conversion.
• Evaluate caching only if uncached derivation produces a meaningful regression.
• Write the decision record.
• Apply schema/runtime changes only if derivation is accepted.
• Rerun full tests and benchmarks after any accepted implementation.

Acceptance Criteria

The evaluation is complete when:

1. Every catalog entry has been checked for decimal and hexadecimal derivation equivalence.
2. Unicode normalization and supplementary code-point handling are explicitly verified.
3. Stored and derived FlatBuffers variants have measured size results.
4. Stored and derived HTML conversion paths have benchmark results using the established benchmark methodology.
5. A decision record states KEEP STORED, DERIVE ON ACCESS, or DERIVE WITH BOUNDED CACHE with evidence.
6. No production schema change occurs before that decision record exists.
7. If derivation is accepted, the full parser parity suite remains green and generated runtime data no longer stores unnecessary HTML payloads.

Non-Goals

• Do not redesign the Unicode trie in this issue.
• Do not change shortcode or tag indexes in this issue.
• Do not replace FlatBuffers with another binary format in this issue.
• Do not change HTML entity formatting for aesthetic reasons.
• Do not remove IEmoji.htmlDec or IEmoji.htmlHex solely because the values become computed.
• Do not introduce an unbounded cache of derived HTML strings.