Skip to content

Feat/shorthand data#3089

Draft
antoniomuso wants to merge 25 commits into
mainfrom
feat/shorthand-data
Draft

Feat/shorthand data#3089
antoniomuso wants to merge 25 commits into
mainfrom
feat/shorthand-data

Conversation

@antoniomuso
Copy link
Copy Markdown
Contributor

@antoniomuso antoniomuso commented Oct 22, 2025

Description

This PR adds pack functionality to both coList and coPlainText. These methods optimize storage usage by reducing space while enabling faster data retrieval.

An example of a coList insertion now looks like:

[{"after":"start","op":"app","value":"<", "compacted": true},"p",">","H","e",...]

and a coPlainText insertion looks like:

[{"after":"start","op":"app","value":"<", "compacted": true},"p>Hello World!</p>"]

Every object is converted into an array, so the final result fr coPlainText will be:

[["<", "start", 1, 5],"p>Hello World!</p>"]

for coList:

[["<", "start", 1, 5],"p",">","H","e",...]

The OPID is converted into a string:

 `${opID.sessionID}:${opID.txIndex}:${opID.changeIdx}:${opID.branch ?? ""}`

Shared Packing with CoList

CoList (task items):
  10 items    : Pack OFF   7188b → Pack ON   6711b (6.6% savings)
  25 items    : Pack OFF  16932b → Pack ON  15715b (7.2% savings)
  50 items    : Pack OFF  32963b → Pack ON  30516b (7.4% savings)
  100 items   : Pack OFF  64799b → Pack ON  59884b (7.6% savings)

CoPlainText (text insertion):
  50 chars    : Pack OFF   3325b → Pack ON    929b (72.1% savings)
  100 chars   : Pack OFF   6061b → Pack ON   1197b (80.3% savings)
  250 chars   : Pack OFF  14262b → Pack ON   2002b (86.0% savings)
  500 chars   : Pack OFF  27930b → Pack ON   3334b (88.1% savings)

Custom Packing for CoPlainText

CoList (task items):
  10 items    : Pack OFF   7188b → Pack ON   6711b (6.6% savings)
  25 items    : Pack OFF  16932b → Pack ON  15715b (7.2% savings)
  50 items    : Pack OFF  32963b → Pack ON  30516b (7.4% savings)
  100 items   : Pack OFF  64799b → Pack ON  59884b (7.6% savings)

CoPlainText (text insertion):
  50 chars    : Pack OFF   3325b → Pack ON    685b (79.4% savings)
  100 chars   : Pack OFF   6061b → Pack ON    753b (87.6% savings)
  250 chars   : Pack OFF  14261b → Pack ON    952b (93.3% savings)
  500 chars   : Pack OFF  27929b → Pack ON   1288b (95.4% savings)

Performance

 bench  import with pack=false content - colist.pack.bench.ts > CoList - Pack ON vs Pack OFF comparison
 > 10 items - Pack comparison
 > Pack ON: 6765b | Pack OFF: 7188b | Savings: 5.9%

    1.02x faster than appendItems with pack=true
    1.59x faster than import with pack=true content
    2.29x faster than appendItems with pack=false

   bench  import with pack=false content - colist.pack.bench.ts > CoList - Pack ON vs Pack OFF comparison
 > 50 items - Pack comparison
 > Pack ON: 30569b | Pack OFF: 32965b | Savings: 7.3%

    1.39x faster than appendItems with pack=true
    1.56x faster than import with pack=true content
    2.81x faster than appendItems with pack=false

   bench  import with pack=false content - colist.pack.bench.ts > CoList - Pack ON vs Pack OFF comparison
 > 100 items - Pack comparison
 > Pack ON: 59937b | Pack OFF: 64801b | Savings: 7.5%

    1.19x faster than appendItems with pack=true
    1.51x faster than import with pack=true content
    2.24x faster than appendItems with pack=false

   bench  small items with pack=true - colist.pack.bench.ts > CoList - Pack comparison with different item sizes
 > Small items (~100 bytes each, 50 items)
 > Pack ON: 4301b | Pack OFF: 6693b | Savings: 35.7%

    1.77x faster than import small items (pack=false)
    2.39x faster than import small items (pack=true)
    4.64x faster than small items with pack=false

   bench  import large items (pack=false) - colist.pack.bench.ts > CoList - Pack comparison with different item sizes
 > Large items (~1KB each, 50 items)
 > Pack ON: 158353b | Pack OFF: 160748b | Savings: 1.5%

    1.67x faster than import large items (pack=true)
    2.08x faster than large items with pack=true
    2.24x faster than large items with pack=false

   bench  create with pack=true - colist.pack.bench.ts > CoList - Performance scaling with pack ON vs OFF
 > Batch size: 25 items
 > Pack ON: 15772b | Pack OFF: 16933b | Savings: 6.9%

    3.70x faster than create with pack=false

   bench  create with pack=true - colist.pack.bench.ts > CoList - Performance scaling with pack ON vs OFF
 > Batch size: 50 items
 > Pack ON: 30569b | Pack OFF: 32965b | Savings: 7.3%

    1.07x faster than create with pack=false

   bench  create with pack=true - colist.pack.bench.ts > CoList - Performance scaling with pack ON vs OFF
 > Batch size: 100 items
 > Pack ON: 59937b | Pack OFF: 64801b | Savings: 7.5%

    1.14x faster than create with pack=false

   bench  create with pack=true - colist.pack.bench.ts > CoList - Performance scaling with pack ON vs OFF
 > Batch size: 200 items
 > Pack ON: 119253b | Pack OFF: 129049b | Savings: 7.6%

    1.15x faster than create with pack=false

   bench  NAPI: import with pack=true content - colist.pack.bench.ts > CoList - NAPI Crypto with pack ON vs OFF
 > NAPI: 100 items comparison
 > NAPI - Pack ON: 59936b | Pack OFF: 64799b | Savings: 7.5%

    1.13x faster than NAPI: import with pack=false content
    1.64x faster than NAPI: append with pack=true
    1.90x faster than NAPI: append with pack=false

   bench  import with pack=false content - colist.pack.bench.ts > CoPlainText - Pack ON vs Pack OFF comparison
 > 50 characters insertion
 > Pack ON: 737b | Pack OFF: 3269b | Savings: 77.5%

    1.13x faster than import with pack=true content
    1.54x faster than insertAfter with pack=true
    2.37x faster than insertAfter with pack=false

   bench  insert 500 chars with pack=true - colist.pack.bench.ts > CoPlainText - Pack ON vs Pack OFF comparison
 > 500 characters paragraph
 > Pack ON: 1341b | Pack OFF: 27930b | Savings: 95.2%

    3.00x faster than import with pack=true content
    8.84x faster than import with pack=false content
    21.32x faster than insert 500 chars with pack=false

   bench  import with pack=true content - colist.pack.bench.ts > CoPlainText - Pack ON vs Pack OFF comparison
 > Large article (~2000 characters)
 > Pack ON: 3703b | Pack OFF: 208887b | Savings: 98.2%

    1.15x faster than insert 2000 chars with pack=true
    9.09x faster than import with pack=false content
    23.49x faster than insert 2000 chars with pack=false

Manual testing instructions

Tests

  • Tests have been added and/or updated
  • Tests have not been updated, because:
  • I need help with writing tests

Checklist

  • I've updated the part of the docs that are affected the PR changes
  • I've generated a changeset, if a version bump is required
  • I've updated the jsDoc comments to the public APIs I've modified, or added them when missing

@vercel
Copy link
Copy Markdown

vercel Bot commented Oct 22, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
clerk-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
design-system Ready Ready Preview Comment Nov 7, 2025 0:19am
file-upload-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
form-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
gcmp-homepage Ready Ready Preview Comment Nov 7, 2025 0:19am
image-upload-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-chat Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-chat-1 Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-chat-2 Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-filestream Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-image-upload Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-inspector Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-multi-cursors Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-organization Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-paper-scissors Error Error Nov 7, 2025 0:19am
jazz-richtext Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-richtext-prosekit Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-richtext-tiptap Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-todo Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-vector-search Ready Ready Preview Comment Nov 7, 2025 0:19am
jazz-version-history Ready Ready Preview Comment Nov 7, 2025 0:19am
music-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
passkey-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
passphrase-auth-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
quint-ui Ready Ready Preview Comment Nov 7, 2025 0:19am
reactions-demo Ready Ready Preview Comment Nov 7, 2025 0:19am
1 Skipped Deployment
Project Deployment Preview Comments Updated (UTC)
jazz-homepage Ignored Ignored Preview Nov 7, 2025 0:19am

after?: number,
privacy: "private" | "trusting" = "private",
options?: {
disablePacking?: boolean;
Copy link
Copy Markdown
Contributor Author

@antoniomuso antoniomuso Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about adding an options parameter, it could be a good way to enable features like compression for individual files.

@gdorsi gdorsi marked this pull request as draft October 29, 2025 11:49
@gdorsi
Copy link
Copy Markdown
Collaborator

gdorsi commented Feb 17, 2026

TODO:

  • Versioning the change at transaction level. We want to leverage the parsJSON error management and append a version byte at the end of the stringified payload
  • Make the packing disabled by default everywhere
  • Expose an API from jazz-tools to enable packing on all or specific coValue types

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants