-
Notifications
You must be signed in to change notification settings - Fork 131
feat: add minimal implementation of changesStream for registry mirroring #1025
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the changes endpoint need to return changes in order?
If so, this doesn't work. There is no guaruantee in Postgres that a sequential table is actually sequential on insert because multiple transactions can be in flight at once, and sequential IDs are assigned at the start of a transaction, not once the transaction commits.
I suggest you change this as follows:
-
Instead of a seperate database query to insert changes into the change stream, do this using
TRIGGER
in postgres on thepackage_versions
table. Add a trigger that on insert creates thechanges
entry. -
Add a
row_version
column to thechanges
table that is typed asxid8 NOT NULL DEFAULT pg_current_xact_id()
. -
On query, limit the returned results to
WHERE row_id <= pg_snapshot_xmin(pg_current_snapshot())
.
What this will do is ensure that we only show changes where all previous transactions have already committed or failed.
There is no strict time order requirement for npm changes. When the npmmirror system receives a Disorder or duplication within a certain period of time are There are some other changeTypes in npm, such as tag changes, maintainer changes, etc. These changes are not all triggered by db changes. In npmmirror, we uniformly distribute events, and these behaviors are completely asynchronous. |
5ec32d4
to
7366888
Compare
a1dc9af
to
850314d
Compare
Hi, jsr team
This PR introduces a minimal implementation of the changesStream interface (#110) to enable registry mirroring capabilities. Given that
11 months
have passed since our initial discussion and the feature remains unscheduled, I've taken the liberty to propose this implementation based on our proven approach in npmmirror.Key implementation notes:
application/json
endpointFor maintainers:
Context:
As developers in mainland China experience consistent network challenges accessing the JSR registry directly, this feature would significantly improve developer experience for one of the world's largest developer communities.
Thanks for building this vital infrastructure for the JavaScript ecosystem.
Looking forward to your feedback! 🙏🏻