Skip to content

WIP: Report download progress #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
Draft

WIP: Report download progress #155

wants to merge 11 commits into from

Conversation

simolus3
Copy link
Contributor

This requires changes to the core extension to work, hence the WIP.

This adds interfaces for tracking download progress during a sync operation. Following the design doc:

  1. The ProgressWithOperations class provides information about how many rows have been downloaded already and how many need to be downloaded in total to complete the current sync (and uses that to provide a fraction getter).
  2. The SyncDownloadProgress class keeps track of downloaded operations for each bucket. It can compute aggregate ProgressWithOperations results for the next complete sync (across all buckets) or up until a given priority (includes all buckets with that or a higher priority).
  3. An instance of SyncDownloadProgress is made available from SyncStatus.

The implementation uses new columns added to ps_buckets tracking the operation count at the last checkpoint and the amount of downloaded operations since then:

  1. When we receive a new checkpoint (or a diff), we fetch local progress counters from ps_buckets and use that to populate the initial progress status.
  2. When receiving sync lines, the core extension will update the local counters. We update the SyncDownloadProgress instance in memory.
  3. For a completed checkpoint, we reset the progress and update the "operation count at last checkpoint" column in the database.

I've also refactored some things around copying sync statutes internally (we don't have to write our own copy methods with special args to reset the upload + download error, optional arguments in Kotlin don't have to be constant so passingdownloadError = null will actually reset the value).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant