Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Find end of WAL on safekeepers using WalStreamDecoder. #2262

Merged
merged 1 commit into from
Aug 14, 2022

Conversation

arssher
Copy link
Contributor

@arssher arssher commented Aug 12, 2022

We could make it inside wal_storage.rs, but taking into account that

  • wal_storage.rs reading is async
  • we don't need s3 here
  • error handling is different; error during decoding is normal
    I decided to put it separately.

Test
cargo test test_find_end_of_wal_last_crossing_segment
prepared earlier by @yeputons passes now.

Fixes #544 and https://github.com/neondatabase/cloud/issues/2004
Supersedes #2066

@arssher arssher requested review from knizhnik and yeputons August 12, 2022 16:02
We could make it inside wal_storage.rs, but taking into account that
 - wal_storage.rs reading is async
 - we don't need s3 here
 - error handling is different; error during decoding is normal
I decided to put it separately.

Test
cargo test test_find_end_of_wal_last_crossing_segment
prepared earlier by @yeputons passes now.

Fixes #544
      neondatabase/cloud#2004
Supersedes #2066
@arssher arssher force-pushed the find-end-of-wal-decoder branch from a22a0a5 to f322f85 Compare August 12, 2022 17:37
@arssher
Copy link
Contributor Author

arssher commented Aug 14, 2022

Failed test_large_schema
https://github.com/neondatabase/neon/runs/7811109009?check_suite_focus=true
is caused by a separate issue, being fixed at
neondatabase/postgres#193
so I'm merging this.

@arssher arssher merged commit 431393e into main Aug 14, 2022
@arssher arssher deleted the find-end-of-wal-decoder branch August 14, 2022 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

deal with wal records bigger than wal_seg_size
2 participants