Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] [Mongo-cdc] Fallback to timestamp startup mode when resume token has expired #8754

Open
wants to merge 26 commits into
base: dev
Choose a base branch
from

Conversation

jw-itq
Copy link
Contributor

@jw-itq jw-itq commented Feb 17, 2025

Purpose of this pull request

Synchronize the Flink-CDC code to fix the Mongo-CDC issue.

When MongoDB CDC connector tries to create cursor with an expired resuming token during stream task fetching stage, it will crash with a fatal exception: error due to Command failed with error 280 (ChangeStreamFatalError): cannot resume stream; the resume token was not found.

This PR added fallback logic to create cursor with timestamp, which only runs when:

Mongo CDC is in StreamTaskFetch stage
a ChangeStreamFatalError (280) is raised
Current ChangeStreamOffset has a valid timestamp field

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

jw-itq added 24 commits August 22, 2024 19:10
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jw-itq , Do we have some way to add test case verify it?

@jw-itq
Copy link
Contributor Author

jw-itq commented Feb 18, 2025

Thanks @jw-itq , Do we have some way to add test case verify it?

ok, I'll give it a try. It might take some time.

@jw-itq
Copy link
Contributor Author

jw-itq commented Feb 19, 2025

This PR also fixes the issue where MongoDB CDC fails to resume from the savepoint during restart recovery.

@github-actions github-actions bot added the e2e label Feb 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants