TDL-24520: Sync "archived" list_members
#64
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
This PR adds custom sync logic for
list_members
to include archived members.The Mailchimp Docs say that the
status
field can return the following values: "subscribed", "unsubscribed", "cleaned", "pending", "transactional", or "archived".The tap seemed to be unable to query for and return members with an "archived" status.
This PR writes the archived members to the same
list_members
stream. This minimizes the work in an downstream processes by avoiding weird joins and deduplication work.Design Decisions
Use the same
tap_stream_id
Again, these records come from the same endpoint. The default status in the endpoint seems to just not return
archived
members.Use a query param
status=archived
in a second queryIn testing, we could not get records to return properly when the following query parameters were used:
status=archived,subscribed
status=subscribed,archived
status=subscribed,archived,unsubscribed,cleaned,pending,transactional
This is reinforced by the Mailchimp Docs giving us a
bash
example for this endpointI ripped out a lot of text for clarity, but you can see that they don't expect
status
to be more than one value.Manual QA steps
Risks
list_members
sync should be the same. But now we're adding one more call (or series of calls) to get thearchived
members.archived
membersRollback steps