Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-24520: Sync "archived" list_members #64

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

luandy64
Copy link
Contributor

Description of change

This PR adds custom sync logic for list_members to include archived members.

The Mailchimp Docs say that the status field can return the following values: "subscribed", "unsubscribed", "cleaned", "pending", "transactional", or "archived".

The tap seemed to be unable to query for and return members with an "archived" status.

This PR writes the archived members to the same list_members stream. This minimizes the work in an downstream processes by avoiding weird joins and deduplication work.

Design Decisions

Use the same tap_stream_id

Again, these records come from the same endpoint. The default status in the endpoint seems to just not return archived members.

Use a query param status=archived in a second query

In testing, we could not get records to return properly when the following query parameters were used:

  • status=archived,subscribed
  • status=subscribed,archived
  • status=subscribed,archived,unsubscribed,cleaned,pending,transactional

This is reinforced by the Mailchimp Docs giving us a bash example for this endpoint

curl -X GET \
     'https://${dc}.api.mailchimp.com/3.0/lists/{list_id}/members
      ?fields=<SOME_ARRAY_VALUE>
      ...
      &status=<SOME_STRING_VALUE>
      ...

I ripped out a lot of text for clarity, but you can see that they don't expect status to be more than one value.

Manual QA steps

  • Ran the tap
  • Ran a bunch of curls
  • Created two members on one email list and archived and unarchived them while running syncs

Risks

  • Low: The list_members sync should be the same. But now we're adding one more call (or series of calls) to get the archived members.
    • There is no way to turn off the extra calls for archived members

Rollback steps

  • revert this branch, bump the version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant