reset problem with shofco syncs

[siddhant3030]

Describe the bug
We've some missing data from commcare and we created a new sync to test this. New sync which we just created pulled

1. zzz_case - we have these many records - 73,697 loaded (test connection ) - Test commcare gender (connection name)
2. zzz_case - after reset in the main connection - 34,394 loaded - (Gender - Commcare)

both is using the same source. but one is pulling only few record after reset.

[fatchat]

In [8]: wclient.execute("select count (distinct id) from staging_gender.zzz_case ")
Out[8]: [{'count': 34394}]

In [9]: wclient.execute("select count (distinct id) from staging_gender_test.zzz_case ")
Out[9]: [{'count': 73697}]

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

[fatchat]

created new connection to an E2E Testing destination: 73,780 loaded

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

[fatchat]

reset the old connection, synced 34,394

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

[fatchat]

note that even though the new connections went from 73,697 to 73,780 after an hour or so had passed, the original connection is stuck at 34,394

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

[fatchat]

inspecting the catalog

SELECT id, catalog FROM public.connection
where id in ('f70ff943-9c4c-4dd6-b459-ff6993022605', '96bbe77d-64d8-4687-bbc2-7c82001523d4')

the catalog contains a list of 39 streams

the index of zzz_case in the original connection's catalog is 30
the index of zzz_case in the new connection's catalog is 38 i.e. last place

In the connector's Python code we have Form.read_records populating CommcareStream.forms

and Case.read_records refers to this list to decide whether or not to create a new record (if any(f in CommcareStream.forms for f in record["xform_ids"]))

So Case.read_records needs to run after all the Form.read_records run

And these streams are probably run in the order in which they appear in the catalog

Attempt: replace the old connection's catalog with the new connection's catalog (could have just moved the object for zzz_case to the end but there are some new keys like is_resumable so may as well get those in there

Cleared and syncing

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

[fatchat]

yes that fixed it
73,780 records

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

[fatchat]

long-term fix will be done via #121

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>