-
Notifications
You must be signed in to change notification settings - Fork 14
SyncAndRestoreAPI
- CommCare 2.12 - Added the concept of "extension" indices affecting syncing
There are two modes with which a remote device will interact with a server's Case database, restore mode, and sync mode. restore mode presumes that the server will only need to push back a full set of a user's previous case submissions (if a phone is lost, and needs to be replaced, for instance). sync mode presumes that the server is potentially pushing down cases to users which they had not seen before, reassigned from other users, or created from an external system. Essentially, restore mode is identical to sync mode, merely only performing the base case synchronization from scratch.
Case data is submitted from devices to the server via JavaRosa Form HTTP submissions of XForms containing Case XML transactions. Servers are expected to parse these blocks as outlined in the specification and maintain a repository of the cases which are maintained there.
When a user syncs with the server, uses their CommCare application, or logs into the sever to use an online CommCare interaction, there exists a set of cases that should be "available" to that user. This (roughly) includes cases that are owned by that user, or cases that are derivative to cases that the user owns. It will not include all cases (even if, for instance, the user was connecting on a web browser to a server storing all cases). This set of cases is referred to as the user's scope.
When syncing the server and phone both determine the current user's scope. They then identify what is currently on each device, what cases should be in scope and out of scope, and what changes need to be made to reconcile the scope so that a user on any device "sees" the same set of data.
This document outlines both how the user's current scope is determined and how a remote device and the central server should negotiate to reconcile them and receive updates for all cases in that scope.
The first aspect of scope is "ownership". All cases have the concept of an "owner", which is either an individual user or a group of users.
Case ownership can be determined in two ways. Direct ownership or group ownership. In direct ownership, the owner of a case is specified by setting the owner_id of the case to be the id of a registered user. In group ownership, the server defines a group which has an id, and is associated with a set of user id's. If the owner_id is set to that of a group, each user id associated with a group id is considered an owner of the case.
The groups an individual user is a part of should be defined on the server, and sent to the phone during any restore actions as defined below. The groups are sent down as a fixture associated with an individual's user_id.
<fixture id="user-groups" user_id="ID_OF_RESTORING_USER">
<groups>
<group id="ID_OF_GROUP">
<name>NAME_OF_GROUP</name>
</group>
<group id="ID_OF_GROUP">
<name>NAME_OF_GROUP</name>
</group>
...
</groups>
</fixture>Case indices are the second component of identifying a user's scope. A case index can work in two directions. If a case has child index from a case C to a case P this outlines that case C relies on the existence of case P to function correctly. This means that as long as case C is inside the user's current scope that case P will be included as well, regardless of whether case P is owned by the user, or even if case P is closed.
If, conversely, a case has an extension index to another case, this signifies that the extension case E is only meaningful in the context of the host case H, and that the absence of the host case should also result in the absence of the extension case. If case H is not relevant in the user's scope, E will be removed.
Although a case can have multiple named indices to another case, it can only have one relationship with that case in order for a reasonably deterministic sync to take place. The child index relationship takes precedence to the extension relationship, so if case C1 has a child index to case C2 [IC12] and has an extension index to that case [IE12] the retention table for those cases is
| C1 Alive | C1 Dead | |
| C2 Alive | retained(C1, C2) | retained(C1, C2) |
| C2 Dead | retained(C1) released(C2) | released (C1, C2) |
Both the server and remote devices should have a consistent set of cases which are relevant to the user. In order to determine what cases to send over or manipulate, first the set of relevant cases for a user is defined, then it is identified which need to be communicated.
The first time that a device requests a sync from the server, the server identifies the full set of data to be transmitted to the phone using the following process. After any manipulations of case data, clients should ensure that the scope of data seen by the user is consistent with the outcome of this process.
The set of all owned cases for a user cuo is defined, this includes all cases whose owner is the user's ID, or any cases which are owned by groups that the user is a member of. Then all of the indexed cases identified by the indices of cases in cuo are recursively added to the set (regardless of owner) to identify the full universe of cases which are in the user's scope cu.
This forms a set of directed graphs with two kinds of special nodes. Root nodes, which are defined as nodes which themselves maintain no outgoing indices, and Leaf nodes, which are nodes which have no incoming indices. The algorithm itself assumes that the generated graph(s) will be acyclical.
Once cu is established, it is pared down to contain only live cases. live cases are determined from the following process. Note that a case can only bear one mark at a time and the result of a pass should only be able to set the mark to one value.
- Traverse all cases. if it is closed it should be marked dead, otherwise if a case has an extension index it should be marked abandoned, if neither condition applies, it should be marked as live
- Traverse all cases starting at the leaves, moving breadth first. Any case which is live should mark any nodes that it indexes as live
- Traverse all cases in reverse starting at the roots, depth first. Any case which is live and is the target of an index of type extension should mark the indexing case as live if its current mark is abandoned.
The set cu is transformed by identifying and removing all cases which are not live into cul.
The initial sync to the device consists of the <registration/> for the user who is requesting the data restore, <case/> transactions for all cases in cul, the <fixture/> for the groups the user is in if relevant, and a <sync/> payload as defined below, containing ut0, the starting sync token.
If the phone will continue synchronizing with the server, each form that it sends will include the HTTP header
| Header | Values |
|---|---|
| X-CommCareHQ-LastSyncToken | utN - the last sync token received when this form was completed. |
When the next synchronization action occurs, the user will submit utN (the last sync token received from a successful sync). This signals that the server should only send down transactions clN+1 defined by
[
clN (The set of cases that the user should have after the last sync, N)
cld_N (The set of cases that were changed only by the user in transactions tagged by utN)
]
clN+1 (The set of cases which were modified by the server and need to be updated on the phone)
->
cul (The current set of cases consider to be authoritative)
As well as any modifications to the user registration, or the group fixtures.
If the phone fails to process a sync action (WLOG referred to as utN+1), often due to a failed download of the restore payload, when it attempts to sync in the future, it will include the last successful token (utN), and will continue to use that token in submissions up until another sync action. A CommCare server must be prepared to generate a new payload (clN+1_2) and a new token (utN+1_2) for a previously existing token (utN) until it identifies a submission or sync action with the new token submitted. If the existing token (utN) is submitted again, it can be presumed that any new token (utN+1_n) can be discarded, along with any state associated with it.
The payload to be returned by the server to tag the current sync with a sync token should be the following, occurring at the top of the envelope before any case, fixture, or user registration transactions.
<sync xmlns="http://commcarehq.org/sync">
<restore_id>SYNC_TOKEN_HERE</restore_id>
</sync>Special care should be taken when processing case attachments as outlined on the appropriate API page
A device can additionally request a confirmation from a server about the what the current set cul should be when it is requesting a sync. It does so by producing a digest of the current live cases on the phone and submitting it to be verified on the server.
The digest is computed by first establishing a consistent set cul on the device by purging any cases from the phone which are not live.
The set of cases on the device should now consist of the join of the two sets
clN (Case set after the last sync) cld_N (Case transactions from the device since the last sync)
The devices will then iterate through the cases to generate the digest
digest = 0x00
for all case c:
digest = digest ⊕MD5(c.caseid)
The server can then recreate the set (clN, cld_N) and the associated digest and verify that the state is the same before producing and sending the next set of transactions and state tokens.
The digest should be submitted with the "CommCare State Hash" prefix:
ccsh:[digest]
If the state is not identical, the server should respond with an HTTP response code 412 for Precondition Failed. The server should not send down any case blocks as part of this transaction. At this point it is the phone's responsibility to reconcile the case lists, typically by submitting any unsent case transactions, submitting logs, purging its case database, and starting from scratch with the server's state.
Compatible servers should provide a URL which can be be queried with a GET request. The request should be capable of accepting any of the following URL parameters
| Parameter | Values | Optional? |
|---|---|---|
| since | utN - the last sync token received | Yes - Defaults to sync from scratch |
| state | stN - the state token for the phone's current state | Yes - Defaults to not checking the state |
| version | Case XML Spec version (1.0, 2.0, etc) | Yes - Defaults to 1.0 |
The server should respond with an XML payload as specified in the OpenRosaRequest API with the appropriate transactions for the sync action.