Architecture

TBD's architecture consists of a few major pieces:

• Client

  The client is divided into two parts: the mutator and the adjustable algorithm. The mutator is responsible for loading and updating the input dataset, while the adjustable implements the logic of the incremental computation being performed.

  The client is also responsible for communicating with the master to start the initial run of the algorithm and any runs of change propagation.

• Master

  The master is responsible for launching the datastore, communicating with the client, and initiating the first worker when a new adjustable algorithm is run.

• Datastore

  The datastore contains the values of all of the modifiables that are created. It has a key-value store interface, backed by an in-memory hash table.

  Eventually, the data will be partitioned across several datastores, each of which will will be backed by disk storage and an in-memory cache.

• Workers

  The workers are responsible for running the adjustable algorithm. When the client initiates a new computation, the master launches the first worker. Additional workers may be launched by calls to 'par' inside the adjustable algorithm.

  Workers also manage change propagation. When a modifiable is updated, the datastore informs all of the workers that have read that modifiable, who then also inform all of their parents between themselves and the root worker that they have changes to propagate. When a new round of change propagation is started, updates are processed in order, starting from the top worker.