Documentation of a decentralized sequencer

warp-academy-docs/docs/sdk/advanced/bundled-interaction.md

@@ -200,8 +200,3 @@ query {
 ## Interaction retrieval via Warp gateway
 
 The Warp `/gateway/interactions/:id` endpoint allows to retrieve the interaction info based on its original id.
-
-## Further development
-
-1. Blind/trustless sequencing - https://github.com/warp-contracts/gateway/issues/48
-2. Sequencer decentralization - https://github.com/warp-contracts/gateway/issues/93

warp-academy-docs/docs/sdk/advanced/vrf.md

@@ -1,5 +1,11 @@
 # Verifiable randomness
 
+:::caution
+Random values will be handled in a new way once the decentralized sequencer is launched.
+More details can be found [here](/docs/sequencer/lifecycle/ordering#random-values).
+For backward compatibility, the function `SmartWeave.vrf.randomInt(maxValue)` will still work, but it will use the new function `SmartWeave.randomNumber(maxValue)` under the hood.
+:::
+
 You have an option to use random values in contracts via Verifiable Random Functions.  
 This may be a very useful feature in gaming, dynamic NFT generation, etc.
 

warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md

@@ -1,7 +1,7 @@
 # Warp Sequencer - legacy
 
 :::caution
-Please note, that you are viewing old Warp Sequencer docs. Since `[email protected]` nested bundles have been introduced instead of the format described in this file. In order to read about current interaction format, please head to [Warp Sequencer](../advanced/bundled-interaction.md).
+Please note, that you are viewing old Warp Sequencer docs. Since `[email protected]` nested bundles have been introduced instead of the format described in this file. In order to read about current interaction format, please head to [Warp Sequencer](../../sequencer/introduction.md).
 :::
 
 Warp Sequencer is a module that is responsible for submitting users' transactions into Arweave (via [Bundlr Network](https://bundlr.network/)).

warp-academy-docs/docs/sequencer/api-methods.md

@@ -0,0 +1,100 @@
+# API methods
+
+Every sequencer node provides endpoints through a REST API that allows querying the network state or sending new interactions. 
+Below are listed the methods directly related to the functionality of the sequencer.
+
+:::tip
+In addition to the methods listed below, sequencer nodes offer endpoints related to the framework and consensus algorithm. 
+For example, a list of these methods for the testnet network can be found at: https://sequencer-0.testnet.warp.cc.
+:::
+
+## Submits a new interaction to the sequencer
+```
+/api/v1/data-item
+```
+This method is used to submit a new interaction to the sequencer.
+The request payload should contain the interaction in the form of a DataItem, conforming to the [ANS-104](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md) format. 
+Upon receiving such a request, the node validates the interaction. 
+If it is valid, the node encapsulates it in a message, adds a sequencer transaction to its mempool, and broadcasts it to the other nodes in the network.
+
+### Possible responses
+
+| Code | Description                                | Example value                    |
+| ---- | ------------------------------------------ | -------------------------------- |
+| 200  | The interaction was successfully submitted |<pre>{data_item_id: 'GHBDu8MoQcTSrvTHF_5LGwe_YEXb6dgMfGuWgIgcK-U',<br /> nonce: 1,<br /> sender: 'warp16ayu6zt7pxlmgf708agsxv2qmnfpfrh0pay3e0'}</pre> |
+| 400  | The request was invalid                    |<pre>{type: 'broadcast response with non-zero code',<br /> message:<br />  {Code: 1110,<br />   RawLog: 'data item does not have "Contract" tag:<br />            no contract tag'},<br /> status: { code: 400, text: 'Bad Request'}}</pre>|
+| 409  | Invalid `Nonce` value for the given sender |<pre>{type: 'broadcast response with non-zero code',<br /> message:<br />  {Code: 32,<br />   RawLog: 'account sequence mismatch, expected 3, got 4:<br />            incorrect account sequence'},<br /> status: { code: 409, text: 'Conflict'}}</pre>|
+| 503  | Mempool is full                            |<pre>{type: 'broadcast response with non-zero code',<br /> message:<br />  {Code: 20,<br />   RawLog: 'mempool is full'},<br /> status: { code: 503, text: 'Service Unavailable'}}</pre>|
+
+## Expected nonce value
+```
+/api/v1/nonce
+```
+This method returns the expected next nonce value for a given sender.
+
+### Request
+The request body should be in the form of JSON:
+```json
+{
+    "signature_type":1,
+    "owner":"..."
+}
+```
+Where:
+- `signature_type` - the integer that specifies the type of signature that the sender uses. Possible values that are currently supported by the sequencer:
+  - `1` - Arweave signature
+  - `3` - Ethereum signature
+- `owner` - the public key of the owner
+
+### Possible responses
+
+| Code | Description                                | Example value                    |
+| ---- | ------------------------------------------ | -------------------------------- |
+| 200  | The nonce was successfully returned        |<pre>{address: 'warp17jupac353t0euh95f70thl5vs8xcs4m0ydjq5q',<br /> nonce: 0}</pre> |
+| 400  | The request was invalid                    |<pre>{type: 'invalid request',<br /> message: '...',<br /> status: {code: 400,text: 'Bad Request'}}</pre>|
+
+## Get transaction by data item id
+```
+/api/v1/tx-data-item-id
+```
+
+This method returns the hash of the sequencer transaction that contains interactions with the given ID.
+The request body must contain JSON with the following field:
+- `data_item_id` - the ID of the interaction.
+
+The response body contains the following field:
+- `tx_hash` - the hash of the sequencer transaction.
+
+### Possible responses
+
+| Code | Description                                    | Example value                    |
+| ---- | ---------------------------------------------- | -------------------------------- |
+| 200  | The transaction hash was successfully returned |<pre>{tx_hash:'...'}</pre> |
+| 400  | The request was invalid                        |<pre>{type: 'invalid request',<br /> message: '...',<br /> status:{ code: 400, text: 'Bad Request'}}</pre>|
+| 404  | No transaction found                           |<pre>{type: 'not found',<br /> message: 'transaction not found for the given data item id',<br /> status:{code:404,text: 'Not Found'}}</pre>|
+
+
+:::tip
+With the hash of a transaction, querying the sequencer node for transaction details is possible using the following endpoint: `/cosmos/tx/v1beta1/txs/{hash}`.
+:::
+
+## Get transaction by sender and nonce
+```
+/api/v1/tx-sender-nonce
+```
+
+This method returns the hash of the sequencer transaction for a given sender address and nonce. 
+The request body must contain JSON with the following fields:
+- `sender` - the address of the sender.
+- `nonce` - the nonce of the interaction.
+
+The response body contains the following field:
+- `tx_hash` - the hash of the sequencer transaction.
+
+### Possible responses
+
+| Code | Description                                    | Example value                    |
+| ---- | ---------------------------------------------- | -------------------------------- |
+| 200  | The transaction hash was successfully returned |<pre>{tx_hash:'...'}</pre> |
+| 400  | The request was invalid                        |<pre>{type: 'invalid request',<br /> message: '...',<br /> status:{ code: 400, text: 'Bad Request'}}</pre>|
+| 404  | No transaction found                           |<pre>{type: 'not found',<br /> message: 'transaction not found for the given sender and nonce',<br /> status:{code:404,text: 'Not Found'}}</pre>|

warp-academy-docs/docs/sequencer/centralized.md

@@ -0,0 +1,158 @@
+# Warp Centralized Sequencer
+
+:::caution
+This section describes the operation of the current centralized sequencer, which will soon be replaced by the new [decentralized solution](/docs/sequencer/decentralized).
+:::
+
+## How it works
+
+Instead of posting the interaction transactions directly to Arweave mainnet, Warp SDK creates a data item with interaction input and tags which is then signed with user's wallet. Data item is then sent to Warp Sequencer (`/gateway/v2/sequencer/register` endpoint) (this is the default behaviour of Warp's SDK `contract.writeInteraction`
+function, when `forMainnet` instance is being used).
+
+The Warp Sequencer then:
+
+#### 1. Generates a sort key
+
+A sort key is generated from:
+
+1. current mainnet network height
+2. current sequence value
+3. original transaction id
+4. current mainnet block hash
+
+In the original SmartWeave protocol specification, a _sort key_ is defined
+as a `[ block_height,sha256(transactionId + blockHash) ]`, where:
+
+1. `block_height` - current network height, l-padded with `0` to 12 chars, e.g. for block height `1015556`, the
+   result is `000001015556`
+2. `sha256(transactionId + blockHash)` - a `sha256` hash of the concatenated buffers of the transaction id and block
+   hash,
+   e.g. for txId `UR_35HORbjjZ_NnUqinkZuWkcNB1-gBST3Rezt5JrDs` and block
+   hash `ixWCxRN36DjVUxQRa68xIeoZLfvLDTtX78e0ae8RAAJjOPpDBuVKVaEKYOpq7bLS`,
+   the result is `44edd70f2018924f22a878a558a8f2d5cae8bc1f718d567df43bf52b6384d260`.
+
+The complete _sort key_ for the above values would
+be: `000001015556,44edd70f2018924f22a878a558a8f2d5cae8bc1f718d567df43bf52b6384d260`.
+
+The generated sort keys are then used by the SmartWeave protocol to lexicographically sort the transactions.
+
+The Warp Sequencer extends this default mechanism by the current sequence value.  
+The formula for the _sort key_ is extended to:
+`[ block_height,sequence_value,sha256(transactionId + blockHash) ]`
+
+This sequence value can be obtained from the Sequencer's node timestamp, database or other sources.
+In its current implementation - a Sequencer node timestamp value is being used.  
+This in effect gives a fair transactions ordering - the transactions will have the sequence assigned in order in which
+they are processed by the Sequencer.
+
+Assuming transaction id `La_NpAFAWxGj-VIiLfg7NbBfox0RZ8uuEJSOOZykd48`, block
+hash `-o88tFYsMG9RXSGcNXX5sVDuSV5uHy7zuFRj6vYo91e3mXpmng6qw322Ip0-EguA`,
+block height `1015560` and current Sequencer value `1663069424541`, the generated _sort key_ would
+be `000001015560,1663069424541,a21ac8a60326ba8c2bb8caa05cff3334a22e9960ef55de0b5392caa30b484d0a`
+
+**NOTE** All the transactions sent to Arweave directly, have the sequence value assigned to `0000000000000000`.
+This effectively means that if transactions to a given contract are sent both directly to Arweave mainnet and Warp
+Sequencer -
+if two transactions happen to be at the same block height, the "direct" Arweave transactions take precedence.
+This also means that the sequencing algorithm is fully backwards compatible with the original SmartWeave protocol.
+
+#### 2. Generates tags for the Bundlr transaction
+
+| Tag Name                                    | Tag Value                                                            |
+| ------------------------------------------- | -------------------------------------------------------------------- |
+| `Sequencer`                                 | `Warp`                                                               |
+| `Sequencer-Owner`                           | The original owner/signar of the contract transaction                |
+| `Sequencer-Mills`                           | The sequence value used by the Sequencer for this transaction        |
+| `Sequencer-Sort-Key`                        | The generated sort key for this transaction                          |
+| `Sequencer-Prev-Sort-Key`                   | The sort key of the previous transaction                             |
+| `Sequencer-Tx-Id`                           | The original transaction id                                          |
+| `Sequencer-Block-Height`                    | The block height used for generating the sort key                    |
+| `Sequencer-Block-Id`                        | The block hash used for generating the sort key                      |
+| `Sequencer-Block-Timestamp`                 | The timestamp of the block that was used for generating the sort key |
+| ...all the tags of the original transaction |                                                                      |
+
+:::tip
+The `Sequencer-Prev-Sort-Key` tells what is the sort key of the 'previous' transaction in the sequencer and
+can be used to verify whether all transactions have been properly loaded (i.e. if one
+decides to load them directly from L1 nodes) and none is missing.
+:::
+
+Additional set of tags are added in case user requests generating a random value using VRF (Verifiable Random Function):
+
+| Tag Name     | Tag Value                                                               |
+| ------------ | ----------------------------------------------------------------------- |
+| `vrf-index`  | The original hash generated by the VRF (using `sort_key` as input data) |
+| `vrf-proof`  | The original proof generated by the VRF                                 |
+| `vrf-bigint` | A BigInt value evaluated from the hash generated by the VRF             |
+| `vrf-pubkey` | The public key used by the VRF                                          |
+
+Verifiable randomness can be used by contracts that require using random values - e.g. gaming contracts, nft/loot
+generating contracts, etc.
+Using the `sort_key`, `vrf-proof` and `vrf-pubkey`, the client can always verify the generated random value.
+
+#### 3. Wrap original data item in a bundle and uploads the bundle to Bundlr
+
+Apart from all the tags from point 2, some additional tags are added to the wrapping bundle:
+
+| Tag Name         | Tag Value         |
+| ---------------- | ----------------- |
+| `Bundle-Format`  | `binary`          |
+| `Bundle-Version` | `2.0.0`           |
+| `App-Name`       | `Warp`            |
+| `Action`         | `WarpInteraction` |
+
+In order to send original data item to Bundlr, we use the concept of nested bundles and set an interaction data item inside a bundle. If you're not faimiliar with this concept, here is a quick summary:
+
+:::info
+Bundling allows to write multiple data items into one top level transaction. A data item differs from a regular transaction by not allowing to transfer AR tokens and passing reward but it has most of the transaction's properties - such as owner, data, tags, owner and id.
+
+In a nutshell, the nested bundles concept means that a data item of a bundle can also be a bundle containg other data items. According to ANS-104 standard it can lead to theoretically unbounded levels of nesting.
+
+You can read the specification for nested bundles standard in [ANS-104](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md#31-nested-bundle).
+:::
+
+:::info
+Obviously the data-item is signed by the Sequencer's wallet - and as such can be easily verified.
+The Sequencer wallet address is `jnioZFibZSCcV8o-HkBXYPYEYNib4tqfexP0kCBXX_M`.
+:::
+
+**NOTE** The original data item is not modified in any way - this is to preserve the original
+signature!
+
+After receiving proper response and receipt from Bundlr, the Warp gateway indexes the contract interaction
+internally - to make it instantly available.
+
+#### 4. Finally, the Warp gateway returns the response from the Bundlr to the client.
+
+## Interaction data item retrieval (generated by the Warp Sequencer) via Arweave gateway
+
+Use the GQL endpoint, with the original data item id passed in the `Contract` tag. Note that all the
+interactions will be part of a bundle (i.e. will have the `edges.node.bundledIn.id` value set).
+
+```qql
+query {
+  transactions(
+    ids: ["1UIhK4vL5lc2X4aMsJFmMpJqfdgrjznVzi2F17yLBlc"]
+  ) {
+    edges {
+      node {
+          id
+          tags {
+            name
+            value
+          }
+          block {
+            height
+          }
+          bundledIn {
+            id
+          }
+        }
+    }
+  }
+}
+```
+
+## Interaction retrieval via Warp gateway
+
+The Warp `/gateway/interactions/:id` endpoint allows to retrieve the interaction info based on its original id.