diff --git a/warp-academy-docs/docs/sdk/advanced/bundled-interaction.md b/warp-academy-docs/docs/sdk/advanced/bundled-interaction.md index 987bb87..0ff4290 100644 --- a/warp-academy-docs/docs/sdk/advanced/bundled-interaction.md +++ b/warp-academy-docs/docs/sdk/advanced/bundled-interaction.md @@ -200,8 +200,3 @@ query { ## Interaction retrieval via Warp gateway The Warp `/gateway/interactions/:id` endpoint allows to retrieve the interaction info based on its original id. - -## Further development - -1. Blind/trustless sequencing - https://github.com/warp-contracts/gateway/issues/48 -2. Sequencer decentralization - https://github.com/warp-contracts/gateway/issues/93 diff --git a/warp-academy-docs/docs/sdk/advanced/vrf.md b/warp-academy-docs/docs/sdk/advanced/vrf.md index 54ec0b0..422afc4 100644 --- a/warp-academy-docs/docs/sdk/advanced/vrf.md +++ b/warp-academy-docs/docs/sdk/advanced/vrf.md @@ -1,5 +1,11 @@ # Verifiable randomness +:::caution +Random values will be handled in a new way once the decentralized sequencer is launched. +More details can be found [here](/docs/sequencer/lifecycle/ordering#random-values). +For backward compatibility, the function `SmartWeave.vrf.randomInt(maxValue)` will still work, but it will use the new function `SmartWeave.randomNumber(maxValue)` under the hood. +::: + You have an option to use random values in contracts via Verifiable Random Functions. This may be a very useful feature in gaming, dynamic NFT generation, etc. diff --git a/warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md b/warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md index 0481924..38340fe 100644 --- a/warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md +++ b/warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md @@ -1,7 +1,7 @@ # Warp Sequencer - legacy :::caution -Please note, that you are viewing old Warp Sequencer docs. Since `warp-contracts@1.4.13` nested bundles have been introduced instead of the format described in this file. In order to read about current interaction format, please head to [Warp Sequencer](../advanced/bundled-interaction.md). +Please note, that you are viewing old Warp Sequencer docs. Since `warp-contracts@1.4.13` nested bundles have been introduced instead of the format described in this file. In order to read about current interaction format, please head to [Warp Sequencer](../../sequencer/introduction.md). ::: Warp Sequencer is a module that is responsible for submitting users' transactions into Arweave (via [Bundlr Network](https://bundlr.network/)). diff --git a/warp-academy-docs/docs/sequencer/api-methods.md b/warp-academy-docs/docs/sequencer/api-methods.md new file mode 100644 index 0000000..8ef0f79 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/api-methods.md @@ -0,0 +1,100 @@ +# API methods + +Every sequencer node provides endpoints through a REST API that allows querying the network state or sending new interactions. +Below are listed the methods directly related to the functionality of the sequencer. + +:::tip +In addition to the methods listed below, sequencer nodes offer endpoints related to the framework and consensus algorithm. +For example, a list of these methods for the testnet network can be found at: https://sequencer-0.testnet.warp.cc. +::: + +## Submits a new interaction to the sequencer +``` +/api/v1/data-item +``` +This method is used to submit a new interaction to the sequencer. +The request payload should contain the interaction in the form of a DataItem, conforming to the [ANS-104](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md) format. +Upon receiving such a request, the node validates the interaction. +If it is valid, the node encapsulates it in a message, adds a sequencer transaction to its mempool, and broadcasts it to the other nodes in the network. + +### Possible responses + +| Code | Description | Example value | +| ---- | ------------------------------------------ | -------------------------------- | +| 200 | The interaction was successfully submitted |
{data_item_id: 'GHBDu8MoQcTSrvTHF_5LGwe_YEXb6dgMfGuWgIgcK-U',
 nonce: 1,
 sender: 'warp16ayu6zt7pxlmgf708agsxv2qmnfpfrh0pay3e0'}
| +| 400 | The request was invalid |
{type: 'broadcast response with non-zero code',
 message:
  {Code: 1110,
   RawLog: 'data item does not have "Contract" tag:
            no contract tag'},
 status: { code: 400, text: 'Bad Request'}}
| +| 409 | Invalid `Nonce` value for the given sender |
{type: 'broadcast response with non-zero code',
 message:
  {Code: 32,
   RawLog: 'account sequence mismatch, expected 3, got 4:
            incorrect account sequence'},
 status: { code: 409, text: 'Conflict'}}
| +| 503 | Mempool is full |
{type: 'broadcast response with non-zero code',
 message:
  {Code: 20,
   RawLog: 'mempool is full'},
 status: { code: 503, text: 'Service Unavailable'}}
| + +## Expected nonce value +``` +/api/v1/nonce +``` +This method returns the expected next nonce value for a given sender. + +### Request +The request body should be in the form of JSON: +```json +{ + "signature_type":1, + "owner":"..." +} +``` +Where: +- `signature_type` - the integer that specifies the type of signature that the sender uses. Possible values that are currently supported by the sequencer: + - `1` - Arweave signature + - `3` - Ethereum signature +- `owner` - the public key of the owner + +### Possible responses + +| Code | Description | Example value | +| ---- | ------------------------------------------ | -------------------------------- | +| 200 | The nonce was successfully returned |
{address: 'warp17jupac353t0euh95f70thl5vs8xcs4m0ydjq5q',
 nonce: 0}
| +| 400 | The request was invalid |
{type: 'invalid request',
 message: '...',
 status: {code: 400,text: 'Bad Request'}}
| + +## Get transaction by data item id +``` +/api/v1/tx-data-item-id +``` + +This method returns the hash of the sequencer transaction that contains interactions with the given ID. +The request body must contain JSON with the following field: +- `data_item_id` - the ID of the interaction. + +The response body contains the following field: +- `tx_hash` - the hash of the sequencer transaction. + +### Possible responses + +| Code | Description | Example value | +| ---- | ---------------------------------------------- | -------------------------------- | +| 200 | The transaction hash was successfully returned |
{tx_hash:'...'}
| +| 400 | The request was invalid |
{type: 'invalid request',
 message: '...',
 status:{ code: 400, text: 'Bad Request'}}
| +| 404 | No transaction found |
{type: 'not found',
 message: 'transaction not found for the given data item id',
 status:{code:404,text: 'Not Found'}}
| + + +:::tip +With the hash of a transaction, querying the sequencer node for transaction details is possible using the following endpoint: `/cosmos/tx/v1beta1/txs/{hash}`. +::: + +## Get transaction by sender and nonce +``` +/api/v1/tx-sender-nonce +``` + +This method returns the hash of the sequencer transaction for a given sender address and nonce. +The request body must contain JSON with the following fields: +- `sender` - the address of the sender. +- `nonce` - the nonce of the interaction. + +The response body contains the following field: +- `tx_hash` - the hash of the sequencer transaction. + +### Possible responses + +| Code | Description | Example value | +| ---- | ---------------------------------------------- | -------------------------------- | +| 200 | The transaction hash was successfully returned |
{tx_hash:'...'}
| +| 400 | The request was invalid |
{type: 'invalid request',
 message: '...',
 status:{ code: 400, text: 'Bad Request'}}
| +| 404 | No transaction found |
{type: 'not found',
 message: 'transaction not found for the given sender and nonce',
 status:{code:404,text: 'Not Found'}}
| diff --git a/warp-academy-docs/docs/sequencer/centralized.md b/warp-academy-docs/docs/sequencer/centralized.md new file mode 100644 index 0000000..87aa6db --- /dev/null +++ b/warp-academy-docs/docs/sequencer/centralized.md @@ -0,0 +1,158 @@ +# Warp Centralized Sequencer + +:::caution +This section describes the operation of the current centralized sequencer, which will soon be replaced by the new [decentralized solution](/docs/sequencer/decentralized). +::: + +## How it works + +Instead of posting the interaction transactions directly to Arweave mainnet, Warp SDK creates a data item with interaction input and tags which is then signed with user's wallet. Data item is then sent to Warp Sequencer (`/gateway/v2/sequencer/register` endpoint) (this is the default behaviour of Warp's SDK `contract.writeInteraction` +function, when `forMainnet` instance is being used). + +The Warp Sequencer then: + +#### 1. Generates a sort key + +A sort key is generated from: + +1. current mainnet network height +2. current sequence value +3. original transaction id +4. current mainnet block hash + +In the original SmartWeave protocol specification, a _sort key_ is defined +as a `[ block_height,sha256(transactionId + blockHash) ]`, where: + +1. `block_height` - current network height, l-padded with `0` to 12 chars, e.g. for block height `1015556`, the + result is `000001015556` +2. `sha256(transactionId + blockHash)` - a `sha256` hash of the concatenated buffers of the transaction id and block + hash, + e.g. for txId `UR_35HORbjjZ_NnUqinkZuWkcNB1-gBST3Rezt5JrDs` and block + hash `ixWCxRN36DjVUxQRa68xIeoZLfvLDTtX78e0ae8RAAJjOPpDBuVKVaEKYOpq7bLS`, + the result is `44edd70f2018924f22a878a558a8f2d5cae8bc1f718d567df43bf52b6384d260`. + +The complete _sort key_ for the above values would +be: `000001015556,44edd70f2018924f22a878a558a8f2d5cae8bc1f718d567df43bf52b6384d260`. + +The generated sort keys are then used by the SmartWeave protocol to lexicographically sort the transactions. + +The Warp Sequencer extends this default mechanism by the current sequence value. +The formula for the _sort key_ is extended to: +`[ block_height,sequence_value,sha256(transactionId + blockHash) ]` + +This sequence value can be obtained from the Sequencer's node timestamp, database or other sources. +In its current implementation - a Sequencer node timestamp value is being used. +This in effect gives a fair transactions ordering - the transactions will have the sequence assigned in order in which +they are processed by the Sequencer. + +Assuming transaction id `La_NpAFAWxGj-VIiLfg7NbBfox0RZ8uuEJSOOZykd48`, block +hash `-o88tFYsMG9RXSGcNXX5sVDuSV5uHy7zuFRj6vYo91e3mXpmng6qw322Ip0-EguA`, +block height `1015560` and current Sequencer value `1663069424541`, the generated _sort key_ would +be `000001015560,1663069424541,a21ac8a60326ba8c2bb8caa05cff3334a22e9960ef55de0b5392caa30b484d0a` + +**NOTE** All the transactions sent to Arweave directly, have the sequence value assigned to `0000000000000000`. +This effectively means that if transactions to a given contract are sent both directly to Arweave mainnet and Warp +Sequencer - +if two transactions happen to be at the same block height, the "direct" Arweave transactions take precedence. +This also means that the sequencing algorithm is fully backwards compatible with the original SmartWeave protocol. + +#### 2. Generates tags for the Bundlr transaction + +| Tag Name | Tag Value | +| ------------------------------------------- | -------------------------------------------------------------------- | +| `Sequencer` | `Warp` | +| `Sequencer-Owner` | The original owner/signar of the contract transaction | +| `Sequencer-Mills` | The sequence value used by the Sequencer for this transaction | +| `Sequencer-Sort-Key` | The generated sort key for this transaction | +| `Sequencer-Prev-Sort-Key` | The sort key of the previous transaction | +| `Sequencer-Tx-Id` | The original transaction id | +| `Sequencer-Block-Height` | The block height used for generating the sort key | +| `Sequencer-Block-Id` | The block hash used for generating the sort key | +| `Sequencer-Block-Timestamp` | The timestamp of the block that was used for generating the sort key | +| ...all the tags of the original transaction | | + +:::tip +The `Sequencer-Prev-Sort-Key` tells what is the sort key of the 'previous' transaction in the sequencer and +can be used to verify whether all transactions have been properly loaded (i.e. if one +decides to load them directly from L1 nodes) and none is missing. +::: + +Additional set of tags are added in case user requests generating a random value using VRF (Verifiable Random Function): + +| Tag Name | Tag Value | +| ------------ | ----------------------------------------------------------------------- | +| `vrf-index` | The original hash generated by the VRF (using `sort_key` as input data) | +| `vrf-proof` | The original proof generated by the VRF | +| `vrf-bigint` | A BigInt value evaluated from the hash generated by the VRF | +| `vrf-pubkey` | The public key used by the VRF | + +Verifiable randomness can be used by contracts that require using random values - e.g. gaming contracts, nft/loot +generating contracts, etc. +Using the `sort_key`, `vrf-proof` and `vrf-pubkey`, the client can always verify the generated random value. + +#### 3. Wrap original data item in a bundle and uploads the bundle to Bundlr + +Apart from all the tags from point 2, some additional tags are added to the wrapping bundle: + +| Tag Name | Tag Value | +| ---------------- | ----------------- | +| `Bundle-Format` | `binary` | +| `Bundle-Version` | `2.0.0` | +| `App-Name` | `Warp` | +| `Action` | `WarpInteraction` | + +In order to send original data item to Bundlr, we use the concept of nested bundles and set an interaction data item inside a bundle. If you're not faimiliar with this concept, here is a quick summary: + +:::info +Bundling allows to write multiple data items into one top level transaction. A data item differs from a regular transaction by not allowing to transfer AR tokens and passing reward but it has most of the transaction's properties - such as owner, data, tags, owner and id. + +In a nutshell, the nested bundles concept means that a data item of a bundle can also be a bundle containg other data items. According to ANS-104 standard it can lead to theoretically unbounded levels of nesting. + +You can read the specification for nested bundles standard in [ANS-104](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md#31-nested-bundle). +::: + +:::info +Obviously the data-item is signed by the Sequencer's wallet - and as such can be easily verified. +The Sequencer wallet address is `jnioZFibZSCcV8o-HkBXYPYEYNib4tqfexP0kCBXX_M`. +::: + +**NOTE** The original data item is not modified in any way - this is to preserve the original +signature! + +After receiving proper response and receipt from Bundlr, the Warp gateway indexes the contract interaction +internally - to make it instantly available. + +#### 4. Finally, the Warp gateway returns the response from the Bundlr to the client. + +## Interaction data item retrieval (generated by the Warp Sequencer) via Arweave gateway + +Use the GQL endpoint, with the original data item id passed in the `Contract` tag. Note that all the +interactions will be part of a bundle (i.e. will have the `edges.node.bundledIn.id` value set). + +```qql +query { + transactions( + ids: ["1UIhK4vL5lc2X4aMsJFmMpJqfdgrjznVzi2F17yLBlc"] + ) { + edges { + node { + id + tags { + name + value + } + block { + height + } + bundledIn { + id + } + } + } + } +} +``` + +## Interaction retrieval via Warp gateway + +The Warp `/gateway/interactions/:id` endpoint allows to retrieve the interaction info based on its original id. \ No newline at end of file diff --git a/warp-academy-docs/docs/sequencer/data-format.md b/warp-academy-docs/docs/sequencer/data-format.md new file mode 100644 index 0000000..e7315df --- /dev/null +++ b/warp-academy-docs/docs/sequencer/data-format.md @@ -0,0 +1,166 @@ +# Data Format + +This chapter specifies the format of data stored by the sequencer network. + +## Genesis state + +The genesis state of the Warp Sequencer is the initial data that is used to start the blockchain. This data includes: +- Information about the Arweave block from which the sequencer will start. +- Data required to calculate the prev sort key for interactions handled by the sequencer. + +### Arweave Block Information + +The first type of data needed for the sequencer to start is information about the Arweave block that was last processed by the previous centralized sequencer, +as well as data about the Arweave block that will be added first to the sequencer blockchain, along with the interactions it contains. +These details are extracted from the `arweave_block.json` file, and a sample representation is provided below: + +```json +{"lastArweaveBlock": + {"height":1318268, + "timestamp":1702040807, + "hash":"VwqjV_2VMlh5DzLHUbk0n3EN_a5aRbe-H2xIFACZ9AmZrujSwwmEdxt4B_M0jYxl"}, + "nextArweaveBlock": + {"blockInfo": + {"height":1318269, + "timestamp":1702040913, + "hash":"CQCojSIk5irYvdnsvWKm1d62Oh5z9G3X7qAbNjcu2Yh-KYzgz1IYT9GIpaYIs6cJ"}, + "transactions": + [{"id":"UAie4Tyqlz5eCTQK_aqoRzWznZ0UCdckTm6E4pjA_HE", + "contract":"KTzTXT_ANmF84fWEKHzWURD1LWd9QaFR9yfYUwH2Lxw", + "sort_key":"000001318269,0000000000000,56802e09c3d3310140df996fa70be6eccc8c28e715c601e39d820a6be816f54c"}, + {"id":"6t-OqbyGeQ-yigDbwRcChtjwpB8w5uswg_yvd9LHE5A", + "contract":"KTzTXT_ANmF84fWEKHzWURD1LWd9QaFR9yfYUwH2Lxw", + "sort_key":"000001318269,0000000000000,6849c6ab2791631ed9645dfb9a7eea6868bcb5f6aa1fea19700bc48a4f9345b9"}, + {"id":"1iJSdydrcR2YqEsZ8DVrKIrIIdv-MG7WvRL6cCe76Is", + "contract":"KTzTXT_ANmF84fWEKHzWURD1LWd9QaFR9yfYUwH2Lxw", + "sort_key":"000001318269,0000000000000,e6472aac3115163f3a14de480744be6dc6fc29f860397e50662ea18d7a9bff2f"}]}} +``` +Where: +- `lastArweaveBlock` is information about the block that was last processed by the previous sequencer. +- `nextArweaveBlock` is the Arweave block that will be added first to the sequencer. + +### Previous Sort Keys for Contracts + +Information about the sort key of the last interaction for each contract is crucial for assigning prev sort keys to interactions handled by the sequencer. +This data is sourced from the `prev_sort_keys.json` file, structured as follows: +```json +[{"contract":"007g_77MJ1eJaeRvDYwj4Po-6_2_G-5DGyN7eX3AL4o", + "sortKey":"000001025889,1664397554325,6dca2bac0ee65c6f3a3b0f3e5fe5d162c130bc5c39ea203524b5b799d1ad7f04"}, + {"contract":"008p0JdJA0XY79o139Us9OgXbcLmxiKubZ_7v31HZOI", + "sortKey":"000001119402,1676452495692,13468b18dd98a9c0fe9cb3e6f5b482ad14da1b744dea660a373fadb0ff0360d9"}, + {"contract":"0094QB2iMzuy7h-W5_577li7kat2jb3EXYmDxRYm2yM", + "sortKey":"000001237223,1691593745386,b4c26ef252025f97b6684fb88e5a2c66ec4f8b1b9d5899e1c05859550aa3265b"}, + {"contract":"00Bh4GLtT1kY4sAu3QbsXH3ZSuXa2iBaVXO64LJovp8", + "sortKey":"000000801138,0000000000000,1ed5d58b01f4deda06dcd61bd4e1d8494e82142df0d8333758d8a6d0648eb78f"}, + ... +] +``` + +## Messages format + +Sequencer blocks encapsulate [transactions](https://docs.cosmos.network/main/learn/advanced/transactions), which in turn contain [messages](https://docs.cosmos.network/main/build/building-modules/messages-and-queries). +Within the realm of Warp Sequencer, two fundamental types of messages play a pivotal role: those tailored for incoming interactions and those designed for Arweave blocks retrieved by the sequencer. +Messages are defined and serialized in the [Protocol Buffers](https://protobuf.dev/) format. + +### Interaction message + +The `MsgDataItem` message is used to store an interaction sent to the sequencer. The message has the following structure: +```protobuf +message MsgDataItem { + bytes data_item = 1; + string sort_key = 2; + string prev_sort_key = 3; + bytes random = 4; +} +``` + +### Arweave block message + +The `MsgArweaveBlock` message is used to store an Arweave block that has been downloaded by the sequencer. The message has the following fields: + +```protobuf +message MsgArweaveBlock { + ArweaveBlockInfo block_info = 1; + repeated ArweaveTransactionWithInfo transactions = 2; +} +``` +The first field of the message consists of basic information about the Arweave block: +```protobuf +message ArweaveBlockInfo { + uint64 height = 1; + uint64 timestamp = 2; + string hash = 3; +} +``` +The second field contains a list of the following interaction data: +```protobuf +message ArweaveTransactionWithInfo { + ArweaveTransaction transaction = 1; + string prev_sort_key = 2; + bytes random = 3; +} +``` +Above, we have the values assigned by the sequencer, as well as the data about the original interaction with Arweave: +```protobuf +message ArweaveTransaction { + string id = 1; + string contract = 2; + string sort_key = 3; +} +``` + +## Data stored in sequencer nodes + +Sequencer nodes need to store certain data in order to process interactions. This data includes: +- The data of the last Arweave block that has been added to the sequencer. +- The sort key of the last interactions for all contracts. + +The consistency of this data between nodes is checked with each sequencer block, which contains the [AppHash](https://github.com/cometbft/cometbft/blob/v0.37.x/spec/core/data_structures.md#header) - a hash that is calculated based on the above data, among other things. + +### Last Arweave block + +Information about the most recent Arweave block added to the sequencer is crucial for various processes, particularly in the calculation of sort key values for incoming interactions. +It is essential to recall that the first component of this key is the height of the referenced Arweave block. + +The details about the last block can be obtained by traversing backward through the last sequencer blocks until the message with the Arweave block is found. +However, for performance reasons, this data is stored in a dedicated store and is easily accessible. + +The format of the data stored is as follows: +```protobuf +message LastArweaveBlock { + ArweaveBlockInfo arweave_block = 1; + int64 sequencer_block_height = 2; +} +``` +This struct contains two fields: +- `arweave_block` - basic Arweave block data. +- `sequencer_block_height` - height of the sequencer block where the last Arweave block was added. + +### Previous sort keys + +Every interaction passing through the sequencer that is not the initial interaction of a particular contract must be assigned a `prev sort key`. +To achieve this, sequencer nodes maintain a map where, for each key representing the transaction identifier of a contract, a value is set as the sort key of the last interaction for that contract. +Each entry in this map takes the following form: +```protobuf +message PrevSortKey { + string contract = 1; + string sortKey = 2; +} +``` + +### Arweave blocks queue + +In addition to the above stores, each validator in the sequencer network downloads and maintains a list of upcoming Arweave blocks to be added to the sequencer network. The block synchronization process is detailed [here](/docs/sequencer/lifecycle/receiving#syncing-with-arweave). + +Unlike the aforementioned stores, the Arweave blocks queue is not cross-validated between nodes because nodes may be at different stages of block retrieval. This consistency is verified when adding an Arweave block to the sequencer block and validating these blocks. + +The data format stored in the queue is as follows: + +```protobuf +message NextArweaveBlock { + ArweaveBlockInfo blockInfo = 1; + repeated ArweaveTransaction transactions = 2; +} +``` + +This structure includes information about the upcoming Arweave block and a list of transactions that will be incorporated into the sequencer network upon processing. diff --git a/warp-academy-docs/docs/sequencer/decentralized.md b/warp-academy-docs/docs/sequencer/decentralized.md new file mode 100644 index 0000000..3554e27 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/decentralized.md @@ -0,0 +1,151 @@ +# Warp Decentralized Sequencer + +In contrast to the current [centralized sequencer](/docs/sequencer/centralized), our upcoming [decentralized sequencer](https://github.com/warp-contracts/sequencer) solution +marks a significant evolution in our approach to interaction sequencing within the Arweave ecosystem. +Embracing the principles of decentralization, transparency, and community consensus, this innovative system aims +to distribute the responsibility of determining the order of interactions across a network of sequencers. + +## Criteria + +The selection of a new sequencer solution was guided by several crucial criteria, ensuring a robust and efficient system. These standards include: + +#### Decentralization +The transition to a decentralized model is essential to foster transparency and accountability. +Rather than relying on a single entity, the ordering of transactions should be distributed among multiple nodes. +This decentralization eliminates a single point of control (i.e. the Warp team) and enhances the overall resilience of the system. +#### Security +Security is of paramount importance when decentralizing the sequencer. +By distributing the interaction processing across a network of nodes, the risk of a single point of failure is significantly reduced. +The design must prioritize the implementation of robust security measures to protect against potential attacks. +#### Performance +Ensuring low latency is crucial to enable swift and efficient interaction processing. +The Warp sequencer benefits from not burdening itself with contract state evaluation, simplifying the performance requirements. +However, a timely confirmation process remains a top priority when designing the solution to maintain a seamless user experience. +#### Simplicity +While decentralization introduces complexity, it is essential to strive for simplicity in the final solution. +A streamlined approach minimizes implementation time, mitigates technical risks, and enhances stability. +Furthermore, a simpler system is easier to maintain and less prone to vulnerabilities, ensuring a reliable and secure environment. + +## Functionality +The functionalities provided by the sequencer can be categorized into the following areas: + +#### Receiving Interactions + +Applications that rely on the [SmartWeave Protocol](/docs/sdk/advanced/smartweave-protocol) to provide smart contract functionality use the [Warp SDK](/docs/sdk/overview). +The SDK allows for deploying contracts and reading their state, as well as creating and sending interactions with a contract. +The latter functionality requires a sequencer, which [receives and validates](/docs/sequencer/lifecycle/receiving) the signed interaction from the user. +A key feature of this functionality is its speed, so that sending an interaction to the sequencer does not block the user from taking further actions. + +#### Ordering + +The sequencer's primary function is to arrange received interactions in a [linear order](/docs/sequencer/lifecycle/ordering). +This order is determined collectively, meaning that all sequencer nodes must agree on it. +In the first phase of operation of the decentralized sequencer, the decision about the order of interactions is made in a rotational manner, with each node proposing a block of interactions in turn. +The node that is currently proposing a block is called a *proposer*. + +The default implementation of the sequencer node assumes that the order of interactions in the block will be the same as the order in which the node received them. +However, since other nodes cannot verify this order, in the future sequencer development phases, the decision about the order of interactions in a proposed block will be made in a more collaborative manner. + +#### Syncing + +Warp is a Layer 2 solution for Arweave that uses the sequencer to significantly improve the speed of smart contract interactions. +This means that interactions take seconds instead of several minutes. +However, the speed is not always a priority. +In such use cases, sending interactions directly to Arweave is also possible. + +[Warp Gateway](/docs/gateway/overview) provides indexing for both Layer 2 and Layer 1 interactions. +This requires the sequencer to order all interactions in a linear fashion, which is necessary for the correct evaluation of the contract state. +The sequencer periodically [downloads interactions](/docs/sequencer/lifecycle/receiving#syncing-with-arweave) from Arweave and stores them in its own blockchain. +This ensures that all interactions are available to the gateway, regardless of whether they were sent directly to Arweave or through Warp Sequencer. +To ensure that no interaction is skipped during the evaluation, the sequencer also assigns a pointer to its predecessor in the contract to each interaction. + +It is important to emphasize once again that Layer 1 interactions will be processed with a delay, due to the nature of the Arweave consensus mechanism. + +#### Randomness + +Random values are often used in smart contracts for a variety of purposes, such as selecting winners in a lottery, determining which player goes first in a game, or generating unique identifiers. +However, generating random numbers in a decentralized environment can be challenging, as it is crucial to ensure that the values are unpredictable. +Yet, the value cannot be randomly drawn during the evaluation of the contract, as this process must be deterministic. + +To address this challenge, the sequencer generates a [random value](/docs/sequencer/lifecycle/ordering#random-values) for each interaction. +This value is based on the hash of the block in which the interaction is located, as well as the hash of the transaction itself. +This ensures that the value is deterministic, meaning that it can be verified by other nodes in the network, but it is also unpredictable. + +#### Relaying + +Arweave is a permanent, decentralized storage network that provides a foundation for a variety of applications, including smart contracts. +All interactions from SmartWeave protocol, including those processed by the Warp Sequencer, are eventually stored on Arweave. + +To achieve this, the sequencer, after determining the order of interactions, [sends](/docs/sequencer/lifecycle/relaying) all calculated data to Arweave using [Turbo](https://ardrive.io/turbo/). +This transfer is carried out by one of the nodes, which is selected rotationally, as with block production. +The node that performs this task is called the *relayer*. +The remaining nodes then verify whether the relayer has completed its task based on the proof provided by the relayer (confirmation signed by Turbo). + +#### Notifying + +Once the sequencer has ordered all interactions in a linear order, it is possible to query them. Queries can be made via [API](/docs/sequencer/api-methods). It is also possible to listen for events that are emitted during the operation of the sequencer. One can subscribe to these events using WebSocket. + +Different applications can use these capabilities to monitor the sequencer's progress. For example, there is an application that subscribe to events emitted by the sequencer to store incoming interactions in a database, which is then used by the [Warp Gateway](/docs/gateway/overview). Thus, interactions that are sent to the sequencer are indexed shortly after and are ready for further processing. + +## Roles + +Based on the above functionalities, we can identify the following roles that sequencer nodes fulfill. + +#### Validator + +Validators form the sequencing network and play a critical role in ensuring the security and integrity of the decentralized mechanism. +Their responsibilities include validating incoming interactions and broadcasting them to other network nodes. +Validators also participate in the consensus process, where they independently validate blocks and ensure they have agreed upon the same value consistent with the protocol rules. +To become a validator, individuals must hold a certain amount of the network’s native token as a stake, which can be slashed if they engage in misconduct or fail to fulfill their obligations. +All active validators also fulfill the role of *syncer*, and from this group the role of *proposer* and *relayer* is also chosen. + +#### Syncer + +All active validators are required to download Arweave blocks and the interactions contained within them. +This data is needed to include it in a block proposal (in the case of fulfilling the role of proposer) or to validate such a proposal. +Due to the nature of the Arweave consensus algorithm and in order to ensure the determinism of the sequencer consensus, Arweave blocks are added to the sequencer network with a suitable delay. + +#### Proposer + +The role of the proposer within the sequencing network is vital for the creation of new blocks. +Selected from the pool of active validators, the proposer is chosen using a [weighted round-robin algorithm](https://docs.cometbft.com/main/spec/consensus/proposer-selection). +In other words, validators who stake more tokens have a higher chance of proposing a block. However, every validator eventually gets a chance to become a proposer. +This promotes fairness while still incentivizing validators to hold more tokens. + +The primary responsibility of the proposer is to construct a block containing an ordered list of interactions and propose it to the other validators. +If the remaining nodes reach a consensus by voting with more than 2/3 of the votes, the proposed block is committed to the blockchain. + +#### Relayer + +Interactions pass through the sequencer to establish their order before being sent to Arweave. +The relayer plays a crucial role in this process by facilitating the decentralized transmission of interactions from the sequencer to Arweave. +Instead of directly sending data to Arweave, the relayer leverages [Turbo](https://ardrive.io/turbo/), which guarantees the reliable delivery of data to the blockchain. + +The relayer’s task is to send the original user-signed interactions and including information about the sequencer chain’s block where the transactions were placed, particularly the interaction order within the block. + +The relayer is selected among the validators using the same algorithm as the proposer. +The difference is that the proposer is selected every block, while the relayer is selected for a certain number of blocks. + +The network verifies the relayer’s adherence to their obligations by ensuring the delivery of valid receipts. +The relayer is incentivized through rewards to maximize the amount of data sent and faces penalties, including token slashing, for misbehavior. +In case of non-compliance, the responsibility is transferred to the next designated relayer. + + +## Technology Stack + +The decentralized sequencer operates as a blockchain, where the sequencer network achieves consensus +by producing blocks containing a sequentially ordered list of interactions. +However, this is a blockchain specifically designed to establish the order of interactions. +Ultimately, these interactions find their way to the main chain, Arweave. + +The sequencer network is built on [Cosmos SDK](https://v1.cosmos.network/sdk), a modular framework that enables the tailored development of blockchains. +This platform introduces a division into two layers. +The application layer is responsible for implementing business logic — in this context, establishing the linear order of interactions for a given SmartWeave contract. +The second layer is the consensus layer, which is implemented using the well-known [Tendermint](https://tendermint.com) algorithm. +And more precisely, its extension, i.e. [Comet BFT](https://docs.cometbft.com/v0.37/introduction). + +Tendermint employs a Byzantine Fault Tolerant (BFT) consensus algorithm, signifying its resilience to certain types of errors +and attacks, even in scenarios where some nodes in the network are faulty or operate in a dishonest manner +(referred to as Byzantine nodes in this context). Tendermint ensures consistent and secure operation across multiple machines. +Consistency, in this context, means that each machine observes precisely the same list of interactions and calculates an identical state. +Meanwhile, security is guaranteed by functioning even when up to 1/3 of the machines operate erroneously for various reasons. diff --git a/warp-academy-docs/docs/sequencer/introduction.md b/warp-academy-docs/docs/sequencer/introduction.md new file mode 100644 index 0000000..1b622ca --- /dev/null +++ b/warp-academy-docs/docs/sequencer/introduction.md @@ -0,0 +1,52 @@ +# Warp Sequencer + +The **Warp Sequencer** serves as a crucial module responsible for submitting users' transactions to the Arweave blockchain. Throughout this process, transactions undergo indexing in the [Warp Gateway](/docs/gateway/overview), ensuring their immediate availability for subsequent operations, such as state evaluation. This stands in contrast to traditional processing, where users typically experience delays of several minutes for transaction mining and confirmation. + +The primary aim of the Warp Sequencer is to enhance both Developer and User Experiences. Normally, when an interaction with a contract is being sent to Arweave, a waiting period ensues: + +1. **Transaction Mining (~2 minutes):** The time required for the transaction to be mined. +2. **Transaction Confirmation (~20 minutes, assuming at least 10 blocks):** The duration for the transaction to receive proper confirmation. + +This cumulative waiting time amounts to approximately 20-25 minutes, presenting challenges: + +1. **Disruption to Developer Experience (DX):** For instance, developers seeking to rapidly test contract functions on the mainnet face significant delays. +2. **Impact on User Experience (UX):** Users engaging with a dApp/protocol must endure prolonged waits to witness the outcomes of their interactions with a contract. This renders applications effectively unusable, particularly for users accustomed to the efficiency of web2 applications. + +The Warp Sequencer addresses these issues, introducing a more streamlined and efficient process for Arweave transactions, ultimately optimizing both developer workflows and user interactions. + +:::note +The significance of waiting for proper confirmation cannot be overstated, particularly in the context of smart contract interactions. +Consider a scenario where a cache, that is evaluating the contracts state for all the interactions returned at any given time by the +default Arweave (arweave.net) gateway. +If the cache does not wait for the proper transactions' confirmation, there is a risk of storing a contract state that has been evaluated from transactions within forked blocks. Even more concerning is the possibility of storing state data from transactions that were never included in any block. +::: + +## Ordering Interactions + +As the name implies, the primary function of the sequencer is to arrange incoming interactions in a sequence, establishing a linear order for them. The [Smartweave](/docs/sdk/advanced/smartweave-protocol) protocol determines the order of interactions based on the transaction ID and the Arweave block hash. However, information about the block in which a particular interaction is located and its corresponding hash becomes available only when the interaction is successfully recorded on the Arweave. Nonetheless, in accordance with the aforementioned rationale, waiting for this extended duration is undesirable. + +The order of interactions directly affects the contract state calculation process. To reconcile the need for order and the imperative for swift processing, the sequencer assigns an order to interactions shortly after receiving them from the client. Subsequently, the [Warp Gateway](/docs/gateway/overview) returns interactions for a given contract in the sequence dictated by the sequencer, enabling the computation of the contract state. This approach balances the necessity for order and the need for efficient interaction processing. + +:::info +For more information, refer to the [ordering](/docs/sequencer/lifecycle/ordering) phase of the interaction lifecycle. +::: + +## Final Destination on Arweave + +Another pivotal functionality of the sequencer is to transmit interactions to the [Turbo](https://ardrive.io/turbo/infographic/), responsible for ensuring their finality by delivering them to Arweave. + +In the realm of blockchain integrity, Arweave serves as a foundational layer that provides an immutable and tamper-resistant record of interactions. This inherent characteristic ensures the long-term integrity and reliability of the entire system. Arweave offers a permanent storage solution, making it well-suited for the enduring preservation of interactions. + +This means that even in the case of potential unavailability of Warp infrastructure, interactions remain accessible. This resilience is possible as interactions can be retrieved directly from Arweave, providing a fail-safe mechanism for continued accessibility even during infrastructure disruptions. + +:::info +For more information, see the [relaying](/docs/sequencer/lifecycle/ordering) phase of the interaction lifecycle. +::: + +## Decentralization + +The currently running sequencer is centralized, offering advantages in terms of simplicity and efficiency. However, this solution falls short of the requisite conditions expected in a decentralized ecosystem. The ability to determine the order of interactions profoundly influences the state of contracts, and we would not want to leave such authority in a single entity's hands. Such decisions should be made collectively. + +Therefore, the current sequencer will soon be replaced by a decentralized network of sequencers that will reach a consensus on the order of interactions. This transition is driven by our commitment to cultivating a truly decentralized environment that aligns with the principles of transparency, security, and community consensus. The forthcoming decentralized sequencer network aims to enhance the robustness and trustworthiness of the system, fostering a collaborative approach to transaction sequencing. + +The current centralized version will be described in detail in the next section, but the remainder of this chapter will focus on the new decentralized solution. diff --git a/warp-academy-docs/docs/sequencer/lifecycle/creating.md b/warp-academy-docs/docs/sequencer/lifecycle/creating.md new file mode 100644 index 0000000..6561250 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/lifecycle/creating.md @@ -0,0 +1,52 @@ +# Creating and Sending Interactions + +At the inception of the interaction lifecycle, the creation of an interaction is initiated by a user or application. +The interaction is created using the [Warp SDK](/docs/sdk/overview), which provides a set of methods for interacting with the contract. + +## Format + +The interaction can be in the form of a DataItem or an Arweave transaction. +By default, interactions are sent to the Warp Sequencer, which guarantees fast processing. +In this case, the interaction is created as a DataItem that conforms to the [ANS-104](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md) standard. +However, it is also possible to send interactions directly to Arweave. +In this case, the interaction will be created as an [Arweave transaction](https://docs.arweave.org/developers/arweave-node-server/http-api#transaction-format). +Details about the structure of interactions can be found in the section on the [SmartWeave](/docs/sdk/advanced/smartweave-protocol#contract-interactions) protocol. + +## Nonce + +A crucial security feature implemented in the interaction submission process to the Warp Sequencer is the use of a *nonce* — a counter incremented with each interaction sent by a particular sender. +This counter must be set in the interaction using a tag. +Consequently, successive interactions from the same address must contain progressively incremented nonce values. +The nonce serves two primary purposes: +* **Protection Against Replay Attacks** - the nonce acts as a defense mechanism against replay attacks, thwarting malicious actors from resubmitting a valid interaction. +* **Ensuring Sequential Order** - additionally, the nonce provides assurance that interactions with sequentially incremented nonce values will enter into a contract in precisely the same order. + +The Warp Sequencer offers an [API method](/docs/sequencer/api-methods#expected-nonce-value) enabling the verification of the expected nonce value for the next interaction from a specific sender address. +The Warp SDK first queries the sequencer for this value, but when creating subsequent interactions within the same contract and sender, it simply increments the counter. + +## Signing + +The constructed interaction, now including the nonce, is signed by a compatible wallet type to ensure its authenticity and integrity. +Warp SDK supports signatures from Arweave and Ethereum wallets, providing flexibility in authentication methods +(see [Warp Signature plugin](/docs/sdk/advanced/plugins/signature)). + +## Submission and confirmation + +Once signed, the interaction is ready for submission to the Warp Sequencer. +This step marks the completion of the interaction creation process from the SDK perspective. +The SDK process can conclude at this point, especially if immediacy is the primary concern. +However, for scenarios requiring confirmation that the interaction has been indexed by the gateway and is ready for processing during contract state evaluation, the option `waitForConfirmation = true` can be set. The confirmation wait time for Warp Sequencer and Warp Gateway is approximately 1 second. In the case of confirmation of submission to Arweave, this duration is significantly longer. + +:::tip +Setting the `waitForConfirmation` option is done as follows: +```typescript +const contract = warp.contract('CONTRACT_TX_ID').setEvaluationOptions({ + waitForConfirmation: true, +}); +``` +By default, this option is enabled for interactions sent to the Warp Sequencer and disabled for interactions sent directly to Arweave. +::: + +:::info +Version 1.5.0 of the SDK will support the transition period. This means that it will be compatible with both the current centralized sequencer and the new decentralized network. Because the process of creating interactions for these two solutions differs (for example, with the nonce tag), the SDK will first query the Warp Gateway to determine which version of the sequencer is currently running and then use the appropriate method. This also means that this version of the SDK will handle the switchover between sequencers. +::: diff --git a/warp-academy-docs/docs/sequencer/lifecycle/ordering.md b/warp-academy-docs/docs/sequencer/lifecycle/ordering.md new file mode 100644 index 0000000..2e9a132 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/lifecycle/ordering.md @@ -0,0 +1,169 @@ +# Ordering Interactions + +## Sort Key + +Interactions are ordered based on a unique key that is assigned to each interaction. +We will refer to such a key as **sort key**. +The SmartWeave protocol uses the following pair to establish the order: +``` +[block_height, sha256(transactionId + blockHash)] +``` +Where: +* `block_height` is the height of the Arweave block that contains the interaction. +* The second element of the pair is equal to the `sha256` hash of the concatenated transaction ID and block hash. +This means that the order of transactions within a block is unpredictable until the block is mined. + +The centralized [Warp Sequencer](/docs/sequencer/centralized) introduced an additional element to the sort key. The pair was replaced by a triplet, where the second element is the timestamp at which the sequencer received the interaction. However, for interactions sent directly to Arweave, this second element receives the value `0000000000000`. + +For example, for the transaction ID `La_NpAFAWxGj-VIiLfg7NbBfox0RZ8uuEJSOOZykd48`, block hash `o88tFYsMG9RXSGcNXX5sVDuSV5uHy7zuFRj6vYo91e3mXpmng6qw322Ip0-EguA`, block height `1015560`, and sequencer timestamp `1663069424541`, the sort key value for this transaction is: +``` +000001015560,1663069424541,a21ac8a60326ba8c2bb8caa05cff3334a22e9960ef55de0b5392caa30b484d0a +``` +If the same transaction were to go directly to Arweave, it would have the assigned key: +``` +000001015560,0000000000000,a21ac8a60326ba8c2bb8caa05cff3334a22e9960ef55de0b5392caa30b484d0a +``` +This solution ensured that interactions were still ordered in the same order as defined by the SmartWeave protocol. At the same time, the centralized sequencer did not have to wait for the interaction to be placed in an Arweave block, and it used the data from the current Arweave block to calculate the sort key. + +In a decentralized sequencer, the timestamp of the interaction cannot be taken into account because each node received the interaction at a different time, or some nodes may not have had the interaction in their mempool at all. Therefore, in this solution, the sort key is calculated based on the data from the sequencer block in which the interaction was placed. + +The key consists of the following triplet: +``` +Arweave block height, Sequencer block height, Interaction index in the Sequencer block +``` +Where: +* `Arweave block height` is the height of the last Arweave block that was [fetched](/docs/sequencer/lifecycle/receiving#syncing-with-arweave) by the sequencer and added to the sequencer's blockchain. +* `Sequencer block height` is the height of the block that was agreed upon by the sequencer nodes and that contains the given interaction. +* Interactions in the sequencer block are placed in a list, and each element of the list is assigned an index in the order in which it is placed. +This index is the third component of the sort key triplet. + +For example, if the last block added to the sequencer's chain was an Arweave block with a height of 1322128, and the interaction was placed in a sequencer block with a height of 1322128, being the third interaction in that block, its sort key would have the value: +``` +000001322128,0000001322128,00000002 +``` + +Interactions sent directly to Arweave are assigned the same sort key as in the case of a centralized sequencer, following the SmartWeave protocol, but with all zeros in the second position. +This implies, in particular, that when an Arweave block along with its contained interactions is added to a specific sequencer block, it becomes the first transaction in that block, and the interactions are the first ones to have a sort key with a given Arweave height in the first position. + +## Previous Sort Key + +---- +[DIAGRAM WITH LINKED LIST OF INTERACTIONS] + +---- + +All interactions, regardless of how they were sent, are ordered by the sequencer in a linear order. +This means that for each interaction, which is not the first interaction of a given contract, we can identify its predecessor within that contract. +For this purpose, we use a value we will call the `prev sort key`. + +This value is useful prior to the evaluation of the contract state when retrieving the [list of interactions](/docs/gateway/http/get/interactions) for that contract from the gateway, ensuring that no interactions were overlooked. + +To calculate the prev sort key, each sequencer node maintains a map (refer to: [Data Format](/docs/sequencer/data-format#previous-sort-keys)) in which, for each contract ID, the sort key of the last interaction associated with that contract is stored. + +## Block Proposal +As sequencer nodes accept interactions and place them in their mempool, these interactions do not yet have assigned `sort key` and `prev sort key` values. These values can only be calculated at the time of creating a proposal for the next sequencer block. + +---- +[DIAGRAM FOR PROPOSER] + +---- + +The process is as follows: + +1. One of the sequencer nodes is selected as the proposer of the next block. +This is done according to the [Proposer Selection Procedure](https://docs.cometbft.com/main/spec/consensus/proposer-selection), which is a weighted round-robin algorithm. +The remaining nodes wait until the selected proposer sends them their proposal or the specified time for this action elapses (in this case, the next proposer is selected). + +2. The proposer creates a new block based on its local queue of Arweave blocks and the list of interactions in the local mempool. + +3. If the oldest Arweave block in the local queue predates the last sequencer block by 1/2 hour, it is added as the first in the new sequencer block. +This means that all subsequent interactions will receive a sort key whose first component aligns with the height of this block. + +4. Subsequently, interactions from the mempool are added to the sequencer block following their order of placement in the mempool. +The size of the sequencer block is limited, as defined in the network configuration. +Hence, if there are many interactions in the mempool, not all may fit into the next block. +However, in most cases, all interactions are included, and the proposer's mempool is emptied. + +5. Once the list of interactions for the block is determined, sort key values are calculated for them. +All the necessary data for these keys is known, including the height of the new block and the indices of individual interactions. +Additionally, the proposer calculates the prev sort key value for each interaction (including those in a potentially added Arweave block) and assigns them random values (more on this [below](#random-values)). + +6. It is possible that the newly created sequencer block may also include other transactions more related to the network's operation than directly to smart contract functionality. + +7. After creating the block proposal, it is signed and sent to the remaining nodes (validators) in the network for verification, followed by a voting process on the proposal. + +## Block validation + +Once the block proposal is sent to the remaining sequencer nodes, those nodes proceed to validate it. +In addition to the standard checks, such as whether the block has the correct format and is correctly signed, the interactions contained in the block are also validated. +The process unfolds as follows: + +1. If the oldest Arweave block in the validator's local queue is older than 1/2 hour from the previous sequencer block, the validator checks whether such a block is added to the beginning of the sequencer block proposal. +The basic data of the Arweave block and the interactions contained in it are compared between the block proposal and the local queue. + +2. Next, the list of interactions added to the block is examined. The correctness of these interactions is validated, specifically checking if they include the correct `Nonce` value and are correctly signed by the sender. + +3. During interaction validation, the validator reproduces the process of calculating the sort key, prev sort key, and random value, verifying if the obtained values match those computed by the proposer. + +4. If the block has successfully passed the validation process, the validator votes to add the block to the chain. Otherwise, it votes against. + +Upon acceptance of the block by the sequencer network, the node must review its mempool and the Arweave block queue. +Interactions that have been added to the block are removed from the mempool. +Additionally, the remaining interactions in the mempool are checked for correctness, such as verifying the nonce value. +From the Arweave block queue, the block is removed if it has been added to the sequencer block. + +## Random values + +When creating a block proposal, the proposer assigns interactions not only values related to their order but also pseudorandom values. +These values are utilized by certain contracts, such as those associated with games or lotteries. +Since state evaluation must be a deterministic process, these values cannot be generated during state computation. +Otherwise, different [D.R.E.](/docs/dre/overview) nodes might calculate different states. + +The following conditions must be met when calculating random values: + +1. The value cannot be predicted, especially by users who create the interaction, so it cannot be set by the sender. +2. There should be a way to verify whether the value has not been tampered with. +3. For interactions sent directly to Arweave, this value should be independent of the data computed by the sequencer, providing the opportunity to use smart contracts on Arweave without engaging with the Warp ecosystem. + +A solution could be to assign this value by the sequencer in a non-predictable but deterministic way: +``` +sha256(Sequencer Block Hash + Sort Key) +``` +This entails calculating the `sha256` hash from the concatenation of the sequencer block hash and the sort key. +This approach satisfies condition 1, as users cannot predict this value until it is included in the relevant block. +Simultaneously, sequencer nodes have minimal influence on the block hash (detailed analysis [below](#malicious-proposer)), making it straightforward to verify whether they have calculated this value correctly. + +For interactions sent directly to Arweave, condition 3 must also be met. Therefore, in this case, the value is equal to: +``` +sha256(Sort Key) +``` +:::tip +Random values calculated by the sequencer in the form of hashes are available in contract code through the function: +```typescript +SmartWeave.randomNumber(maxValue) +``` +which returns a natural number in the range `[1, maxValue]`. +::: + +## Malicious Proposer + +The sequencer network operates correctly under the assumption that over 2/3 of its nodes are honest. +However, in the context of ordering interactions, this assumption may not be sufficient. +Every active validator occasionally takes on the role of proposer, which has a significant impact on the order of interactions. +The default sequencer implementation assumes that interactions are placed in a block in the order in which they were received. +However, sequencer nodes lack the capability to verify whether the proposer indeed preserves such an order. +The rotational selection of the proposer ensures that the majority of blocks have appropriately ordered interactions, but this falls short of an adequate security level. + +The same applies to random values. +The proposer has minimal influence on the block hash, because it has the ability to decide which votes approving the previous block to add to the current block (it must collect more than 2/3 of the votes). +Consequently, it has very limited control over the random values in the next block. +However, a malicious collaboration between two consecutive proposers would be required. + +To address these issues, future versions of the decentralized sequencer will introduce the following developments: +- **Threshold encryption:** To prevent the proposer from seeing which interactions to order, eliminating any incentive for manipulation. +- **More Collective Ordering:** Nodes can send the order of interactions resulting from their mempool together with their votes on a given block, and the proposer will be to a certain extent obliged to follow the order sent by other nodes. +- **Random (or hash-based) Order** can also be considered, as it is currently in the SmartWeave protocol. +- **VDF for Random Numbers:** A verifiable delay function (VDF) will be computed from the block hash. +The proposer, when creating a block, cannot predict or precompute it due to time constraints. + +More details about future sequencer developments can be found on the [roadmap](/docs/sequencer/roadmap). \ No newline at end of file diff --git a/warp-academy-docs/docs/sequencer/lifecycle/overview.md b/warp-academy-docs/docs/sequencer/lifecycle/overview.md new file mode 100644 index 0000000..5fdd426 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/lifecycle/overview.md @@ -0,0 +1,23 @@ +# Interaction Lifecycle + +Interactions are used to change the state of contracts and invoke computations defined in the contract's source code. +However, before any of that can happen, the interaction must go through a lifecycle, which is not such a simple process. + +In brief, the lifecycle of an interaction begins when it is [created](/docs/sequencer/lifecycle/creating) and sent by the [Warp SDK](/docs/sdk/overview). +The interaction is then [received](/docs/sequencer/lifecycle/receiving) by the Warp Sequencer, which establishes order relative to other interactions. +The process of [ordering](/docs/sequencer/lifecycle/ordering) involves generating keys on which the interactions are sorted, as well as generating random values in case the contract needs them. +The interaction is then [sent](/docs/sequencer/lifecycle/relaying) to its final destination, Arweave. + +Meanwhile, through notifications dispatched by the sequencer, interactions are stored in a database used by the [Warp Gateway](/docs/gateway/overview) to construct its indexes. +The gateway can then be leveraged by SDK or [Warp D.R.E.](/docs/dre/overview) nodes to retrieve a list of interactions for a contract, enabling the evaluation of the contract's state. + +The path for interactions sent by the client directly to Arweave is slightly different. +Sequencer nodes [download](/docs/sequencer/lifecycle/receiving#syncing-with-arweave) these interactions to insert them into the linear order along with interactions that are sent directly to the sequencer. +Due to the characteristics of the Arweave blockchain, these interactions must be downloaded with a certain delay (hence the clock icon in the diagram below). +For this type of interaction, the sequencer also sends data to Arweave, but only information related to the ordering of the interactions and random values. + +The gateway receives a common list of interactions regardless of whether they were sent to the sequencer or to Arweave. + +![Decentralized Warp Sequencer](/img/docs/sequencer/sequencer.png) + +The stages of the interaction lifecycle will be described in detail in the following sections. \ No newline at end of file diff --git a/warp-academy-docs/docs/sequencer/lifecycle/receiving.md b/warp-academy-docs/docs/sequencer/lifecycle/receiving.md new file mode 100644 index 0000000..0a50f24 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/lifecycle/receiving.md @@ -0,0 +1,91 @@ +# Receiving and Validating Interactions + +The sequencer accepts interactions in the form of [DataItem](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md) via the endpoint `/api/v1/data-item` (refer to: [API Methods](/docs/sequencer/api-methods#submits-a-new-interaction-to-the-sequencer)). +Interactions added to the sequencer's blockchain take the form of messages. +The message that encapsulates an interaction in the DataItem format is `MsgDataItem` (see: [Messages format](/docs/sequencer/data-format#interaction-message)). +Consequently, upon receiving an interaction, a sequencer node generates a message of type `MsgDataItem` based on the received interaction. +Finally, a transaction is created whose only message is this interaction. + +:::tip +A [transaction](https://docs.cosmos.network/main/learn/advanced/transactions) in the Cosmos SDK represents a set of messages that trigger state transitions within the blockchain. +::: + +---- +[DIAGRAM FOR RECEIVING INTERACTIONS] + +---- + +## Validation + +Subsequently, the transaction undergoes a validation process to ensure its correctness. +The validation of the interaction encompasses the following checks: + +* The interaction has the correct format (according to the [standard](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md#21-verifying-a-dataitem)). +* The interaction has the correct tags (according to the [SmartWeave](/docs/sdk/advanced/smartweave-protocol#contract-interactions) protocol). +* It has a tag with the correct Nonce value. +* The interaction is correctly signed. + +In addition to validating the interaction itself, the transaction is also validated if it was created by another sequencer node. +In this scenario, the following checks are performed: + +* The transaction contains exactly one message containing the interaction. +* No additional fields are set (`Memo`, `TimeoutHeight`, etc.). +* The public key of the transaction signature belongs to the owner of the interaction. + +In addition to the checks mentioned above, the sequencer also creates an account for the sender of the interaction (if it does not exist yet) and updates the `Nonce` value for that account. + +Once the transaction successfully completes the validation process, it is broadcasted to the remaining nodes in the network and added to the mempool for further processing. + +## Mempool + +The mempool, short for memory pool, in sequencer networks serves as a temporary storage area for interactions that have been broadcasted to the sequencer but have not yet been included in a block. +It functions as a dynamic buffer, holding unconfirmed transactions until they are selected for inclusion in a future block during the consensus process. +The mempool plays a crucial role in maintaining transaction flow, ensuring the synchronization of nodes within the network, allowing for efficient transaction management and reducing potential bottlenecks in the processing pipeline. +The mempool's transient nature ensures that only valid and properly formatted transactions proceed to be included in the blockchain, promoting the integrity of the sequencer network. + +The order of interactions within the mempool follows the "First-Come-First-Served" (FCFS) principle. +This means that interactions are processed and included in the mempool in the order in which they are received by the node. +FCFS ensures a fair and straightforward queuing mechanism, where interactions are handled based on their arrival time, contributing to a transparent and predictable processing flow. + +:::info +The mempool has a maximum size. +When the mempool is full, the sequencer will stop accepting new interactions until the mempool has been cleared. +This is done to prevent the mempool from becoming too large and to ensure that the sequencer can process interactions in a timely manner. +::: + +## Syncing with Arweave + +In addition to receiving interactions from the SDK, sequencers also download interactions that have been directly submitted to Arweave. +To achieve this, each sequencer node periodically queries Arweave for the latest blocks, following a specific algorithm: + +* Several Arweave peers are queried simultaneously. +* A block is processed further if the majority of peers return the same block. +* Blocks are also validated for an extra layer of security. +* Blocks whose height is less than the current network height by 10 are downloaded. This is to prevent forks. +* For each retrieved block, interactions related to the SmartWeave protocol are extracted. +* The downloaded block, along with its interactions, is appended to the local Arweave block queue, where it awaits inclusion in the sequencer's blockchain. + +The sequencer selectively stores only the essential information required for establishing the linear order of interactions, rather than duplicating data readily available in Arweave. +For Arweave blocks, the sequencer retains the necessary data such as height, timestamp, and block hash (and specifically the value of [indep_hash](https://docs.arweave.org/developers/arweave-node-server/http-api#block-format)). +Regarding interactions, it focuses on the interaction ID and contract ID. +This streamlined data storage approach ensures that the sequencer maintains the required details for sequencing without unnecessary duplication of data already present in Arweave. + +### Arweave as an External Data Source + +From the perspective of the sequencer network, Arweave is an external data source. +This means that it can be considered a source of non-determinism, which can prevent sequencer nodes from reaching consensus. +This is extremely important, as it can lead to inconsistency between nodes and the network coming to a halt. + +Therefore, it is important to ensure that the Arweave block download mechanism is as secure as possible. +This is why the following safeguards are in place: +* **Height-Based Block Selection:** +Only blocks that are less than 10 blocks behind the current height are downloaded. +This helps to prevent the situation where different sequencer nodes download different blocks as a result of a fork. + +* **Timestamp Synchronization:** +Downloaded Arweave blocks are held in the local queue until their timestamp is at least half an hour earlier than the timestamp of the current sequencer block. +This time window allows sequencer nodes to accommodate potential delays or reattempt block retrieval in case of network issues. + +* **Consistency Retry Mechanism:** +If 10 consecutive attempts to achieve consensus fail due to inconsistencies in Arweave blocks, local queues of these blocks are cleared. +Nodes then initiate a fresh attempt to retrieve Arweave blocks. diff --git a/warp-academy-docs/docs/sequencer/lifecycle/relaying.md b/warp-academy-docs/docs/sequencer/lifecycle/relaying.md new file mode 100644 index 0000000..7ae6970 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/lifecycle/relaying.md @@ -0,0 +1,102 @@ +# Relaying Process + +The Warp Sequencer functions as an intermediate step in the lifecycle of interactions but it is not the final destination. +Arweave, as the Layer 1 solution, is the ultimate repository for these interactions. + +## The Crucial Role of Arweave + +While the Warp Sequencer efficiently establishes the order of interactions within its Layer 2 framework, the ultimate goal is to securely anchor these interactions on Arweave, the decentralized Layer 1 protocol. +This transition from Warp to Arweave is not just a procedural step; it is a strategic move with profound implications for the reliability, permanence, and decentralization of the interactions within the network. + +Why Arweave integration matters: +#### Data Permanence +Arweave is renowned for its immutable and permanent storage capabilities. +Unlike the transitory nature of Layer 2 solutions, Arweave ensures that once interactions are recorded, they become an indelible part of the blockchain, creating an enduring record of the network's history. +#### Decentralized Security +Arweave's decentralized security model provides a stark contrast to the relatively smaller Warp network. +By anchoring interactions on Arweave, the network leverages the decentralized and tamper-resistant nature of the blockchain, enhancing the reliability of interactions. +#### Focused Acceleration +Warp Sequencer is purposefully designed to serve as a catalyst for smart contracts on Arweave, rather than aspiring to become an independent network. +The primary ambition is to expedite and enhance the functionality of smart contracts within the robust infrastructure provided by Arweave. +#### Network Collaboration +Warp's role is collaborative rather than competitive. +By focusing on Arweave as the primary repository for interactions, the network aligns itself with the overarching vision of building a sustainable and resilient decentralized ecosystem. +This collaboration fosters a robust foundation for future growth and innovation on Arweave. + +## Decentralized Interaction Submission to Arweave + +In the same decentralized spirit as the sequencing process, the submission of interactions to Arweave should also unfold through a decentralized mechanism. +It is imperative that every interaction sender can be confident that by submitting their interaction to the sequencer, it will ultimately find its way to Arweave. + +---- +[DIAGRAM FOR RELAYER] + +---- + +### Mechanism of Interaction Submission + +- The node that is currently responsible for forwarding interactions to Arweave is called the *Relayer*. +- Only one node serves as the relayer at any given time. +- The selection of the relayer from the pool of active validators follows a process analogous to [proposer selection](/docs/sequencer/lifecycle/ordering#block-proposal). +- The difference is that the proposer is selected for each round, and the relayer is selected after the previous relayer's turn has ended. +- The selected relayer has a defined timeframe to send a specified number of interactions to Arweave. +- The relayer receives rewards for each successfully submitted block, incentivizing them to dispatch as many blocks as possible. +It is crucial to note that the relayer can only send a block for which all blocks of lower height have been sent. +- The relayer utilizes [Turbo](https://ardrive.io/turbo/), responsible for ensuring finality in data submission to Arweave. +- After sending data to Turbo, the relayer receives a receipt, which is then added to the sequencer network. +- Other nodes validate the received receipt, particularly checking if it is correctly signed by Turbo. +- Failure to fulfill the task within the specified time may result in penalties for the relayer. +- In case of Turbo unavailability, the relayer has the option to report this to other nodes to avoid penalties for non-submission, given that the issue is beyond their control. + +:::info +In the initial phase of decentralized sequencer operation, the relayer functions as an off-network application. +However, a first milestone on the [roadmap](/docs/sequencer/roadmap) for sequencer development involves migrating this functionality onto the sequencer itself, following the outlined scheme. +::: + +### Data Format for Arweave Submission + +The format of data sent to Arweave depends on the type of interaction. + +Interactions that were sent directly to Arweave and then retrieved by the sequencer do not need to be sent back to Arweave. +For such interactions, only the values assigned by the sequencer are sent. +This includes the sort key, prev sort key, random value, information about the Arweave block containing the interaction, and details about the sequencer block in which the Arweave block was added. + +For interactions sent to the sequencer, the same data as mentioned above is sent. +However, the original interaction that the sequencer received from the client is also sent. +This is accomplished through a nesting mechanism (see: [Nested bundle](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md#31-nested-bundle)). +In other words, a DataItem is sent, which contains sequencer-specific data and simultaneously wraps another DataItem that is the original interaction. + +The DataItem containing sequencer-specific data includes the following tags. + +#### Global Tags + +| Tag Name | Tag Value | +| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| `Sequencer` | `Warp` | +| `Source` | - `warp` for interactions sent to Warp Sequencer
- `arweave` for interactions sent directly to Arweave | +| `Env` | Environment - `prod` for the production environment, example of other environments: `dev`/`test` | + +#### Block Data Tags + +| Tag Name | Tag Value | +| ------------------------------------------- | --------------------------------------------------------------- | +| `Arweave-Block-Height` | Height of the latest Arweave block added to the sequencer | +| `Arweave-Block-Timestamp` | Timestamp of the latest Arweave block added to the sequencer | +| `Arweave-Block-Hash` | Hash of the latest Arweave block added to the sequencer | +| `Sequencer-Height` | Height of the sequencer block | +| `Sequencer-Timestamp` | Timestamp of the sequencer block | + +#### Interaction Data Tags + +| Tag Name | Tag Value | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------| +| `Tx-Id` | Interaction ID | +| `Contract-Tx-Id` | Transaction ID with the contract | +| `Sort-Key` | Sort key | +| `Prev-Sort-Key` | Sort key of the previous interaction in this contract (only if it is not the first interaction) | +| `Random` | Pseudorandom value for the interaction | + +:::tip +The above data is used, among others, by the [Arweave Interactions Loader](/docs/sdk/advanced/arweave-interaction-loader) +(specifically, by the implementation adapted for the decentralized sequencer). +::: \ No newline at end of file diff --git a/warp-academy-docs/docs/sequencer/roadmap.md b/warp-academy-docs/docs/sequencer/roadmap.md new file mode 100644 index 0000000..bee85e6 --- /dev/null +++ b/warp-academy-docs/docs/sequencer/roadmap.md @@ -0,0 +1,31 @@ +# Roadmap + +The implementation plan for decentralizing the Warp sequencer involves a gradual transition from a centralized architecture to a distributed system. +This plan encompasses several key steps, including adopting a BFT consensus engine, decentralizing the relayer, implementing transaction encryption, introducing order mechanisms, and opening up to external participants. + +1. The initial step involves replacing the current centralized sequencer with a new decentralized network of sequencers. +The transition from one solution to another should occur as seamlessly as possible for users, ensuring data continuity. +A [Warp SDK](/docs/sdk/overview) version supporting this process will be released. +Once all clients have switched to the new SDK version, the old sequencer will be shut down and the new one will be activated shortly thereafter. +After this step, the sequencer will operate in a decentralized manner, except for the relayer component. + +2. The second step involves decentralizing the [relayer](/docs/sequencer/lifecycle/relaying) component. +Sequencer nodes gradually assume the relayer's role in transmitting data from the sequencer to Arweave. +This decentralization redistributes responsibilities among multiple sequencer nodes, enhancing system resilience and reducing dependence on a single point of failure. +It contributes to a more distributed and robust system architecture. + +3. The next stage focuses on implementing transaction encryption to protect against content-based attacks. +Encryption mechanisms are incorporated to preserve the confidentiality of transaction data. +This strengthens the overall security of the system by making it more difficult for the [proposer](/docs/sequencer/lifecycle/ordering#malicious-proposer) to manipulate the order of transactions for malicious purposes. + +4. The plan’s fourth step aims to restrict the proposer’s freedom in selecting and sequencing transactions. +This is achieved by introducing mechanisms that impose an order based on trusted external sources or through consensus among validators. +This approach enhances fairness, transparency, and decentralization in the transaction [ordering](/docs/sequencer/lifecycle/ordering) process, minimizing the potential for manipulation or biases. +Further discussions are required to finalize the details of this step. + +5. The final step involves gradually opening up the network, which was previously controlled by the Warp team. +This phase occurs once the decentralization measures have been successfully implemented and thoroughly tested. +Opening up the network fosters inclusivity, community involvement, and innovation, allowing external stakeholders to contribute, propose improvements, and participate in consensus and decision-making processes. + +Through these steps, the Warp protocol aims to establish a decentralized sequencer network that not only ensures the reliable ordering and processing of transactions but also encourages community involvement and trust among participants. +By embracing decentralization, the Warp sequencer will pave the way for a more secure, scalable, and resilient future for the Warp contracts protocol and its ecosystem. \ No newline at end of file diff --git a/warp-academy-docs/docs/syncer/introduction.md b/warp-academy-docs/docs/syncer/introduction.md index 1e74c36..579f9cb 100644 --- a/warp-academy-docs/docs/syncer/introduction.md +++ b/warp-academy-docs/docs/syncer/introduction.md @@ -39,7 +39,7 @@ Afterward, Syncer moves to assign the ID of the latest block that it has process It's worth noting the deep integration between the Warp Syncer and Sequencer, both vital components of the SDK that closely interplay. At present, the Sequencer derives `SortKey` based on the latest block height served by Syncer. -If you're keen on delving deeper into the intricacies of Warp Sequencer, you can find more information [*here*](/docs/sdk/advanced/bundled-interaction). +If you're keen on delving deeper into the intricacies of Warp Sequencer, you can find more information [*here*](/docs/sequencer/introduction). In the earlier version of Warp Syncer, the `SortKey` for L2 interactions was determined using the `block_height` metric, fetched directly from the arweave.net gateway. Now, the Sequencer sources the block height metric from the internal Postgres database. This metric represents the latest block height captured after the primary synchronization module, outlined [*here*.](/docs/syncer/introduction#high-level-syncer-workflow) This updated approach offers enhanced finality guarantees for the synced block. By eliminating the risk of including forked blocks, it ensures that the Sequencer's L2 transactions will always have a `SortKey` higher than the last fully synced Arweave block. This maintains the integrity of the transaction sequence and ensures determinism. diff --git a/warp-academy-docs/docusaurus.config.js b/warp-academy-docs/docusaurus.config.js index e807668..b306ee7 100644 --- a/warp-academy-docs/docusaurus.config.js +++ b/warp-academy-docs/docusaurus.config.js @@ -195,7 +195,7 @@ const config = { prism: { theme: lightCodeTheme, darkTheme: darkCodeTheme, - additionalLanguages: ['rust', 'toml'], + additionalLanguages: ['rust', 'toml', 'protobuf'], }, algolia: { appId: 'MCHYFKZV38', diff --git a/warp-academy-docs/sidebars/docsSidebar.js b/warp-academy-docs/sidebars/docsSidebar.js index 1600a77..767e7a6 100644 --- a/warp-academy-docs/sidebars/docsSidebar.js +++ b/warp-academy-docs/sidebars/docsSidebar.js @@ -102,9 +102,27 @@ const sidebars = { ], }, { - type: 'doc', + type: 'category', label: 'Warp Sequencer', - id: 'sdk/advanced/bundled-interaction', + items: [ + { type: 'doc', label: 'Introduction', id: 'sequencer/introduction' }, + { type: 'doc', label: 'Centralized Sequencer', id: 'sequencer/centralized' }, + { type: 'doc', label: 'Decentralized Sequencer', id: 'sequencer/decentralized' }, + { + type: 'category', + label: 'Interaction Lifecycle', + items: [ + { type: 'doc', label: 'Overview', id: 'sequencer/lifecycle/overview' }, + { type: 'doc', label: 'Creating and sending', id: 'sequencer/lifecycle/creating' }, + { type: 'doc', label: 'Receiving and validating', id: 'sequencer/lifecycle/receiving' }, + { type: 'doc', label: 'Ordering', id: 'sequencer/lifecycle/ordering' }, + { type: 'doc', label: 'Relaying', id: 'sequencer/lifecycle/relaying' }, + ], + }, + { type: 'doc', label: 'Data Format', id: 'sequencer/data-format' }, + { type: 'doc', label: 'API Methods', id: 'sequencer/api-methods' }, + { type: 'doc', label: 'Roadmap', id: 'sequencer/roadmap' } + ], }, { type: 'category', diff --git a/warp-academy-docs/static/img/docs/sequencer/sequencer.png b/warp-academy-docs/static/img/docs/sequencer/sequencer.png new file mode 100644 index 0000000..f0bfc6f Binary files /dev/null and b/warp-academy-docs/static/img/docs/sequencer/sequencer.png differ diff --git a/warp-academy-docs/tutorials/evm-wallet.md b/warp-academy-docs/tutorials/evm-wallet.md index 58fe083..75beb74 100644 --- a/warp-academy-docs/tutorials/evm-wallet.md +++ b/warp-academy-docs/tutorials/evm-wallet.md @@ -6,7 +6,7 @@ Apart from Arweave wallet, you can connect EVM wallet to your contract. By readi 1. You can connect EVM wallet only in browser environment. 2. User needs to have Metamask plugin installed. -3. It is possible to use EVM wallet only when bundling transaction. See [this feature](/docs/sdk/advanced/bundled-interaction). +3. It is possible to use EVM wallet only when bundling transaction. See [this feature](/docs/sequencer/introduction). Most importantly - these obstacles will not last forever as Warp team is planning to develop the feature further. For now, treat is an experimental feature which can expand the capabilities of your application by inviting users from other blockchain than Arweave. As always - if you find it useful or would like to make a suggestion, please reach to the Warp team on [on Discord](https://discord.com/invite/PVxBZKFr46). diff --git a/warp-academy-docs/tutorials/ml/chapter-5.md b/warp-academy-docs/tutorials/ml/chapter-5.md index d2f5c3f..59b5559 100644 --- a/warp-academy-docs/tutorials/ml/chapter-5.md +++ b/warp-academy-docs/tutorials/ml/chapter-5.md @@ -80,7 +80,7 @@ async function postBtcPrice(message, context) { exports.postBtcPrice = postBtcPrice; ``` :::info -The above function is posting the transactions via [Warp Sequencer](../../docs/sdk/advanced/bundled-interaction). When using the Warp Sequencer, +The above function is posting the transactions via [Warp Sequencer](../../docs/sequencer/introduction). When using the Warp Sequencer, you don't need to have any ARs, tokens or weird, web2-ish API keys. All you need is a wallet (which can be generated on the fly) that will allow to identify who is interacting with the contract.