From 1e1c2a1ed8af2c4fb860d29621526093a4e51294 Mon Sep 17 00:00:00 2001 From: SK Date: Wed, 26 Jun 2024 12:12:34 +0530 Subject: [PATCH] broken link fixes and security content --- .../docs/architecture/apis-and-interfaces.mdx | 12 +- .../consensus-and-sharding-mechanism.mdx | 59 ++++++++ .../architecture/high-level-architecture.mdx | 143 +++++++++++++++++- content/docs/architecture/meta.json | 2 +- .../architecture/validator-operations.mdx | 10 +- content/docs/faqs/glossary.mdx | 2 - content/docs/faucet/claim.mdx | 6 + vercel.json | 15 -- 8 files changed, 217 insertions(+), 32 deletions(-) create mode 100644 content/docs/architecture/consensus-and-sharding-mechanism.mdx delete mode 100644 vercel.json diff --git a/content/docs/architecture/apis-and-interfaces.mdx b/content/docs/architecture/apis-and-interfaces.mdx index c5eae70..eee4d7b 100644 --- a/content/docs/architecture/apis-and-interfaces.mdx +++ b/content/docs/architecture/apis-and-interfaces.mdx @@ -66,9 +66,8 @@ Shardeum supports several interaction models to cater to different types of netw * `package.json`: Lists dependencies and scripts for building and running the server. * `tsconfig.json`: TypeScript configuration file (not retrieved, file does not exist). - **Setup Instructions** -Setup Instructions +**Setup Instructions** Docker Setup: @@ -160,12 +159,9 @@ The Shardeum Explorer Server allows users to interact with the Shardeum blockcha The Explorer Server provides several API endpoints categorized into different groups: -* **Blocks:** - * Retrieve information about blocks. -* **Transactions:** - * Fetch details about specific transactions. -* **Accounts:** - * Query information related to specific accounts on the blockchain. +* **Blocks:** Retrieve information about blocks. +* **Transactions:** Fetch details about specific transactions. +* **Accounts:** Query information related to specific accounts on the blockchain. #### Example API Usage diff --git a/content/docs/architecture/consensus-and-sharding-mechanism.mdx b/content/docs/architecture/consensus-and-sharding-mechanism.mdx new file mode 100644 index 0000000..ddb4d2f --- /dev/null +++ b/content/docs/architecture/consensus-and-sharding-mechanism.mdx @@ -0,0 +1,59 @@ +--- +title: Consensus and Sharding Mechanism +description: Consensus and Sharding Mechanism in Shardeum +--- + +### Why are they so important? + +Shardeum's approach to consensus and sharding involves complex mechanisms to ensure network robustness and efficiency. The consensus mechanism is deeply connected with the network's dynamic sharding system, allowing for effective node coordination and lifecycle management. Here you can explore what [sharding is in-depth](/docs/introduction/sharding). + +### **1. Consensus Mechanism:** + +* The Shardeum network utilizes a trustless consensus system where nodes must agree on the data before it becomes part of the next cycle. +* This process uses advanced algorithms to ensure that all nodes reach a consensus on the transaction records. +* Each node generates cycle records, akin to traditional blockchain blocks but are unique to Shardeum's structure. +* Here you can take a deep dive into the [consensus algorithms used by Shardeum](/docs/faqs/glossary#proof-of-stake-pos) and [Proof of Quorum](/docs/faqs/glossary#proof-of-quorum-poq)", focusing on their design and implementation. + +### **2. Node Lifecycle Management:** + +* **Node Awareness:** In Shardeum, every active validator is aware of all other active validators. This comprehensive awareness is crucial for maintaining deterministic sharding and ensuring accurate consensus. +* **Lifecycle:** Nodes in Shardeum transition from a single seed node to a network full of active and standby nodes. Active nodes handle transaction processing, while standby nodes are prepared to become active as needed, enhancing network resilience. +* **Selection Process:** The transition from standby to active nodes is determined by a deterministic lottery, ensuring randomness and reducing the risk of manipulation. +* Here you can delve deeply into [the Shardeum Node Lifecycle](/docs/architecture/node-lifecycle). + +### **3. Dynamic Sharding Approach:** + +* Shardeum employs dynamic sharding to distribute data and computational tasks across multiple nodes efficiently. This system adjusts the shard allocation based on the network's needs, optimizing both storage and processing resources. +* Shards are managed so that no single node has a complete set of data, enhancing security and performance. The network's sharding mechanism is designed to adapt to changes in node count and transaction volume, allowing for scalable and responsive network operations. +* These combined mechanisms enable Shardeum to manage a large network of nodes, handle high transaction volumes, and maintain security and consistency across the distributed ledger. +* You may want to see this page, if you want to see [more technical details, regardless how data and transaction sharding are implemented, including the mathematical foundations](/docs/introduction/sharding). + + +## Sharding + +Sharding is a method of partitioning data and computation to achieve scalability, allowing a network to handle more transactions per second (TPS) as it grows. Shardeum's sharding mechanism dynamically distributes data and transaction processing across multiple nodes, ensuring efficient resource utilization and maintaining low fees. + +### 2. Access Lists and Memory Control + +* Access lists are used to manage and control memory access during transaction execution. In Shardeum, each transaction must specify the parts of memory it will read from or write to. This pre-determined list of memory locations, known as an access list, allows the system to predict and schedule transactions efficiently, avoiding conflicts and ensuring that transactions touching the same memory locations are handled correctly. This mechanism was introduced because the EVM lacks built-in concurrency primitives, requiring external control over memory access to manage concurrency effectively. +* **Automatic Access List Generation:** Shardeum employs an automatic access list generation mechanism to streamline this process. When a transaction is initiated, the system automatically determines which memory locations will be accessed. This includes both the data that will be read and the data that will be modified. This automatic generation helps to maintain consistency and predictability in transaction execution. The system can compute the access list before running the transaction, although this can be challenging due to the dynamic nature of certain operations, such as generating random numbers based on block data or gas prices. +* **Concurrency Control:** A major challenge in a sharded network like Shardeum's is ensuring that multiple transactions do not conflict when accessing the same memory locations. Shardeum addresses this by using access lists to control concurrency. If two transactions attempt to modify the same memory location simultaneously, the system schedules them in a way that ensures one completes before the other begins, thereby maintaining data integrity. This scheduling is crucial for preventing nondeterministic behavior in transaction execution. +* **Retry and Recirculation Mechanisms:** To enhance user experience, Shardeum includes retry and recirculation mechanisms for transactions that fail due to unexpected memory access issues. Instead of immediately rejecting such transactions, the system can retry them or recirculate them until they can be executed successfully. +* **Gas Pricing and Optimization:** Access lists also facilitate more sophisticated gas pricing mechanisms. By analyzing access patterns and memory usage, Shardeum can dynamically adjust gas prices to reflect the actual computational and storage costs. For example, operations that access multiple storage locations might incur higher fees than those that only access a few. This dynamic pricing helps to balance load across the network and prevent any single transaction type from disproportionately consuming resources. Also by knowing in advance which parts of memory will be accessed, gas estimation can be done more accurately. +* **Real-World Example: Dynamic Recipient Transactions:** Consider a transaction where the recipient is determined dynamically. For instance, a smart contract might generate a random number based on the block data and current gas price. If the random number is greater than 0.5, the funds are sent to Bob; if it is less than 0.5, the funds are sent to Sally. This transaction's access list must account for both possible recipients' addresses. The system pre-runs the transaction to generate the access list, but the actual execution might still encounter new memory requirements due to the dynamic nature of the recipient address. +* **Surprise Memory Access:** Shardeum supports a technical design for handling surprise memory access during transaction execution. If a transaction needs to access a memory location not included in its initial access list, it can request permission at the protocol level. This request is processed as a transaction where nodes in the network check if the new memory access can be granted without disrupting the order of existing transactions. If approved, the transaction can proceed; if not, it is retried or recirculated without user intervention to avoid a poor user experience. +* **Security Considerations:** Access lists also enhance security by preventing unauthorized memory access. Transactions are restricted to accessing only the memory locations specified in their access lists. If a transaction attempts to access a location not listed, it is immediately flagged and halted. This mechanism prevents potential exploits where a malicious transaction might try to read or modify data it should not have access to. + +### 3. Access List Warm-Up Technology + +* The access list warm-up process involves running a preliminary phase where hints about the memory locations that will be accessed are gathered. This phase is used to parallelize data fetching, significantly reducing the time required to generate access lists. By warming up memory slots, Shardeum can efficiently prepare for transaction execution, minimizing delays caused by serial data fetches. +* **Performance Improvements:** This technology allows for a significant reduction in access list generation times. During testing, complex transactions that previously took up to 50 seconds to generate an access list saw a drastic reduction in processing time to just a few seconds. This improvement is critical for maintaining high throughput and low latency in the network, especially for transactions that involve numerous memory accesses. +* **Security Enhancements:** Access list warm-up also enhances security by decoupling the process of access list generation from the actual transaction execution. This separation ensures that potential security risks associated with user-defined access lists are mitigated. By pre-fetching data in a controlled manner, the network can prevent malicious attempts to exploit memory access patterns. +* **Compatibility and Flexibility:** The warm-up process is designed to be flexible and compatible with various transaction types and memory access patterns. It supports both the predefined access lists and those that might change dynamically during execution. This flexibility ensures that even transactions with unpredictable memory usage can be handled efficiently. + +### 4. Dynamic Sharding and Address Space Management + +* Shardeum employs dynamic sharding, where nodes are responsible for specific ranges of the address space, ensuring balanced data distribution and efficient processing. Each node dynamically adjusts its range of responsibility as new nodes join or leave the network. This dynamic approach ensures that the network can scale smoothly without significant disruptions. This approach contrasts with vertical scaling, where a single node is upgraded to handle more load, which can be less efficient and more costly. +* **Adaptation to Network Conditions:** Dynamic sharding allows Shardeum to adapt to changing network conditions and workloads. As nodes join or leave, the address space is redistributed to maintain balance. This ensures that no single node becomes a bottleneck and that the network can continue to operate efficiently even as it scales. The auto-scaling feature detects when to scale up or down, allowing the network to grow and shrink efficiently. +* **Shard Division:** In a dynamic sharding system, the address space is divided into multiple shards, each managed by a different set of nodes. For example, in a network with 512 nodes and a consensus group size of 128, the network is divided into four shards, each responsible for a quarter of the total address space. As nodes are added or removed, the boundaries of these shards adjust to maintain balance and ensure that each node handles a proportional amount of data and transactions. If every node in the network had a single shard, then every node would store all the data. However, as more nodes are added to the network, the consensus group size determines how the data is divided among the nodes. This division allows each node to handle a portion of the workload, thus improving the network's overall efficiency and speed. For instance, if we have a small Shardeum network with a consensus group size of 128 and 128 nodes or fewer, we essentially have a single shard where every node stores all the data. However, as soon as we add one more node, the network dynamically shards, resulting in significant speedup. This is because the workload and data storage are distributed among more nodes, and each node handles only a fraction of the total transactions and data. +* **Gas Pricing Optimization:** Shardeum also optimizes gas prices by adjusting them based on the complexity and resource usage of transactions. This dynamic pricing model ensures that simple transactions remain inexpensive, while more complex transactions that require more resources are priced accordingly. This approach helps prevent abuse and ensures fair resource allocation across the network. \ No newline at end of file diff --git a/content/docs/architecture/high-level-architecture.mdx b/content/docs/architecture/high-level-architecture.mdx index d6b5701..caabffe 100644 --- a/content/docs/architecture/high-level-architecture.mdx +++ b/content/docs/architecture/high-level-architecture.mdx @@ -25,4 +25,145 @@ description: Shardeum’s high level architecture ### **4. More about how it works**: * **Handling data:** The core protocol takes care of the big picture stuff like making sure all parts of the network are working together. It doesn’t store or deal with specific dapp data; dapps handle their own data and only talk to the core protocol for making sure everyone agrees. -* **Managing nodes:** The rules about how nodes (the individual parts of the network) operate are detailed in the protocol's documentation. [The core protocol manages whether nodes are active or on standby](/docs/architecture/apis-and-interfaces.md), but it doesn’t get involved in the specific data that nodes deal with for dapps. +* **Managing nodes:** The rules about how nodes (the individual parts of the network) operate are detailed in the protocol's documentation. [The core protocol manages whether nodes are active or on standby](/docs/architecture/apis-and-interfaces), but it doesn’t get involved in the specific data that nodes deal with for dapps. + +---------------------------------------------------------------------------------------------------------------------------------------------------------- + +## Overview + +Shardeum is an EVM based L1 that uses dynamic state sharding to achieve linearly scalability while attaining atomic composability across shards. This means Shardeum can increase its TPS capacity with each validator added to the network to retain low fees forever. + +Shardeum provides the highest throughput capacity of any EVM based L1 without sacrificing on decentralization. Developers can deploy and interact with Solidity or Vyper contracts without special considerations for sharding, since contracts are deployed to unique shards automatically while retaining atomic composability across all shards. + +Shardeum is the solution to the blockchain trilemma. As blockchain tries to achieve scalability, decentralization and security, it will only be able to attain two. There must be a trade-off between decentralization and scalability with security an essential element. Shardeum achieves scalability, decentralization and still maintains security. + +Shardeum’s auto-scaling feature allows the network to automatically adjust the number and size of shards based on the current workload. This allows the system to optimize performance and maintain high levels of scalability as it grows and evolves. + + +### 1. Integration + +* Shardeum is built on the foundation of the Shardus Core Protocol, presenting itself as the first [EVM](https://ethereum.org/en/developers/docs/evm/)-compliant Layer 1 blockchain, optimized for dynamic sharding. +* This architecture uniquely enables the network to scale transaction throughput linearly with the addition of nodes, addressing the scalability issues commonly faced by traditional blockchains. +* Shardeum is designed to support a diverse array of DApps, making it a versatile platform for developers and businesses. + +### 2. Shardeum's Features + +* **Dynamic State Sharding:** This is the most advanced and complex way to shard the state of a network because it shards the State, Network, and Transactions dynamically, as opposed to a pre-determined way. +* **Linear Scalability:** Specifically refers to the fact that a system can increase its throughput as more resources are provided, and the relationship between throughput and resources is linear. +* **Low Transaction Fees:** The minimal costs associated with processing and confirming transactions on a blockchain network. These fees are paid by users to incentivize miners or validators to include their transactions in the next block of the blockchain. +* **EVM-based Smart Contract Platform:** Refers to the compatibility of a blockchain network with the Ethereum Virtual Machine (EVM), allowing it to execute smart contracts and transactions that are compatible with Ethereum's existing ecosystem. +* **Solves Scalability Trilemma:** Achieves a balance between decentralization, security, and scalability. Typically, enhancing scalability can compromise either decentralization or security, but innovative solutions like sharding, layer 2 protocols, and consensus algorithm improvements aim to address all three aspects simultaneously. +* **Auto Scaling:** Adjusts computational resources automatically based on demand, ensuring optimal performance and cost efficiency. It handles traffic spikes efficiently, providing a seamless user experience without manual intervention. +* **Security:** Ensures the protection of data integrity, transaction authenticity, and network resilience against attacks. Strong cryptographic protocols and consensus mechanisms are fundamental to maintaining high security in blockchain systems. +* **Immediate Finality:** Ensures that once a transaction is confirmed, it cannot be altered, reversed, or double-spent. This is crucial for applications requiring quick and definite transaction settlements. +* **Low Latency:** Refers to the minimal delay between the initiation of a transaction and its confirmation, essential for applications needing quick transaction processing and user responsiveness. +* **Low Bandwidth:** Operates efficiently with minimal data transmission requirements, reducing operational costs and enabling access in regions with limited internet infrastructure. +* **High Fairness:** Ensures that all participants have equal opportunities to validate transactions and propose new blocks, maintaining fairness through decentralized consensus mechanisms and transparent protocols. +* **High Capacity:** The network's ability to handle a large volume of transactions per second (TPS), crucial for scalability, enabling the network to support a growing number of users and applications without performance degradation. + +### **3. Shardus Core Protocol** +* **Base protocol:** [The Shardus Core Protocol](https://shardus.com/) handles sharding and consensus without needing to know the specifics of application-level transactions. This separation ensures modularity and security, allowing the protocol to function independently of the DApps running on top of it. + Here, you can [take a look](https://github.com/shardeum/shardeum/blob/cd230c50859d28d17b3d013fee52923ba6c495aa/src/index.ts#L3354-L7347) at how we successfully integrate the Shardus software. + +``` +const shardusSetup = (): void => { + /** + * interface tx { + * type: string + * from: string, + * to: string, + * amount: number, + * timestamp: number + * } + */ + shardus.setup({ + sync: sync(shardus, evmCommon), + validateTransaction: validateTransaction(shardus), + validateTxnFields: validateTxnFields(shardus, debugAppdata), + isInternalTx, + async apply(timestampedTx: ShardusTypes.OpaqueTransaction, wrappedStates, originalAppData) { +``` + + +* **API interaction:** DApps interact with the Shardus Core Protocol through a [defined API](/docs/architecture/apis-and-interfaces). This abstraction ensures that the core protocol does not directly access or manipulate application data, maintaining a clear boundary between the protocol and the applications. + + +### **4. EVM Compliance** + + +* **Smart contracts:** Shardeum supports the execution of [Ethereum](https://ethereum.org/en/developers/docs/) smart contracts, allowing developers to deploy their existing contracts without modification. +* **Tool compatibility:** All tools and applications built for Ethereum can run on Shardeum, facilitating easy migration and adoption. +* **Execution environment:** By maintaining EVM compatibility, Shardeum leverages the robustness and familiarity of the Ethereum ecosystem, making it accessible for developers. +* **EVM Initialization:** [Click here to examine the code in our repos.](https://github.com/shardeum/shardeum/blob/cd230c50859d28d17b3d013fee52923ba6c495aa/src/index.ts#L405-L512) + +```typescript +if (ShardeumFlags.UseDBForAccounts === true) { + AccountsStorage.init(config.server.baseDir, `${FilePaths.SHARDEUM_DB}`) + if (isServiceMode()) AccountsStorage.lazyInit() +} + +const appliedTxs = {} +const shardusTxIdToEthTxId = {} + +const defaultBalance = isDebugMode() ? oneSHM * BigInt(100) : BigInt(0) + + +const ERC20TokenBalanceMap: { + to: string + data: unknown + timestamp: number + result: unknown +}[] = [] +const ERC20TokenCacheSize = 1000 + +interface RunStateWithLogs extends RunState { + logs?: [] +} + +let EVM: { -readonly [P in keyof VM] } +let shardeumBlock: ShardeumBlock + +let shardeumStateTXMap: Map + +let shardusAddressToEVMAccountInfo: Map + +export let evmCommon + +let debugAppdata: Map + +async function initEVMSingletons(): Promise { + const chainIDBN = BigInt(ShardeumFlags.ChainID) + + evmCommon = new Common({ chain: 'mainnet', hardfork: Hardfork.Istanbul, eips: [3855] }) + + evmCommon.chainId = (): bigint => { + return BigInt(chainIDBN.toString(10)) + } + + shardeumBlock = new ShardeumBlock({ common: evmCommon }) + + if (ShardeumFlags.useShardeumVM) { + const customEVM = new EthereumVirtualMachine({ + common: evmCommon, + stateManager: undefined, + }) + EVM = await VM.create({ + common: evmCommon, + stateManager: undefined, + evm: customEVM, + }) + } + + shardeumStateTXMap = new Map() + + shardusAddressToEVMAccountInfo = new Map() + + debugAppdata = new Map() +} + +initEVMSingletons() +``` + + + + \ No newline at end of file diff --git a/content/docs/architecture/meta.json b/content/docs/architecture/meta.json index fc1af3f..eceeac8 100644 --- a/content/docs/architecture/meta.json +++ b/content/docs/architecture/meta.json @@ -1,4 +1,4 @@ { "title": "Architecture", - "pages": ["high-level-architecture", "node-lifecycle", "cyclic-operations","transaction-lifecycle","validator-operations","apis-and-interfaces"] + "pages": ["high-level-architecture", "security", "node-lifecycle", "cyclic-operations","transaction-lifecycle","consensus-and-sharding-mechanism","validator-operations","apis-and-interfaces"] } \ No newline at end of file diff --git a/content/docs/architecture/validator-operations.mdx b/content/docs/architecture/validator-operations.mdx index d1d3dfd..1ef5767 100644 --- a/content/docs/architecture/validator-operations.mdx +++ b/content/docs/architecture/validator-operations.mdx @@ -36,12 +36,12 @@ When a user sends a transaction, the state changes (like balance updates) are re This ensures that the entire state can be efficiently verified and the history of state changes can be traced back cryptographically. #### 2.2 Contract Storage -- Smart Contract Accounts:** Smart contracts have specific storage requirements. A smart contract account holds the balance and nonce, and includes a pointer to the code bytes. The code bytes contain the actual machine code of the smart contract. -- Code Bytes:** The code for smart contracts is stored in a type of account called Code Bytes. This is essentially the compiled Solidity code in the form of EVM bytecode. Thus it’s the heaviest one. -- Contract Storage:** Smart contracts also maintain storage, which holds key-value pairs. Each key and value is 256 bits in size, and this storage is used to maintain the state specific to the smart contract. -- Sharded Hash Trie in Contract Storage:** Similar to account storage, contract storage uses the Sharded Hash Trie structure to organize data efficiently. +- Smart Contract Accounts: Smart contracts have specific storage requirements. A smart contract account holds the balance and nonce, and includes a pointer to the code bytes. The code bytes contain the actual machine code of the smart contract. +- Code Bytes: The code for smart contracts is stored in a type of account called Code Bytes. This is essentially the compiled Solidity code in the form of EVM bytecode. Thus it’s the heaviest one. +- Contract Storage: Smart contracts also maintain storage, which holds key-value pairs. Each key and value is 256 bits in size, and this storage is used to maintain the state specific to the smart contract. +- Sharded Hash Trie in Contract Storage: Similar to account storage, contract storage uses the Sharded Hash Trie structure to organize data efficiently. Example: If a smart contract manages a token balance, each user's balance is stored as a key-value pair in the contract storage. The key is the user's address, and the value is their token balance. Changes to these balances are recorded in the SHT, allowing for efficient verification of the entire contract state. -- Complex Data Structures:** For more complex data structures, such as arrays or long strings, the compiler breaks down the data into 256-bit segments. These segments are then stored across multiple contract storage accounts. +- Complex Data Structures: For more complex data structures, such as arrays or long strings, the compiler breaks down the data into 256-bit segments. These segments are then stored across multiple contract storage accounts. ![Contract Storage](/img/new/i11.jpg) diff --git a/content/docs/faqs/glossary.mdx b/content/docs/faqs/glossary.mdx index 04e59d3..e5cf313 100644 --- a/content/docs/faqs/glossary.mdx +++ b/content/docs/faqs/glossary.mdx @@ -3,8 +3,6 @@ title: Glossary of Terms sidebar_position: 2 --- -# Glossary of Terms - * [A](#a) * [Active Nodes](#active-nodes) * [API (Application Programming Interface)](#api-application-programming-interface) diff --git a/content/docs/faucet/claim.mdx b/content/docs/faucet/claim.mdx index d90d33d..68b0f89 100644 --- a/content/docs/faucet/claim.mdx +++ b/content/docs/faucet/claim.mdx @@ -15,6 +15,12 @@ title: Claim Testnet SHM 3. Go to the faucet channel in Shardeum discord. +import { Callout } from 'fumadocs-ui/components/callout'; + + + Atomium faucet only allows 1 claim (8 SHM) every 24 hours + + - Atomium: https://discord.com/channels/933959587462254612/1253938152775749673 - Sphinx: https://discord.com/channels/933959587462254612/1070780355931541514 diff --git a/vercel.json b/vercel.json deleted file mode 100644 index 59e5c65..0000000 --- a/vercel.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "redirects": [ - { - "source": "/faucet/claim", - "destination": "/docs/faucet/claim", - "permanent": true - }, - { - "source": "/faucet/claim", - "destination": "/docs/faucet/claim", - "permanent": true - } - ] - } - \ No newline at end of file