GITBOOK-255: change request with no subject merged in GitBook

Author: testgroundbot

Diff for: .gitbook/assets/Screenshot 2023-05-18 at 13.39.53.png

@@ -8,4 +8,4 @@ Boost exposes libp2p interfaces for making storage and retrieval deals, a web in
 
 ![Web UI - Storage Space screen](<.gitbook/assets/Boost - storage space.png>)
 
-![Web UI - Sealing Pipeline screen](<.gitbook/assets/Boost - sealing pipeline (1).png>)
+![Web UI - Sealing Pipeline screen](<.gitbook/assets/Boost - sealing pipeline.png>)

Diff for: .gitbook/assets/Screenshot 2023-05-18 at 13.45.14.png

@@ -32,6 +32,9 @@
 * [Troubleshooting](troubleshooting.md)
 * [Experimental Features](experimental-features/README.md)
   * [FVM Contract Deals](experimental-features/fvm-contract-deals.md)
+  * [Local Index Directory](experimental-features/local-index-directory/README.md)
+    * [Architecture](experimental-features/local-index-directory/architecture.md)
+    * [Initialisation](experimental-features/local-index-directory/initialisation.md)
 * [GraphQL API](graphql-api.md)
 * [JSON-RPC API](json-rpc-api.md)
 * [FAQ](faq.md)

Diff for: .gitbook/assets/Screenshot 2023-05-18 at 14.01.13.png

@@ -0,0 +1,75 @@
+---
+description: >-
+  This page describes the Local Index Directory component in Boost, what it is
+  used for, how it works and how to start using it
+---
+
+# Local Index Directory
+
+## Background
+
+The Local Index Directory (_LID_) manages and stores indices of deal data so that it can be retrieved by a content identifier (_cid_).
+
+Currently this task is performed by the _DAG store_ component. The DAG store keeps its indexes on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.
+
+LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools.
+
+To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size.
+
+## Storing data on Filecoin
+
+When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way cilents can later retrieve subsets of the original deal data, without retrieving the full deal data.
+
+<figure><img src="../../.gitbook/assets/Screenshot 2023-05-18 at 13.39.53.png" alt=""><figcaption><p>How Boost stores deal data from clients</p></figcaption></figure>
+
+## Retrieving data
+
+When a client makes a request for data by cid, LID:\
+\- checks which piece the cid is in, and where in the piece the data is\
+\- checks which sector the piece is in, and where in the sector the piece is\
+\- reads the data from the sector
+
+<figure><img src="../../.gitbook/assets/Screenshot 2023-05-18 at 13.45.14.png" alt=""><figcaption><p>How clients retrieve their data from Boost</p></figcaption></figure>
+
+## Use cases
+
+The retrieval use cases that the Local Index Directory supports are:
+
+#### Graphsync retrieval
+
+_Request one root cid with a selector, receive many blocks_
+
+LID is able to:\
+\- look up which piece contains the root cid\
+\- look up which sector contains the piece\
+\- for each block, get the offset into the piece for the block
+
+#### Bitswap retrieval
+
+_Request one block at a time_
+
+LID is able to:\
+\- look up which piece contains the block\
+\- get the size of the block (Bitswap asks for the size before getting the block data)\
+\- look up which sector contains the piece\
+\- get the offset into the piece for the block
+
+#### HTTP retrieval
+
+_Request a whole piece_
+
+LID is able to look up which sector contains the piece.
+
+_Request an individual block_
+
+LID is able to:\
+\- look up which piece contains the block\
+\- look up which sector contains the piece\
+\- get the offset into the piece for the block
+
+_Request a file by root cid_
+
+LID is able to:\
+\- look up which piece contains the block\
+\- look up which sector contains the piece\
+\- for each block, get the offset into the piece for the block

Diff for: .gitbook/assets/Screenshot 2023-05-18 at 14.01.23.png

@@ -0,0 +1,35 @@
+---
+description: Local Index Directory architecture and index types
+---
+
+# Architecture
+
+When designing the Local Index Directory we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for:\
+\- small- SPs - holding up to 1PiB), and\
+\- mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data
+
+Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB.
+
+At the moment there are two implementations of LID:\
+\- a simple LevelDB implementation, for small SPs who want to keep all information in a single process database.\
+\- a scalable YugabyteDB implementation, for medium and large size SPs with tens of thousands of deals.
+
+## Index types
+
+In order to support the described retrieval use cases, LID maintains the following indexes:
+
+#### multihash → \[]piece cid
+
+To look up which pieces contain a block
+
+#### piece cid → sector information {sector ID, offset, size}
+
+To look up which sector a piece is in
+
+#### piece cid → map\<mulithash → block offset / size>
+
+To look up where in the piece a block is and the block’s size
+
+<figure><img src="../../.gitbook/assets/Screenshot 2023-05-18 at 14.01.13.png" alt=""><figcaption></figcaption></figure>
+
+<figure><img src="../../.gitbook/assets/Screenshot 2023-05-18 at 14.01.23.png" alt=""><figcaption></figcaption></figure>

Diff for: README.md

@@ -0,0 +1,27 @@
+---
+description: >-
+  This page explains how to initialise LID and start using it to provide
+  retrievals to clients
+---
+
+# Initialisation
+
+Considering that the Local Index Directory is a new feature, Storage Providers should initialise it after upgrading their Boost deployments.
+
+There are two ways a Storage Provider can do that:
+
+1. **Migrate existing indices from the DAG store into LID**: this solution assumes that the Storage Provider has been keeping an _unsealed_ copy for every sector they prove on-chain, and has already indexed all their deal data into the DAG store.\
+   \
+   Typically index sizes for a given sector range between 100KiB up to 1GiB, depending on deal data and its blocks sizes. The DAG store keeps these indices in the repository directory of Boost under the `./dagstore/index` and `./dagstore/datastore` directories. This data should be migrated to LID with the `migrate-lid` utility.\
+
+2. **Recreate indices for deal data based on unsealed copies of sectors**: this solution assumes that the Storage Provider has _unsealed copies_ for every sector they prove on-chain. If this is not the case, then the SP should first trigger an _unseal (UNS)_ job on their system for every sector that contains user data and produce an unseal copy.\
+   \
+   SPs can use the `boostd recover lid` utility to produce an index for all deal data within an unsealed sector and store it in LID so that they enable retrievals for the data. Depending on SPs deployment and where unsealed copies are hosted (NFS, Ceph, external disks, etc.) and the performance of the hosting system, producing an index for a 32GiB sector can take anywhere from a few seconds up to a few minutes, as the unsealed copy needs to be processed by the utility.
+
+## Migrate existing indices from the DAG store into LID
+
+TODO
+
+## Recreate indices for deal data based on unsealed copies of sectors
+
+TODO