From 8808b55b45d2c20bb7d9b1a9262fa8a55e87cb20 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 11 Sep 2023 16:38:53 -0700 Subject: [PATCH 01/24] Skeleton --- plan/README.md | 13 +++++++++++++ plan/ROADMAP.md | 1 + 2 files changed, 14 insertions(+) create mode 100644 plan/README.md create mode 100644 plan/ROADMAP.md diff --git a/plan/README.md b/plan/README.md new file mode 100644 index 00000000..f707fa3f --- /dev/null +++ b/plan/README.md @@ -0,0 +1,13 @@ +This directory contains the planning documentation for the Homestar project. + +Here is a quick overview: + +- πŸ›οΈ [Architecture] +- πŸ—ΊοΈ [Roadmap] + + + + + +[Architecture]: ./ARCHITECTURE.md +[Roadmap]: ./ROADMAP.md diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md new file mode 100644 index 00000000..a8d249a2 --- /dev/null +++ b/plan/ROADMAP.md @@ -0,0 +1 @@ +# Homestar Roadmap From 532085cc478556ee0f6a6c3de9d84e086d356af1 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 11 Sep 2023 17:55:23 -0700 Subject: [PATCH 02/24] Bullet points --- plan/ROADMAP.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index a8d249a2..59cf4b26 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -1 +1,39 @@ # Homestar Roadmap + + + + + +- Networking + - Receipt DHT(?) + - pubsub/gossipsub/episub +- Blockstore + - Host + - Load Wasm & other data by CID + - "Content handles" + - Guest + - Block streaming API (in & out) +- Standard Wasm library +- (Co)effect plugin system + - First-party effects: + - HTTPS + - DNS + - Randomness oracle + - Encryption/decryption +- Dynamic limits + - Fuel + - Memory + - Storage +- Safety + - Dataflow checking +- Trust + - UCAN (re)delegation support + - UCAN invocation support + - Optimistic verification + - Staking + - Slashing + - Later: wasm-on-snark +- Payment + - Delegated payment (channels) + - e.g. ucan-chan and/or interledger + - Settlement From 95005beaff0ccd84d3db2089ff0f4909e681b308 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Tue, 12 Sep 2023 16:46:32 -0700 Subject: [PATCH 03/24] Checkpoint the mermaid --- plan/ROADMAP.md | 133 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 100 insertions(+), 33 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 59cf4b26..f7bb654a 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -1,39 +1,106 @@ # Homestar Roadmap +I can't even begin to describe just how WIP this is at the moment: +```mermaid +flowchart LR + subgraph Networking + Receipt-DHT + pubsub + end + subgraph scheduling + matchmaking-coordination + announce-capabilities + request-tasks + end + subgraph Data + subgraph Host + wasm-load[Load Wasm & other data by CID] + content-handles + end -- Networking - - Receipt DHT(?) - - pubsub/gossipsub/episub -- Blockstore - - Host - - Load Wasm & other data by CID - - "Content handles" - - Guest - - Block streaming API (in & out) -- Standard Wasm library -- (Co)effect plugin system - - First-party effects: - - HTTPS - - DNS - - Randomness oracle - - Encryption/decryption -- Dynamic limits - - Fuel - - Memory - - Storage -- Safety - - Dataflow checking -- Trust - - UCAN (re)delegation support - - UCAN invocation support - - Optimistic verification - - Staking - - Slashing - - Later: wasm-on-snark -- Payment - - Delegated payment (channels) - - e.g. ucan-chan and/or interledger - - Settlement + subgraph Guest + block-streaming[Block streaming API in & out] + end + end + + Wasm_Standard_Library + + subgraph Effects + Effect_Plugin_system + + subgraph first-party-effs + HTTPS + DNS + Randomness_Oracle + Crypto[crypto: encryption & signing] + end + end + + subgraph Dynamic limits + Fuel + Memory + Storage + end + + subgraph Safety + dataflow-checks + end + + subgraph Trust + ucan-delegation + ucan-invocation + + attestation + + subgraph Optimistic-Verification + fair-scheduling + verifying + adjudicating + staking + slashing + end + + wasm-snarks + end + + subgraph Payment + delegated-payment-channels-ucan-or-interlegder + settlement-rails + end + + subgraph DX + subgraph DSLs + Rust + JS + Python + others-eg-apache-beam + end + + subgraph Packaging + static-binary + Docker + brew + crate + end + + docs + end + + subgraph Services + account-system + odd-jobs + dashboard + task-registry + end + + content-handles --> wasm-load + ucan-delegation --> ucan-invocation --> attestation --> Optimistic-Verification -.->|ahead of| wasm-snarks + block-streaming --> wasm-load + matchmaking-coordination --> pubsub + + matchmaking-coordination --> announce-capabilities + matchmaking-coordination --> request-tasks +``` From 70ee70a2448eb8b6a2d5b81b2bb9d345606f9bc4 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Tue, 12 Sep 2023 22:16:20 -0700 Subject: [PATCH 04/24] Update dependency graph --- plan/ROADMAP.md | 87 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 60 insertions(+), 27 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index f7bb654a..49f2b6ba 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -2,8 +2,23 @@ I can't even begin to describe just how WIP this is at the moment: +Dependency Graph + ```mermaid -flowchart LR +flowchart + subgraph Services + account-system + odd-jobs + dashboard + task-registry + ucan-pay + end + + subgraph Payment + delegated-payment-channels-ucan-or-interlegder + settlement-rails + end + subgraph Networking Receipt-DHT pubsub @@ -26,7 +41,10 @@ flowchart LR end end - Wasm_Standard_Library + subgraph Wasm + deterministic-Wasm-runtime + Wasm_Standard_Library + end subgraph Effects Effect_Plugin_system @@ -45,10 +63,6 @@ flowchart LR Storage end - subgraph Safety - dataflow-checks - end - subgraph Trust ucan-delegation ucan-invocation @@ -56,21 +70,17 @@ flowchart LR attestation subgraph Optimistic-Verification - fair-scheduling - verifying - adjudicating - staking - slashing + direction LR + + verify + adjudicate + stake + slash end wasm-snarks end - subgraph Payment - delegated-payment-channels-ucan-or-interlegder - settlement-rails - end - subgraph DX subgraph DSLs Rust @@ -82,25 +92,48 @@ flowchart LR subgraph Packaging static-binary Docker - brew - crate + Nix + Brew + Crate-SDK end docs end - subgraph Services - account-system - odd-jobs - dashboard - task-registry - end - content-handles --> wasm-load - ucan-delegation --> ucan-invocation --> attestation --> Optimistic-Verification -.->|ahead of| wasm-snarks + wasm-snarks -.->|after| Optimistic-Verification + attestation --> ucan-invocation --> ucan-delegation block-streaming --> wasm-load matchmaking-coordination --> pubsub matchmaking-coordination --> announce-capabilities matchmaking-coordination --> request-tasks -``` + + delegated-payment-channels-ucan-or-interlegder --> settlement-rails + + Crypto --> Randomness_Oracle + HTTPS --> DNS + + dashboard --> odd-jobs --> account-system + dashboard --> task-registry --> account-system + + odd-jobs --> deterministic-Wasm-runtime + odd-jobs --> wasm-load + odd-jobs --> scheduling + + ucan-pay --> account-system + ucan-pay --> Payment + + Wasm_Standard_Library --> deterministic-Wasm-runtime + odd-jobs --> Nix --> static-binary + Docker --> static-binary + Brew --> static-binary + static-binary --> Crate-SDK + + adjudicate --> verify + adjudicate --> slash + slash --> stake + stake --> settlement-rails + + slash --> attestation + ``` From 0f63b0203f5960f6f753eee18b9b1333ec91ecb5 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 12:57:52 -0700 Subject: [PATCH 05/24] Backport Figjam to Mermaid --- plan/ROADMAP.md | 283 ++++++++++++++++++++++++++++++------------------ 1 file changed, 179 insertions(+), 104 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 49f2b6ba..db5f80b3 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -6,134 +6,209 @@ Dependency Graph ```mermaid flowchart - subgraph Services - account-system - odd-jobs - dashboard - task-registry - ucan-pay + subgraph Networking + pubsub + car + quic + webt + https + end + + subgraph ca_storage + layered_bs + pub_obj_store + priv_obj_store end - subgraph Payment - delegated-payment-channels-ucan-or-interlegder - settlement-rails + subgraph task_storage + receipt_store + task_registry + wasm_retrieval + host_session_storage end - subgraph Networking - Receipt-DHT - pubsub + subgraph execution + wasm_based_plugin_system + resource_limits + wasm_execution end subgraph scheduling - matchmaking-coordination - announce-capabilities - request-tasks + coordinator + dag_injector + workflow_static_analyser + workflow + affinity_probe + match_maker end - subgraph Data - subgraph Host - wasm-load[Load Wasm & other data by CID] - content-handles - end - - subgraph Guest - block-streaming[Block streaming API in & out] - end + subgraph tracking + host_logging_metrics_traces + workflow_progress_tracker end - subgraph Wasm - deterministic-Wasm-runtime - Wasm_Standard_Library + subgraph capabilities + ucan + did end - subgraph Effects - Effect_Plugin_system - - subgraph first-party-effs - HTTPS - DNS - Randomness_Oracle - Crypto[crypto: encryption & signing] - end + subgraph trust + reputation + optimistic_verification + zk_wasm + validator + content_handle end - subgraph Dynamic limits - Fuel - Memory - Storage + subgraph payment + staking_escrow + payment_channels + settlement + stripe + eth end - subgraph Trust - ucan-delegation - ucan-invocation - - attestation - - subgraph Optimistic-Verification - direction LR - - verify - adjudicate - stake - slash + subgraph first_party_effects + subgraph cryptographic_effects + randomness_oracle_fx + encryption_fx + signature_fx end - wasm-snarks - end - - subgraph DX - subgraph DSLs - Rust - JS - Python - others-eg-apache-beam + subgraph networking_effects + https_fx + dns_fx + email_fx end - subgraph Packaging - static-binary - Docker - Nix - Brew - Crate-SDK + subgraph store_effects + block_object_reader_fx + block_object_writer_fx + block_object_reader_fx + persistence_fx end - - docs end - content-handles --> wasm-load - wasm-snarks -.->|after| Optimistic-Verification - attestation --> ucan-invocation --> ucan-delegation - block-streaming --> wasm-load - matchmaking-coordination --> pubsub - - matchmaking-coordination --> announce-capabilities - matchmaking-coordination --> request-tasks - - delegated-payment-channels-ucan-or-interlegder --> settlement-rails - - Crypto --> Randomness_Oracle - HTTPS --> DNS - - dashboard --> odd-jobs --> account-system - dashboard --> task-registry --> account-system - - odd-jobs --> deterministic-Wasm-runtime - odd-jobs --> wasm-load - odd-jobs --> scheduling - - ucan-pay --> account-system - ucan-pay --> Payment + subgraph services + user_account + swarm_federation + hosted_bootstrap + managed_homestar + self_hostable_homestar + end - Wasm_Standard_Library --> deterministic-Wasm-runtime - odd-jobs --> Nix --> static-binary - Docker --> static-binary - Brew --> static-binary - static-binary --> Crate-SDK + subgraph sdk + javascript_sdk + python_sdk + rust_sdk + end - adjudicate --> verify - adjudicate --> slash - slash --> stake - stake --> settlement-rails + subgraph ui + dashboard + cli + end - slash --> attestation - ``` + pubsub --> car + pubsub --> quic + pubsub --> webt + pubsub --> https + + pub_obj_store --> layered_bs + priv_obj_store --> layered_bs + + layered_bs --> pubsub + + wasm_retrieval --> ca_storage + host_session_storage ----> ca_storage + receipt_store --> ca_storage + task_registry --> wasm_retrieval + + wasm_based_plugin_system --> wasm_execution + resource_limits --> wasm_execution + wasm_execution --> task_registry + wasm_execution --> receipt_store + + coordinator --> match_maker + coordinator --> dag_injector + workflow_static_analyser --> workflow + match_maker --> affinity_probe + match_maker --> workflow + + affinity_probe ---> wasm_based_plugin_system + + host_logging_metrics_traces --> workflow_progress_tracker + workflow_progress_tracker --> coordinator + + ucan --> did + + reputation --> optimistic_verification + optimistic_verification --> coordinator + optimistic_verification --> content_handle + optimistic_verification --> validator + validator -.-> receipt_store + + dag_injector --> workflow_static_analyser + dag_injector -...-> content_handle + workflow --> ucan + + optimistic_verification --> payment_channels + + payment_channels --> staking_escrow + staking_escrow --> settlement + settlement --> stripe + settlement --> eth + + first_party_effects -.......-> wasm_based_plugin_system + + user_account + swarm_federation --> self_hostable_homestar + swarm_federation --> hosted_bootstrap + managed_homestar --> self_hostable_homestar + + dashboard -.-> javascript_sdk + cli --> rust_sdk + ui ----> user_account + + %% zk_wasm --> resource_limits + stripe --> user_account +``` + + + + + + + + + + + + + + + + + + + + + times function run? + + + + + + + + + + + + + + + + +Trust + +Networking Data From 5c6b76a75b1bd2d515c23b58d721c15aa25d0d93 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 13:03:54 -0700 Subject: [PATCH 06/24] Give full names to diagram nodes Signed-off-by: Brooklyn Zelenka --- plan/ROADMAP.md | 60 ++++++++++++++++++++++++------------------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index db5f80b3..8b186817 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -7,49 +7,49 @@ Dependency Graph ```mermaid flowchart subgraph Networking - pubsub - car - quic - webt - https + pubsub[PubSub] + car[CAR Files] + quic[QUIC] + webt[WebTransport] + https[HTTPS] end - subgraph ca_storage - layered_bs - pub_obj_store - priv_obj_store + subgraph ca_storage[Content Addressed Storage] + layered_bs[Layered Blockstore] + pub_obj_store[Public Object Store] + priv_obj_store[Secret Object Store] end - subgraph task_storage - receipt_store - task_registry - wasm_retrieval - host_session_storage + subgraph task_storage[Task Storage] + receipt_store[Receipt Retrieval] + task_registry[Task Retrieval] + wasm_retrieval[Wasm Retrieval] + host_session_storage[Host Session Storage] end - subgraph execution - wasm_based_plugin_system - resource_limits - wasm_execution + subgraph execution[Execution] + wasm_based_plugin_system[Wasm-based Effects Plugin System] + resource_limits[Resource Limits] + wasm_execution[Wasm Runtime] end - subgraph scheduling - coordinator - dag_injector - workflow_static_analyser - workflow - affinity_probe - match_maker + subgraph scheduling[Scheduling] + coordinator[Coordinator] + dag_injector[DAG Injector] + workflow_static_analyser[Workflow Analyzer] + workflow[Workflow] + affinity_probe[Affinity Probe] + match_maker[Match Maker] end - subgraph tracking - host_logging_metrics_traces - workflow_progress_tracker + subgraph tracking[Tracking] + host_logging_metrics_traces[Host: Logging, Metrics, Traces] + workflow_progress_tracker[Workflow Progress Tracker] end subgraph capabilities - ucan - did + ucan[UCAN] + did[DID] end subgraph trust From 0747dd2c61a54ca9b298ff0893f55fcde6eab4f1 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 13:07:55 -0700 Subject: [PATCH 07/24] Layout hacks Signed-off-by: Brooklyn Zelenka --- plan/ROADMAP.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 8b186817..f97aad86 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -171,6 +171,9 @@ flowchart %% zk_wasm --> resource_limits stripe --> user_account + + https_fx ~~~ block_object_reader_fx + randomness_oracle_fx ~~~ https_fx ``` From a0c300d3390dc3b025ae90a9c5a6ff76a797804a Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 13:09:14 -0700 Subject: [PATCH 08/24] Add link to dictionary --- plan/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/plan/README.md b/plan/README.md index f707fa3f..e0b94cd9 100644 --- a/plan/README.md +++ b/plan/README.md @@ -3,6 +3,7 @@ This directory contains the planning documentation for the Homestar project. Here is a quick overview: - πŸ›οΈ [Architecture] +- πŸ“–[Dictionary] - πŸ—ΊοΈ [Roadmap] @@ -10,4 +11,5 @@ Here is a quick overview: [Architecture]: ./ARCHITECTURE.md +[Dictionary]: ./DICTIONARY.md [Roadmap]: ./ROADMAP.md From da44ce2892b11f57959ff1e127e6be6f1fd15bc6 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 14:34:22 -0700 Subject: [PATCH 09/24] More cleanup Signed-off-by: Brooklyn Zelenka --- plan/ROADMAP.md | 118 ++++++++++++++++-------------------------------- 1 file changed, 39 insertions(+), 79 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index f97aad86..d69f8452 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -52,60 +52,60 @@ flowchart did[DID] end - subgraph trust - reputation - optimistic_verification - zk_wasm - validator - content_handle + subgraph trust[Trust] + reputation[Reputation] + optimistic_verification[Optimistic Verification] + zk_wasm[ZK Wasm] + validator[Validator] + adjudicator[Adjudicator] + content_handle[Content Handle] end - subgraph payment - staking_escrow - payment_channels - settlement - stripe - eth + subgraph payment[Payment] + staking_escrow[Staking/Escrow] + payment_channels[Payment Channels] + settlement[Settlement] + stripe[Stripe] + eth[ETH] end - subgraph first_party_effects - subgraph cryptographic_effects - randomness_oracle_fx - encryption_fx - signature_fx + subgraph first_party_effects[1st Party Effects] + subgraph cryptographic_effects[Cryptographic Effects] + randomness_oracle_fx[Randomness Oracle] + encryption_fx[Encryption Effects] + signature_fx[Signature Effects] end - subgraph networking_effects - https_fx - dns_fx - email_fx + subgraph networking_effects[Networking Effects] + https_fx[HTTPS Get/Put/Patch/Post] + dns_fx[DNS Effects] + email_fx[Email Effects] end - subgraph store_effects - block_object_reader_fx - block_object_writer_fx - block_object_reader_fx - persistence_fx + subgraph store_effects[Storeage Effects] + block_object_reader_fx[Object Reader Effect] + block_object_writer_fx[Object Writer Effect] + persistence_fx[Persistence Effect] end end subgraph services - user_account - swarm_federation - hosted_bootstrap - managed_homestar - self_hostable_homestar + user_account[User Account] + swarm_federation[Swarm/Federation] + hosted_bootstrap[Hosted Bootstraps] + managed_homestar[Managed Homestar] + self_hostable_homestar[Self-Hostable Homestar] end - subgraph sdk - javascript_sdk - python_sdk - rust_sdk + subgraph sdk[SDK] + javascript_sdk[JavaScript SDK] + python_sdk[Python SDK] + rust_sdk[Rust SDK] end - subgraph ui - dashboard - cli + subgraph ui[UI] + dashboard[Dashboard] + cli[CLI] end pubsub --> car @@ -155,8 +155,8 @@ flowchart payment_channels --> staking_escrow staking_escrow --> settlement - settlement --> stripe settlement --> eth + settlement --> stripe first_party_effects -.......-> wasm_based_plugin_system @@ -175,43 +175,3 @@ flowchart https_fx ~~~ block_object_reader_fx randomness_oracle_fx ~~~ https_fx ``` - - - - - - - - - - - - - - - - - - - - - times function run? - - - - - - - - - - - - - - - - -Trust - -Networking Data From 797dc730cfb4a977cef70b7f58dda96cfdf4f478 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 14:50:31 -0700 Subject: [PATCH 10/24] Put the DHT back in Signed-off-by: Brooklyn Zelenka --- plan/ROADMAP.md | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index d69f8452..86908918 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -6,18 +6,28 @@ Dependency Graph ```mermaid flowchart - subgraph Networking - pubsub[PubSub] + subgraph networking + direction TB + car[CAR Files] quic[QUIC] webt[WebTransport] https[HTTPS] end + + subgraph routing[Content Routing] + direction TB + + pubsub[PubSub] + receipt-dht[Input Addressed DHT] + hash-dht[Content Addressed DHT] + end subgraph ca_storage[Content Addressed Storage] layered_bs[Layered Blockstore] pub_obj_store[Public Object Store] priv_obj_store[Secret Object Store] + storage_abstraction[Storage Abstraction] end subgraph task_storage[Task Storage] @@ -108,19 +118,18 @@ flowchart cli[CLI] end - pubsub --> car - pubsub --> quic - pubsub --> webt - pubsub --> https - pub_obj_store --> layered_bs priv_obj_store --> layered_bs - layered_bs --> pubsub + layered_bs --> routing + + receipt_store --> storage_abstraction + wasm_retrieval --> storage_abstraction + host_session_storage ----> storage_abstraction + + storage_abstraction --> pub_obj_store + storage_abstraction --> priv_obj_store - wasm_retrieval --> ca_storage - host_session_storage ----> ca_storage - receipt_store --> ca_storage task_registry --> wasm_retrieval wasm_based_plugin_system --> wasm_execution @@ -172,6 +181,8 @@ flowchart %% zk_wasm --> resource_limits stripe --> user_account + routing --> networking + https_fx ~~~ block_object_reader_fx randomness_oracle_fx ~~~ https_fx ``` From 8d6cc6c2ae9e914d6bd6eef50fba539d369a4376 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 15:11:24 -0700 Subject: [PATCH 11/24] Remove Content Handle from dependency diagram Signed-off-by: Brooklyn Zelenka --- plan/ROADMAP.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 86908918..9d778efb 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -57,7 +57,7 @@ flowchart workflow_progress_tracker[Workflow Progress Tracker] end - subgraph capabilities + subgraph capabilities[Capabilities] ucan[UCAN] did[DID] end @@ -68,7 +68,6 @@ flowchart zk_wasm[ZK Wasm] validator[Validator] adjudicator[Adjudicator] - content_handle[Content Handle] end subgraph payment[Payment] @@ -99,7 +98,7 @@ flowchart end end - subgraph services + subgraph reliability[Reliability] user_account[User Account] swarm_federation[Swarm/Federation] hosted_bootstrap[Hosted Bootstraps] @@ -151,13 +150,15 @@ flowchart ucan --> did reputation --> optimistic_verification + reputation --> zk_wasm + optimistic_verification --> coordinator - optimistic_verification --> content_handle - optimistic_verification --> validator - validator -.-> receipt_store + optimistic_verification --> adjudicator + adjudicator --> workflow + adjudicator --> validator + validator --> receipt_store dag_injector --> workflow_static_analyser - dag_injector -...-> content_handle workflow --> ucan optimistic_verification --> payment_channels @@ -178,7 +179,6 @@ flowchart cli --> rust_sdk ui ----> user_account - %% zk_wasm --> resource_limits stripe --> user_account routing --> networking From c8a2c3e0799fa697f5247847a8bce50ccd703ef6 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 15:56:33 -0700 Subject: [PATCH 12/24] Start on roadmap proper (mix of porting from old Figjam and currnet understanding) --- plan/ARCHITECTURE.md | 3 +++ plan/README.md | 12 +++++++++--- plan/ROADMAP.md | 19 ++++++++++++++++--- 3 files changed, 28 insertions(+), 6 deletions(-) create mode 100644 plan/ARCHITECTURE.md diff --git a/plan/ARCHITECTURE.md b/plan/ARCHITECTURE.md new file mode 100644 index 00000000..f7d22f98 --- /dev/null +++ b/plan/ARCHITECTURE.md @@ -0,0 +1,3 @@ +# Architecture + + diff --git a/plan/README.md b/plan/README.md index e0b94cd9..f92ca0e9 100644 --- a/plan/README.md +++ b/plan/README.md @@ -1,15 +1,21 @@ This directory contains the planning documentation for the Homestar project. -Here is a quick overview: +## Internal Links - πŸ›οΈ [Architecture] -- πŸ“–[Dictionary] - πŸ—ΊοΈ [Roadmap] +## Dictionary + +As with most projects, Homestar uses terms in a particular sense. Here's a dictionary to help align what words mean. + +| Term | Defintion | +|------------------|-----------| +| | | + [Architecture]: ./ARCHITECTURE.md -[Dictionary]: ./DICTIONARY.md [Roadmap]: ./ROADMAP.md diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 9d778efb..8b25604a 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -1,8 +1,21 @@ -# Homestar Roadmap +# Roadmap -I can't even begin to describe just how WIP this is at the moment: +## Phase 0: MVP -Dependency Graph +- Networking with libp2p +- Wasm DHT (content addressing) +- Receipt DHT (input addressing) +- Deterministic Wasm Execution +- Matchmaking +- Workflow Coordinator + +## Phase 1 + +## Phase 2 + +## Phase 3 + +## Dependency Graph ```mermaid flowchart From c7930da2053a1285b359056d47695c47711859d5 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 16:40:37 -0700 Subject: [PATCH 13/24] Start porting over roadmap --- plan/README.md | 2 +- plan/ROADMAP.md | 55 +++++++++++++++++++++++++++++++++++-------------- 2 files changed, 41 insertions(+), 16 deletions(-) diff --git a/plan/README.md b/plan/README.md index f92ca0e9..0e332fae 100644 --- a/plan/README.md +++ b/plan/README.md @@ -5,7 +5,7 @@ This directory contains the planning documentation for the Homestar project. - πŸ›οΈ [Architecture] - πŸ—ΊοΈ [Roadmap] -## Dictionary +#f# Dictionary As with most projects, Homestar uses terms in a particular sense. Here's a dictionary to help align what words mean. diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 8b25604a..64b5f44f 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -1,20 +1,5 @@ # Roadmap -## Phase 0: MVP - -- Networking with libp2p -- Wasm DHT (content addressing) -- Receipt DHT (input addressing) -- Deterministic Wasm Execution -- Matchmaking -- Workflow Coordinator - -## Phase 1 - -## Phase 2 - -## Phase 3 - ## Dependency Graph ```mermaid @@ -199,3 +184,43 @@ flowchart https_fx ~~~ block_object_reader_fx randomness_oracle_fx ~~~ https_fx ``` + +## Phase 0: MVP + +The MVP focuses on getting the basics running. + +- Networking + - [x] libp2p pubsub + - [x] Wasm Blob DHT (content addressing) + - [x] Receipt DHT (input addressing) +- Execution + - [x] Deterministic Wasm runtime + - [x] Receipt reuse / short-circuit + - [x] Crash-oblivious resumption +- Scheduler + - [ ] Peer probe + - [ ] Matchmaking + - [ ] Workflow Coordinator +- Application layer + - [ ] Example browser application +- Deployment + - [ ] Package for self-hosting + - [ ] Package for deployment to managed services + +## Phase 1: Integration + +- Developer Tooling + - [ ] Rust Task template + - [ ] JS task template +- Add Iroh networking support (esp. NAT hole punching) +- Add IPVM support to Kubo (`kubo run`) +- Resource limits +- (Co)effect Plugins +- Documentation + +## Phase 2: Marketplace + +- Paymnet + +## Phase 3 + From 0ad6bc0c7d334a04e82d25c948036efcf6ecf5a4 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 22:00:57 -0700 Subject: [PATCH 14/24] Breaking out stages --- plan/ROADMAP.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 64b5f44f..a5754e0c 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -212,8 +212,12 @@ The MVP focuses on getting the basics running. - Developer Tooling - [ ] Rust Task template - [ ] JS task template -- Add Iroh networking support (esp. NAT hole punching) -- Add IPVM support to Kubo (`kubo run`) +- Networking + - [ ] Iroh NAT traversal +- Data + - [ ] Content handle injection + - [ ] Wasm block + - [ ] Add IPVM support to Kubo (`kubo run`) - Resource limits - (Co)effect Plugins - Documentation From c9bcc20152846630513ab6e6dfa3669c5dded1f1 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 22:14:45 -0700 Subject: [PATCH 15/24] Add task registry --- plan/ROADMAP.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index a5754e0c..20c3225b 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -96,8 +96,9 @@ flowchart end end - subgraph reliability[Reliability] + subgraph reliability[UX] user_account[User Account] + task_registry[Task Registry] swarm_federation[Swarm/Federation] hosted_bootstrap[Hosted Bootstraps] managed_homestar[Managed Homestar] @@ -178,6 +179,7 @@ flowchart ui ----> user_account stripe --> user_account + task_registry --> user_account routing --> networking From a6987d2e25cf1279809533c275d1b64aed792fe9 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 22:24:44 -0700 Subject: [PATCH 16/24] Move diagram to readme --- plan/README.md | 191 +++++++++++++++++++++++++++++++++++++++++++++++- plan/ROADMAP.md | 184 ---------------------------------------------- 2 files changed, 190 insertions(+), 185 deletions(-) diff --git a/plan/README.md b/plan/README.md index 0e332fae..3686123f 100644 --- a/plan/README.md +++ b/plan/README.md @@ -5,7 +5,7 @@ This directory contains the planning documentation for the Homestar project. - πŸ›οΈ [Architecture] - πŸ—ΊοΈ [Roadmap] -#f# Dictionary +## Dictionary As with most projects, Homestar uses terms in a particular sense. Here's a dictionary to help align what words mean. @@ -13,6 +13,195 @@ As with most projects, Homestar uses terms in a particular sense. Here's a dicti |------------------|-----------| | | | +## High Level Dependencies + +```mermaid +flowchart + subgraph networking + direction TB + + car[CAR Files] + quic[QUIC] + webt[WebTransport] + https[HTTPS] + end + + subgraph routing[Content Routing] + direction TB + + pubsub[PubSub] + receipt-dht[Input Addressed DHT] + hash-dht[Content Addressed DHT] + end + + subgraph ca_storage[Content Addressed Storage] + layered_bs[Layered Blockstore] + pub_obj_store[Public Object Store] + priv_obj_store[Secret Object Store] + storage_abstraction[Storage Abstraction] + end + + subgraph task_storage[Task Storage] + receipt_store[Receipt Retrieval] + task_registry[Task Retrieval] + wasm_retrieval[Wasm Retrieval] + host_session_storage[Host Session Storage] + end + + subgraph execution[Execution] + wasm_based_plugin_system[Wasm-based Effects Plugin System] + resource_limits[Resource Limits] + wasm_execution[Wasm Runtime] + wit_ipld[WIT/IPLD] + end + + subgraph scheduling[Scheduling] + coordinator[Coordinator] + dag_injector[DAG Injector] + workflow_static_analyser[Workflow Analyzer] + workflow[Workflow] + affinity_probe[Affinity Probe] + match_maker[Match Maker] + end + + subgraph tracking[Tracking] + host_logging_metrics_traces[Host: Logging, Metrics, Traces] + workflow_progress_tracker[Workflow Progress Tracker] + end + + subgraph capabilities[Capabilities] + ucan[UCAN] + did[DID] + end + + subgraph trust[Trust] + reputation[Reputation] + optimistic_verification[Optimistic Verification] + zk_wasm[ZK Wasm] + validator[Validator] + adjudicator[Adjudicator] + end + + subgraph payment[Payment] + staking_escrow[Staking/Escrow] + payment_channels[Payment Channels] + settlement[Settlement] + stripe[Stripe] + eth[ETH] + end + + subgraph first_party_effects[1st Party Effects] + subgraph cryptographic_effects[Cryptographic Effects] + randomness_oracle_fx[Randomness Oracle] + encryption_fx[Encryption Effects] + signature_fx[Signature Effects] + end + + subgraph networking_effects[Networking Effects] + https_fx[HTTPS Get/Put/Patch/Post] + dns_fx[DNS Effects] + email_fx[Email Effects] + end + + subgraph store_effects[Storeage Effects] + block_object_reader_fx[Object Reader Effect] + block_object_writer_fx[Object Writer Effect] + persistence_fx[Persistence Effect] + end + end + + subgraph reliability[UX] + user_account[User Account] + task_registry[Task Registry] + swarm_federation[Swarm/Federation] + hosted_bootstrap[Hosted Bootstraps] + managed_homestar[Managed Homestar] + self_hostable_homestar[Self-Hostable Homestar] + end + + subgraph sdk[SDK] + javascript_sdk[JavaScript SDK] + python_sdk[Python SDK] + rust_sdk[Rust SDK] + end + + subgraph ui[UI] + dashboard[Dashboard] + cli[CLI] + end + + pub_obj_store --> layered_bs + priv_obj_store --> layered_bs + + layered_bs --> routing + + receipt_store --> storage_abstraction + wasm_retrieval --> storage_abstraction + host_session_storage ----> storage_abstraction + + storage_abstraction --> pub_obj_store + storage_abstraction --> priv_obj_store + + task_registry --> wasm_retrieval + + wasm_based_plugin_system --> wasm_execution + resource_limits --> wasm_execution + wasm_execution --> task_registry + wasm_execution --> receipt_store + + coordinator --> match_maker + coordinator --> dag_injector + workflow_static_analyser --> workflow + match_maker --> affinity_probe + match_maker --> workflow + + affinity_probe ---> wasm_based_plugin_system + + host_logging_metrics_traces --> workflow_progress_tracker + workflow_progress_tracker --> coordinator + + ucan --> did + + reputation --> optimistic_verification + reputation --> zk_wasm + + optimistic_verification --> coordinator + optimistic_verification --> adjudicator + adjudicator --> workflow + adjudicator --> validator + validator --> receipt_store + + dag_injector --> workflow_static_analyser + workflow --> ucan + + optimistic_verification --> payment_channels + + payment_channels --> staking_escrow + staking_escrow --> settlement + settlement --> eth + settlement --> stripe + + first_party_effects -.......-> wasm_based_plugin_system + + user_account + swarm_federation --> self_hostable_homestar + swarm_federation --> hosted_bootstrap + managed_homestar --> self_hostable_homestar + + dashboard -.-> javascript_sdk + cli --> rust_sdk + ui ----> user_account + + stripe --> user_account + task_registry --> user_account + + routing --> networking + workflow --> wit_ipld + + https_fx ~~~ block_object_reader_fx + randomness_oracle_fx ~~~ https_fx +``` + diff --git a/plan/ROADMAP.md b/plan/ROADMAP.md index 20c3225b..a40f70a2 100644 --- a/plan/ROADMAP.md +++ b/plan/ROADMAP.md @@ -2,190 +2,6 @@ ## Dependency Graph -```mermaid -flowchart - subgraph networking - direction TB - - car[CAR Files] - quic[QUIC] - webt[WebTransport] - https[HTTPS] - end - - subgraph routing[Content Routing] - direction TB - - pubsub[PubSub] - receipt-dht[Input Addressed DHT] - hash-dht[Content Addressed DHT] - end - - subgraph ca_storage[Content Addressed Storage] - layered_bs[Layered Blockstore] - pub_obj_store[Public Object Store] - priv_obj_store[Secret Object Store] - storage_abstraction[Storage Abstraction] - end - - subgraph task_storage[Task Storage] - receipt_store[Receipt Retrieval] - task_registry[Task Retrieval] - wasm_retrieval[Wasm Retrieval] - host_session_storage[Host Session Storage] - end - - subgraph execution[Execution] - wasm_based_plugin_system[Wasm-based Effects Plugin System] - resource_limits[Resource Limits] - wasm_execution[Wasm Runtime] - end - - subgraph scheduling[Scheduling] - coordinator[Coordinator] - dag_injector[DAG Injector] - workflow_static_analyser[Workflow Analyzer] - workflow[Workflow] - affinity_probe[Affinity Probe] - match_maker[Match Maker] - end - - subgraph tracking[Tracking] - host_logging_metrics_traces[Host: Logging, Metrics, Traces] - workflow_progress_tracker[Workflow Progress Tracker] - end - - subgraph capabilities[Capabilities] - ucan[UCAN] - did[DID] - end - - subgraph trust[Trust] - reputation[Reputation] - optimistic_verification[Optimistic Verification] - zk_wasm[ZK Wasm] - validator[Validator] - adjudicator[Adjudicator] - end - - subgraph payment[Payment] - staking_escrow[Staking/Escrow] - payment_channels[Payment Channels] - settlement[Settlement] - stripe[Stripe] - eth[ETH] - end - - subgraph first_party_effects[1st Party Effects] - subgraph cryptographic_effects[Cryptographic Effects] - randomness_oracle_fx[Randomness Oracle] - encryption_fx[Encryption Effects] - signature_fx[Signature Effects] - end - - subgraph networking_effects[Networking Effects] - https_fx[HTTPS Get/Put/Patch/Post] - dns_fx[DNS Effects] - email_fx[Email Effects] - end - - subgraph store_effects[Storeage Effects] - block_object_reader_fx[Object Reader Effect] - block_object_writer_fx[Object Writer Effect] - persistence_fx[Persistence Effect] - end - end - - subgraph reliability[UX] - user_account[User Account] - task_registry[Task Registry] - swarm_federation[Swarm/Federation] - hosted_bootstrap[Hosted Bootstraps] - managed_homestar[Managed Homestar] - self_hostable_homestar[Self-Hostable Homestar] - end - - subgraph sdk[SDK] - javascript_sdk[JavaScript SDK] - python_sdk[Python SDK] - rust_sdk[Rust SDK] - end - - subgraph ui[UI] - dashboard[Dashboard] - cli[CLI] - end - - pub_obj_store --> layered_bs - priv_obj_store --> layered_bs - - layered_bs --> routing - - receipt_store --> storage_abstraction - wasm_retrieval --> storage_abstraction - host_session_storage ----> storage_abstraction - - storage_abstraction --> pub_obj_store - storage_abstraction --> priv_obj_store - - task_registry --> wasm_retrieval - - wasm_based_plugin_system --> wasm_execution - resource_limits --> wasm_execution - wasm_execution --> task_registry - wasm_execution --> receipt_store - - coordinator --> match_maker - coordinator --> dag_injector - workflow_static_analyser --> workflow - match_maker --> affinity_probe - match_maker --> workflow - - affinity_probe ---> wasm_based_plugin_system - - host_logging_metrics_traces --> workflow_progress_tracker - workflow_progress_tracker --> coordinator - - ucan --> did - - reputation --> optimistic_verification - reputation --> zk_wasm - - optimistic_verification --> coordinator - optimistic_verification --> adjudicator - adjudicator --> workflow - adjudicator --> validator - validator --> receipt_store - - dag_injector --> workflow_static_analyser - workflow --> ucan - - optimistic_verification --> payment_channels - - payment_channels --> staking_escrow - staking_escrow --> settlement - settlement --> eth - settlement --> stripe - - first_party_effects -.......-> wasm_based_plugin_system - - user_account - swarm_federation --> self_hostable_homestar - swarm_federation --> hosted_bootstrap - managed_homestar --> self_hostable_homestar - - dashboard -.-> javascript_sdk - cli --> rust_sdk - ui ----> user_account - - stripe --> user_account - task_registry --> user_account - - routing --> networking - - https_fx ~~~ block_object_reader_fx - randomness_oracle_fx ~~~ https_fx -``` ## Phase 0: MVP From bf0fc675a23b6af58691f2386962ab5ab825da0f Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 22:26:33 -0700 Subject: [PATCH 17/24] Fix duplicate entry --- plan/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plan/README.md b/plan/README.md index 3686123f..9aafb599 100644 --- a/plan/README.md +++ b/plan/README.md @@ -43,7 +43,7 @@ flowchart subgraph task_storage[Task Storage] receipt_store[Receipt Retrieval] - task_registry[Task Retrieval] + task_store[Task Retrieval] wasm_retrieval[Wasm Retrieval] host_session_storage[Host Session Storage] end @@ -146,7 +146,7 @@ flowchart wasm_based_plugin_system --> wasm_execution resource_limits --> wasm_execution - wasm_execution --> task_registry + wasm_execution --> task_store wasm_execution --> receipt_store coordinator --> match_maker From 114c21c255b7a43aae21e9038b043734c9f11432 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 22:35:23 -0700 Subject: [PATCH 18/24] Update README.md Signed-off-by: Brooklyn Zelenka --- plan/README.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/plan/README.md b/plan/README.md index 9aafb599..41b95ef7 100644 --- a/plan/README.md +++ b/plan/README.md @@ -17,7 +17,7 @@ As with most projects, Homestar uses terms in a particular sense. Here's a dicti ```mermaid flowchart - subgraph networking + subgraph networking[Networking] direction TB car[CAR Files] @@ -112,7 +112,6 @@ flowchart subgraph reliability[UX] user_account[User Account] - task_registry[Task Registry] swarm_federation[Swarm/Federation] hosted_bootstrap[Hosted Bootstraps] managed_homestar[Managed Homestar] @@ -127,6 +126,7 @@ flowchart subgraph ui[UI] dashboard[Dashboard] + task_registry[Task Registry] cli[CLI] end @@ -142,8 +142,6 @@ flowchart storage_abstraction --> pub_obj_store storage_abstraction --> priv_obj_store - task_registry --> wasm_retrieval - wasm_based_plugin_system --> wasm_execution resource_limits --> wasm_execution wasm_execution --> task_store @@ -172,6 +170,8 @@ flowchart validator --> receipt_store dag_injector --> workflow_static_analyser + + workflow --> wit_ipld workflow --> ucan optimistic_verification --> payment_channels @@ -193,10 +193,8 @@ flowchart ui ----> user_account stripe --> user_account - task_registry --> user_account routing --> networking - workflow --> wit_ipld https_fx ~~~ block_object_reader_fx randomness_oracle_fx ~~~ https_fx From d7d42a85858634a4f97b7e697aff5a98bcc49498 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 22:56:35 -0700 Subject: [PATCH 19/24] mv plan design --- {plan => design}/ARCHITECTURE.md | 0 {plan => design}/README.md | 0 {plan => design}/ROADMAP.md | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename {plan => design}/ARCHITECTURE.md (100%) rename {plan => design}/README.md (100%) rename {plan => design}/ROADMAP.md (100%) diff --git a/plan/ARCHITECTURE.md b/design/ARCHITECTURE.md similarity index 100% rename from plan/ARCHITECTURE.md rename to design/ARCHITECTURE.md diff --git a/plan/README.md b/design/README.md similarity index 100% rename from plan/README.md rename to design/README.md diff --git a/plan/ROADMAP.md b/design/ROADMAP.md similarity index 100% rename from plan/ROADMAP.md rename to design/ROADMAP.md From e51a82ab685af083ab6ed690324de9ea71c17dff Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 23:04:09 -0700 Subject: [PATCH 20/24] Move roadmapping to star map --- design/README.md | 196 -------------------------------------- design/ROADMAP.md | 236 +++++++++++++++++++++++++++++++++++++--------- 2 files changed, 191 insertions(+), 241 deletions(-) diff --git a/design/README.md b/design/README.md index 41b95ef7..c03fbeb1 100644 --- a/design/README.md +++ b/design/README.md @@ -1,204 +1,8 @@ This directory contains the planning documentation for the Homestar project. -## Internal Links - - πŸ›οΈ [Architecture] - πŸ—ΊοΈ [Roadmap] -## Dictionary - -As with most projects, Homestar uses terms in a particular sense. Here's a dictionary to help align what words mean. - -| Term | Defintion | -|------------------|-----------| -| | | - -## High Level Dependencies - -```mermaid -flowchart - subgraph networking[Networking] - direction TB - - car[CAR Files] - quic[QUIC] - webt[WebTransport] - https[HTTPS] - end - - subgraph routing[Content Routing] - direction TB - - pubsub[PubSub] - receipt-dht[Input Addressed DHT] - hash-dht[Content Addressed DHT] - end - - subgraph ca_storage[Content Addressed Storage] - layered_bs[Layered Blockstore] - pub_obj_store[Public Object Store] - priv_obj_store[Secret Object Store] - storage_abstraction[Storage Abstraction] - end - - subgraph task_storage[Task Storage] - receipt_store[Receipt Retrieval] - task_store[Task Retrieval] - wasm_retrieval[Wasm Retrieval] - host_session_storage[Host Session Storage] - end - - subgraph execution[Execution] - wasm_based_plugin_system[Wasm-based Effects Plugin System] - resource_limits[Resource Limits] - wasm_execution[Wasm Runtime] - wit_ipld[WIT/IPLD] - end - - subgraph scheduling[Scheduling] - coordinator[Coordinator] - dag_injector[DAG Injector] - workflow_static_analyser[Workflow Analyzer] - workflow[Workflow] - affinity_probe[Affinity Probe] - match_maker[Match Maker] - end - - subgraph tracking[Tracking] - host_logging_metrics_traces[Host: Logging, Metrics, Traces] - workflow_progress_tracker[Workflow Progress Tracker] - end - - subgraph capabilities[Capabilities] - ucan[UCAN] - did[DID] - end - - subgraph trust[Trust] - reputation[Reputation] - optimistic_verification[Optimistic Verification] - zk_wasm[ZK Wasm] - validator[Validator] - adjudicator[Adjudicator] - end - - subgraph payment[Payment] - staking_escrow[Staking/Escrow] - payment_channels[Payment Channels] - settlement[Settlement] - stripe[Stripe] - eth[ETH] - end - - subgraph first_party_effects[1st Party Effects] - subgraph cryptographic_effects[Cryptographic Effects] - randomness_oracle_fx[Randomness Oracle] - encryption_fx[Encryption Effects] - signature_fx[Signature Effects] - end - - subgraph networking_effects[Networking Effects] - https_fx[HTTPS Get/Put/Patch/Post] - dns_fx[DNS Effects] - email_fx[Email Effects] - end - - subgraph store_effects[Storeage Effects] - block_object_reader_fx[Object Reader Effect] - block_object_writer_fx[Object Writer Effect] - persistence_fx[Persistence Effect] - end - end - - subgraph reliability[UX] - user_account[User Account] - swarm_federation[Swarm/Federation] - hosted_bootstrap[Hosted Bootstraps] - managed_homestar[Managed Homestar] - self_hostable_homestar[Self-Hostable Homestar] - end - - subgraph sdk[SDK] - javascript_sdk[JavaScript SDK] - python_sdk[Python SDK] - rust_sdk[Rust SDK] - end - - subgraph ui[UI] - dashboard[Dashboard] - task_registry[Task Registry] - cli[CLI] - end - - pub_obj_store --> layered_bs - priv_obj_store --> layered_bs - - layered_bs --> routing - - receipt_store --> storage_abstraction - wasm_retrieval --> storage_abstraction - host_session_storage ----> storage_abstraction - - storage_abstraction --> pub_obj_store - storage_abstraction --> priv_obj_store - - wasm_based_plugin_system --> wasm_execution - resource_limits --> wasm_execution - wasm_execution --> task_store - wasm_execution --> receipt_store - - coordinator --> match_maker - coordinator --> dag_injector - workflow_static_analyser --> workflow - match_maker --> affinity_probe - match_maker --> workflow - - affinity_probe ---> wasm_based_plugin_system - - host_logging_metrics_traces --> workflow_progress_tracker - workflow_progress_tracker --> coordinator - - ucan --> did - - reputation --> optimistic_verification - reputation --> zk_wasm - - optimistic_verification --> coordinator - optimistic_verification --> adjudicator - adjudicator --> workflow - adjudicator --> validator - validator --> receipt_store - - dag_injector --> workflow_static_analyser - - workflow --> wit_ipld - workflow --> ucan - - optimistic_verification --> payment_channels - - payment_channels --> staking_escrow - staking_escrow --> settlement - settlement --> eth - settlement --> stripe - - first_party_effects -.......-> wasm_based_plugin_system - - user_account - swarm_federation --> self_hostable_homestar - swarm_federation --> hosted_bootstrap - managed_homestar --> self_hostable_homestar - - dashboard -.-> javascript_sdk - cli --> rust_sdk - ui ----> user_account - - stripe --> user_account - - routing --> networking - - https_fx ~~~ block_object_reader_fx - randomness_oracle_fx ~~~ https_fx -``` diff --git a/design/ROADMAP.md b/design/ROADMAP.md index a40f70a2..1c7f35d1 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -1,48 +1,194 @@ # Roadmap -## Dependency Graph - - -## Phase 0: MVP - -The MVP focuses on getting the basics running. - -- Networking - - [x] libp2p pubsub - - [x] Wasm Blob DHT (content addressing) - - [x] Receipt DHT (input addressing) -- Execution - - [x] Deterministic Wasm runtime - - [x] Receipt reuse / short-circuit - - [x] Crash-oblivious resumption -- Scheduler - - [ ] Peer probe - - [ ] Matchmaking - - [ ] Workflow Coordinator -- Application layer - - [ ] Example browser application -- Deployment - - [ ] Package for self-hosting - - [ ] Package for deployment to managed services - -## Phase 1: Integration - -- Developer Tooling - - [ ] Rust Task template - - [ ] JS task template -- Networking - - [ ] Iroh NAT traversal -- Data - - [ ] Content handle injection - - [ ] Wasm block - - [ ] Add IPVM support to Kubo (`kubo run`) -- Resource limits -- (Co)effect Plugins -- Documentation - -## Phase 2: Marketplace - -- Paymnet - -## Phase 3 +Refer to the [Homestar Star Map] for the latest roadmap. +## High Level Dependencies + +```mermaid +flowchart + subgraph networking[Networking] + direction TB + + car[CAR Files] + quic[QUIC] + webt[WebTransport] + https[HTTPS] + end + + subgraph routing[Content Routing] + direction TB + + pubsub[PubSub] + receipt-dht[Input Addressed DHT] + hash-dht[Content Addressed DHT] + end + + subgraph ca_storage[Content Addressed Storage] + layered_bs[Layered Blockstore] + pub_obj_store[Public Object Store] + priv_obj_store[Secret Object Store] + storage_abstraction[Storage Abstraction] + end + + subgraph task_storage[Task Storage] + receipt_store[Receipt Retrieval] + task_store[Task Retrieval] + wasm_retrieval[Wasm Retrieval] + host_session_storage[Host Session Storage] + end + + subgraph execution[Execution] + wasm_based_plugin_system[Wasm-based Effects Plugin System] + resource_limits[Resource Limits] + wasm_execution[Wasm Runtime] + wit_ipld[WIT/IPLD] + end + + subgraph scheduling[Scheduling] + coordinator[Coordinator] + dag_injector[DAG Injector] + workflow_static_analyser[Workflow Analyzer] + workflow[Workflow] + affinity_probe[Affinity Probe] + match_maker[Match Maker] + end + + subgraph tracking[Tracking] + host_logging_metrics_traces[Host: Logging, Metrics, Traces] + workflow_progress_tracker[Workflow Progress Tracker] + end + + subgraph capabilities[Capabilities] + ucan[UCAN] + did[DID] + end + + subgraph trust[Trust] + reputation[Reputation] + optimistic_verification[Optimistic Verification] + zk_wasm[ZK Wasm] + validator[Validator] + adjudicator[Adjudicator] + end + + subgraph payment[Payment] + staking_escrow[Staking/Escrow] + payment_channels[Payment Channels] + settlement[Settlement] + stripe[Stripe] + eth[ETH] + end + + subgraph first_party_effects[1st Party Effects] + subgraph cryptographic_effects[Cryptographic Effects] + randomness_oracle_fx[Randomness Oracle] + encryption_fx[Encryption Effects] + signature_fx[Signature Effects] + end + + subgraph networking_effects[Networking Effects] + https_fx[HTTPS Get/Put/Patch/Post] + dns_fx[DNS Effects] + email_fx[Email Effects] + end + + subgraph store_effects[Storeage Effects] + block_object_reader_fx[Object Reader Effect] + block_object_writer_fx[Object Writer Effect] + persistence_fx[Persistence Effect] + end + end + + subgraph reliability[UX] + user_account[User Account] + swarm_federation[Swarm/Federation] + hosted_bootstrap[Hosted Bootstraps] + managed_homestar[Managed Homestar] + self_hostable_homestar[Self-Hostable Homestar] + end + + subgraph sdk[SDK] + javascript_sdk[JavaScript SDK] + python_sdk[Python SDK] + rust_sdk[Rust SDK] + end + + subgraph ui[UI] + dashboard[Dashboard] + task_registry[Task Registry] + cli[CLI] + end + + pub_obj_store --> layered_bs + priv_obj_store --> layered_bs + + layered_bs --> routing + + receipt_store --> storage_abstraction + wasm_retrieval --> storage_abstraction + host_session_storage ----> storage_abstraction + + storage_abstraction --> pub_obj_store + storage_abstraction --> priv_obj_store + + wasm_based_plugin_system --> wasm_execution + resource_limits --> wasm_execution + wasm_execution --> task_store + wasm_execution --> receipt_store + + coordinator --> match_maker + coordinator --> dag_injector + workflow_static_analyser --> workflow + match_maker --> affinity_probe + match_maker --> workflow + + affinity_probe ---> wasm_based_plugin_system + + host_logging_metrics_traces --> workflow_progress_tracker + workflow_progress_tracker --> coordinator + + ucan --> did + + reputation --> optimistic_verification + reputation --> zk_wasm + + optimistic_verification --> coordinator + optimistic_verification --> adjudicator + adjudicator --> workflow + adjudicator --> validator + validator --> receipt_store + + dag_injector --> workflow_static_analyser + + workflow --> wit_ipld + workflow --> ucan + + optimistic_verification --> payment_channels + + payment_channels --> staking_escrow + staking_escrow --> settlement + settlement --> eth + settlement --> stripe + + first_party_effects -.......-> wasm_based_plugin_system + + user_account + swarm_federation --> self_hostable_homestar + swarm_federation --> hosted_bootstrap + managed_homestar --> self_hostable_homestar + + dashboard -.-> javascript_sdk + cli --> rust_sdk + ui ----> user_account + + stripe --> user_account + + routing --> networking + + https_fx ~~~ block_object_reader_fx + randomness_oracle_fx ~~~ https_fx +``` + + + +[Homestar Star Map]: FIXME From 6ac07cbd40f42c1f7b2f9d5ba25c0a0bb5947e63 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 23:27:36 -0700 Subject: [PATCH 21/24] Link to Starmap Signed-off-by: Brooklyn Zelenka --- design/ROADMAP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 1c7f35d1..d0c5fde5 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -1,6 +1,6 @@ # Roadmap -Refer to the [Homestar Star Map] for the latest roadmap. +Refer to the [Homestar Starmap] for the latest detailed roadmap. ## High Level Dependencies @@ -191,4 +191,4 @@ flowchart -[Homestar Star Map]: FIXME +[Homestar Star Map]: https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321)https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321 From 5644af8641583265a6097ad0e836cce13161160c Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 23:28:07 -0700 Subject: [PATCH 22/24] Whoops Signed-off-by: Brooklyn Zelenka --- design/ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design/ROADMAP.md b/design/ROADMAP.md index d0c5fde5..330ec30e 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -191,4 +191,4 @@ flowchart -[Homestar Star Map]: https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321)https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321 +[Homestar Star Map]: https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321 From 675146eb6685ac97f75868658bc72d4a89df9553 Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 23:28:29 -0700 Subject: [PATCH 23/24] typo Signed-off-by: Brooklyn Zelenka --- design/ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 330ec30e..3e0f6453 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -191,4 +191,4 @@ flowchart -[Homestar Star Map]: https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321 +[Homestar Starmap]: https://starmap.site/roadmap/github.com/ipvm-wg/Homestar/issues/321 From 54f64476fc79b3333e965cc6691df8de3f2a7caa Mon Sep 17 00:00:00 2001 From: Brooklyn Zelenka Date: Mon, 18 Sep 2023 23:31:50 -0700 Subject: [PATCH 24/24] Remove redundant node Signed-off-by: Brooklyn Zelenka --- design/ROADMAP.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 3e0f6453..404ec0cb 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -32,7 +32,6 @@ flowchart subgraph task_storage[Task Storage] receipt_store[Receipt Retrieval] - task_store[Task Retrieval] wasm_retrieval[Wasm Retrieval] host_session_storage[Host Session Storage] end @@ -133,7 +132,7 @@ flowchart wasm_based_plugin_system --> wasm_execution resource_limits --> wasm_execution - wasm_execution --> task_store + wasm_execution --> wasm_retrieval wasm_execution --> receipt_store coordinator --> match_maker