diff --git a/examples/compose/attachments/README.md b/examples/compose/attachments/README.md index a5482a2..d62efaf 100644 --- a/examples/compose/attachments/README.md +++ b/examples/compose/attachments/README.md @@ -1,5 +1,85 @@ # Attachment distribution quickstart (PRD-006) +## Two-minute quick start + +`node-a` and `node-b` are **two separate compose projects** simulating two +devices. Each `docker compose up` would otherwise create its own isolated +network, so the two containers could never reach each other. They instead join +a **shared external network** (`peat-mesh`) and dial each other by container +name — which is why that network must be created **first**, before either node +boots. (See ["Why a shared external network"](#why-a-shared-external-network) +below.) + +```bash +# Step 1 — create the shared network both nodes attach to (once) +docker network create peat-mesh +``` + +Then start each node from its own directory. Open two terminals: + +```bash +# Terminal 1 — sender (node A) +cd node-a && mkdir -p outbox +docker compose up -d + +# Terminal 2 — receiver (node B) +cd node-b && mkdir -p inbox +docker compose up -d +``` + +Then drop a file — it auto-delivers: + +```bash +cp myfile.txt node-a/outbox/ +ls node-b/inbox/ # appears here within seconds +``` + +Teardown (remove the network last, after both nodes are down): +```bash +(cd node-a && docker compose down -v) +(cd node-b && docker compose down -v) +docker network rm peat-mesh +``` + +No `peer.sh`. No `send.sh`. Peering is pre-configured via `PEAT_NODE_PEERS` +(deterministic endpoint IDs derived from the shared key + node IDs). The outbox +watcher auto-distributes any file dropped in `node-a/outbox/`. + +**Expected startup log noise:** both nodes dial each other simultaneously at +boot. You may see an `ERROR peat_node: failed to connect to peer … after 3 +attempts` in the first ~15 seconds — this is a red herring. The connection +succeeds immediately after via the peer's simultaneous inbound dial. Look for +`INFO peat_node::node: connected to peer` to confirm peering. Once you see +that, file delivery works normally. + +**Alternative (both nodes in one compose project):** `docker-compose.two-node.yml` +— same zero-friction approach but both nodes share a single Docker network +automatically, so there's no separate `docker network create` step. + +### Why a shared external network + +Each `docker compose up` creates a default network named after its project +(`node-a_default`, `node-b_default`) — **isolated** bridge networks with no +route between them and per-network DNS, so `node-a` can't even resolve or reach +`node-b`. That isolation is the whole point of Compose networks; it's also why +two independent projects can't talk by default. + +A `peat-mesh` network declared `external: true` in both files is the fix: an +externally-created network that neither project owns, that both attach to. Now +both containers sit on **one subnet** and route directly to each other (no host +gateway, no published UDP ports, no NAT) — the faithful "two devices on the same +LAN" model. Because it's `external`, Compose won't create it, so it must exist +before either `up` (and you remove it manually after both are down). + +> An earlier revision bridged the two isolated networks via +> `host.docker.internal` + published UDP ports. That fails on Docker Desktop: +> its userspace proxy doesn't cleanly forward the iroh QUIC/UDP handshake +> between networks, so the dial reaches a foreign endpoint and the TLS handshake +> fails with `error 48: invalid peer certificate: UnknownIssuer`. The shared +> network removes the host hop entirely. + +--- + Two compose files live here: - **`docker-compose.yml`** — single node. Demonstrates sender-side @@ -92,10 +172,10 @@ Real cross-peer file delivery (PRD-006 v1.1, post the inbox-watcher landing). ```bash +mkdir -p outbox-a inbox-b docker compose -f docker-compose.two-node.yml up -d -./peer.sh # bidirectional ConnectPeer -ENDPOINT=http://127.0.0.1:50061 OUTBOX_DIR=outbox-a ./send.sh -ls inbox-b/ # delivered files mirror the sender's outbox layout +cp myfile.txt outbox-a/ # PEAT_NODE_ATTACHMENT_OUTBOX_WATCH auto-distributes +ls inbox-b/ # files mirror the sender's outbox layout docker compose -f docker-compose.two-node.yml down -v ``` @@ -103,31 +183,34 @@ docker compose -f docker-compose.two-node.yml down -v changes, swap the `image:` line for the commented `build:` block in both services.) +Peering is pre-configured via `PEAT_NODE_PEERS` using deterministic endpoint +IDs (derived offline from the shared key + node IDs — same mechanism as +[multi-host delivery](#multi-host-delivery-separate-machines-no-mdns)). No +`peer.sh` step, no `GetStatus` round-trip. `PEAT_NODE_ATTACHMENT_OUTBOX_WATCH` +eliminates `send.sh` — any stable file written to the outbox triggers an +automatic `AllNodes` distribution. + What the two-node setup wires: - `peat-node-a` (`127.0.0.1:50061`) — sender. `./outbox-a` bind-mounted - read-only at `/var/lib/peat/outbox`. `--attachment-root outbox=...` - set; no inbox. + read-only at `/var/lib/peat/outbox`. `PEAT_NODE_ATTACHMENT_ROOT outbox=...` + set; `PEAT_NODE_ATTACHMENT_OUTBOX_WATCH=true`; no inbox. - `peat-node-b` (`127.0.0.1:50062`) — receiver. `./inbox-b` - bind-mounted read-write at `/var/lib/peat/inbox`. `--attachment-inbox` + bind-mounted read-write at `/var/lib/peat/inbox`. `PEAT_NODE_ATTACHMENT_INBOX` set; the receive-side watcher polls the synced `file_distributions` collection every 1s and fetches anything targeting B's iroh endpoint. -`peer.sh` issues `ConnectPeer` in both directions, which is required -for `AllNodes`-scoped distributions: A's `resolve_targets` builds -`target_nodes` from `A.blob_store.known_peers()`, which is populated -only by `ConnectPeer` *into* A. Without A → B as well, A's -distribution doc carries an empty `target_nodes` and B correctly -concludes "not for me." - -When `send.sh` is pointed at A, each delivered file lands in B's inbox -**mirroring the sender's outbox layout**: `outbox-a/hello.txt` arrives at -`inbox-b/hello.txt` (and a file under `outbox-a/sub/` at `inbox-b/sub/`), -byte-identical, latest-wins on re-delivery. Apps watching the inbox can still -correlate a delivery back to the sender via `GetAttachmentDistribution` — the +Each delivered file lands in B's inbox **mirroring the sender's outbox +layout**: `outbox-a/hello.txt` arrives at `inbox-b/hello.txt`, byte-identical, +latest-wins on re-delivery. Apps watching the inbox can still correlate a +delivery back to the sender via `GetAttachmentDistribution` — the `distribution_id` travels in the synced `file_distributions` doc and the receive-side log line, not in the on-disk path. +`peer.sh` is still present for manual-peering workflows (e.g. ad-hoc +`ConnectPeer` calls, scripted testing against arbitrary nodes). It is not +needed for the two-node compose above. + > **Inbox layout changed in v0.4.8 ([#173](https://github.com/defenseunicorns/peat-node/issues/173)).** > Earlier images nested every delivery under `inbox-b/{distribution_id}/{filename}`; > v0.4.8+ mirrors the sender's outbox path instead. A sender-supplied name that diff --git a/examples/compose/attachments/docker-compose.two-node.yml b/examples/compose/attachments/docker-compose.two-node.yml index 045a7c3..cd56363 100644 --- a/examples/compose/attachments/docker-compose.two-node.yml +++ b/examples/compose/attachments/docker-compose.two-node.yml @@ -5,24 +5,35 @@ # only exercises sender-side ingest — useful for the per-size benchmark # but doesn't prove cross-peer delivery. # -# Quickstart: +# Quick start (no peer.sh, no send.sh — just drop a file): +# +# mkdir -p outbox-a inbox-b # docker compose -f docker-compose.two-node.yml up -d -# ./peer.sh # peer the two nodes (B → A and A → B) -# ./send.sh # send hello.txt + the 1/10/100 MiB suite from A -# ls ./inbox-b/ # watch files appear on B's filesystem +# cp myfile.txt outbox-a/ # auto-distributes to inbox-b +# ls inbox-b/ # appears here within seconds # docker compose -f docker-compose.two-node.yml down -v # +# Peering is pre-configured via PEAT_NODE_PEERS (deterministic endpoint IDs +# derived from the shared key + node IDs — no peer.sh required, no GetStatus +# round-trip, no mDNS). PEAT_NODE_ATTACHMENT_OUTBOX_WATCH auto-distributes +# any stable file dropped in the outbox — no SendAttachments call required. +# # Layout: # # ./outbox-a/ → A's --attachment-root "outbox" (read-only into A) # ./inbox-b/ → B's --attachment-inbox (read-write into B) # -# B writes received files at: -# ./inbox-b/{distribution_id}/{filename} +# Files arrive at inbox-b/ mirroring the sender's outbox path (v0.4.8+). # # A has no inbox configured (it's the sender). B has no outbox # configured (it's the receiver). For a real bidirectional setup # operators would mount both on both nodes. +# +# DERIVED ENDPOINT IDs (zero demo key + node IDs above): +# attach-a: 035254fa2cdf94cdd5fbf7ef7fe27efd26e1490d7db3b03b1bf30859be447634 +# attach-b: 70de0ba0781a5e9698106335b7fa8733813db4063c110a375f0c5ed798f4e6f9 +# +# Reproduced with: peat-node derive-id --shared-key "AAAA...A=" --node-id services: peat-node-a: @@ -45,11 +56,16 @@ services: PEAT_NODE_APP_ID: attach-demo PEAT_NODE_SHARED_KEY: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= PEAT_NODE_AUTO_SYNC: "true" - # Pin iroh UDP port so peer-b can reach a stable address. The - # outbound peer URL in peer.sh hardcodes :51071. + # Pin iroh UDP port so the pre-configured peer address is stable. PEAT_NODE_IROH_UDP_PORT: "51071" + # Pre-configured peer: attach-b's derived endpoint ID @ its service DNS + UDP port. + PEAT_NODE_PEERS: 70de0ba0781a5e9698106335b7fa8733813db4063c110a375f0c5ed798f4e6f9@peat-node-b:51072 + PEAT_NODE_DISABLE_MDNS: "true" PEAT_NODE_ATTACHMENT_ROOT: outbox=/var/lib/peat/outbox - RUST_LOG: peat_node=info,peat_node::attachments::inbox=warn + # Hands-off mode: any stable file dropped in ./outbox-a/ is auto-distributed + # to all known peers (no SendAttachments call). Requires v0.4.5+. + PEAT_NODE_ATTACHMENT_OUTBOX_WATCH: "true" + RUST_LOG: peat_node=info,peat_mesh=info,peat_protocol=info,iroh=warn ports: - "50061:50051" volumes: @@ -69,10 +85,13 @@ services: PEAT_NODE_SHARED_KEY: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= PEAT_NODE_AUTO_SYNC: "true" PEAT_NODE_IROH_UDP_PORT: "51072" + # Pre-configured peer: attach-a's derived endpoint ID @ its service DNS + UDP port. + PEAT_NODE_PEERS: 035254fa2cdf94cdd5fbf7ef7fe27efd26e1490d7db3b03b1bf30859be447634@peat-node-a:51071 + PEAT_NODE_DISABLE_MDNS: "true" # B has no outbox (it's the receiver). Inbox enables the receive- - # side watcher. + # side watcher: polls synced file_distributions and auto-fetches blobs. PEAT_NODE_ATTACHMENT_INBOX: /var/lib/peat/inbox - RUST_LOG: peat_node=info,peat_node::attachments::inbox=info + RUST_LOG: peat_node=info,peat_mesh=info,peat_protocol=info,iroh=warn ports: - "50062:50051" volumes: diff --git a/examples/compose/attachments/node-a/docker-compose.yml b/examples/compose/attachments/node-a/docker-compose.yml new file mode 100644 index 0000000..3eee166 --- /dev/null +++ b/examples/compose/attachments/node-a/docker-compose.yml @@ -0,0 +1,70 @@ +# Node A — SENDER (outbox watcher). Runs alongside ../node-b. +# +# Both nodes are separate compose projects that join a SHARED external +# Docker network ("peat-mesh") and dial each other by container name. +# This replaces the earlier host.docker.internal approach: Docker +# Desktop (macOS/Windows) does not route the iroh QUIC/UDP handshake +# between two isolated bridge networks via published host ports — the +# dial reaches a foreign QUIC endpoint and fails the TLS handshake +# ("error 48: invalid peer certificate: UnknownIssuer"). A shared +# user-defined network gives both containers a real, mutually routable +# path and built-in DNS resolution by container_name. +# +# Quick start (run from this directory): +# +# docker network create peat-mesh # once; shared by both nodes +# mkdir -p outbox +# docker compose up -d +# # (also start ../node-b) +# cp myfile.txt outbox/ # auto-distributes to node-b's inbox +# +# Teardown: +# docker compose down -v +# docker network rm peat-mesh # after both nodes are down +# +# SHARED KEY (demo only — unique per-formation, base64): +# fPARVR0eqxhTNqqjKIrYrYTD1VBP6PjGKCAAa39+wl8= +# +# Generate your own with: `head -c 32 /dev/urandom | base64`. The endpoint +# IDs below are derived deterministically from (shared_key + node_id). +# +# DERIVED ENDPOINT IDs (offline, from shared key + node IDs): +# attach-a: fdf9a98cb65237d2f2d70e1af68e7d2f543e14a287775a4e1bf12a35b84bccb6 +# attach-b: a53fca1d6263555d269d1594b2e5b05e88a0bab126b3885b9ee74259bd6ecd2f +# Reproduced with: peat-node derive-id --shared-key "" --node-id + +services: + peat-node-a: + image: ghcr.io/defenseunicorns/peat-node:v0.4.8 + # build: + # context: ../../../.. + container_name: peat-attach-a + networks: + - peat-mesh + environment: + PEAT_NODE_LISTEN: tcp://0.0.0.0:50051 + PEAT_NODE_DATA_DIR: /data/peat-node + PEAT_NODE_APP_ID: attach-demo + PEAT_NODE_NODE_ID: attach-a + PEAT_NODE_SHARED_KEY: fPARVR0eqxhTNqqjKIrYrYTD1VBP6PjGKCAAa39+wl8= + PEAT_NODE_IROH_UDP_PORT: "51071" + # Peer = attach-b's derived endpoint ID @ node-b's container name + iroh port. + PEAT_NODE_PEERS: a53fca1d6263555d269d1594b2e5b05e88a0bab126b3885b9ee74259bd6ecd2f@peat-attach-b:51072 + PEAT_NODE_DISABLE_MDNS: "true" + PEAT_NODE_AUTO_SYNC: "true" + # SENDER: auto-distribute any stable file dropped in ./outbox/ (no gRPC call needed). + PEAT_NODE_ATTACHMENT_ROOT: outbox=/var/lib/peat/outbox + PEAT_NODE_ATTACHMENT_OUTBOX_WATCH: "true" + RUST_LOG: peat_node=info,peat_mesh=info,peat_protocol=info,iroh=warn + ports: + - "50051:50051" # gRPC / Connect API (peer traffic goes over the shared network) + volumes: + - peat-node-a-data:/data/peat-node + - ./outbox:/var/lib/peat/outbox:ro + +networks: + peat-mesh: + external: true + +volumes: + peat-node-a-data: {} diff --git a/examples/compose/attachments/node-b/docker-compose.yml b/examples/compose/attachments/node-b/docker-compose.yml new file mode 100644 index 0000000..7c94bc8 --- /dev/null +++ b/examples/compose/attachments/node-b/docker-compose.yml @@ -0,0 +1,57 @@ +# Node B — RECEIVER (inbox watcher). Runs alongside ../node-a. +# +# Both nodes are separate compose projects that join a SHARED external +# Docker network ("peat-mesh") and dial each other by container name. +# See ../node-a/docker-compose.yml for the full rationale (Docker +# Desktop won't route the iroh QUIC/UDP handshake between isolated +# bridge networks via host.docker.internal). +# +# Quick start (run from this directory): +# +# docker network create peat-mesh # once; shared by both nodes +# mkdir -p inbox +# docker compose up -d +# # (also start ../node-a) +# ls inbox/ # files from node-a appear here +# +# Teardown: +# docker compose down -v +# docker network rm peat-mesh # after both nodes are down +# +# See ../node-a/docker-compose.yml for the shared key and endpoint ID +# derivation details. + +services: + peat-node-b: + image: ghcr.io/defenseunicorns/peat-node:v0.4.8 + # build: + # context: ../../../.. + container_name: peat-attach-b + networks: + - peat-mesh + environment: + PEAT_NODE_LISTEN: tcp://0.0.0.0:50051 + PEAT_NODE_DATA_DIR: /data/peat-node + PEAT_NODE_APP_ID: attach-demo + PEAT_NODE_NODE_ID: attach-b + PEAT_NODE_SHARED_KEY: fPARVR0eqxhTNqqjKIrYrYTD1VBP6PjGKCAAa39+wl8= + PEAT_NODE_IROH_UDP_PORT: "51072" + # Peer = attach-a's derived endpoint ID @ node-a's container name + iroh port. + PEAT_NODE_PEERS: fdf9a98cb65237d2f2d70e1af68e7d2f543e14a287775a4e1bf12a35b84bccb6@peat-attach-a:51071 + PEAT_NODE_DISABLE_MDNS: "true" + PEAT_NODE_AUTO_SYNC: "true" + # RECEIVER: inbox watcher auto-fetches + writes delivered blobs. + PEAT_NODE_ATTACHMENT_INBOX: /var/lib/peat/inbox + RUST_LOG: peat_node=info,peat_mesh=info,peat_protocol=info,iroh=warn + ports: + - "50061:50051" # gRPC / Connect API (host 50061 → container 50051) + volumes: + - peat-node-b-data:/data/peat-node + - ./inbox:/var/lib/peat/inbox + +networks: + peat-mesh: + external: true + +volumes: + peat-node-b-data: {}