Attachment delivery is direct-peer-only — no multi-hop / relayed blob delivery

[kitplummer]

Summary

Attachment byte delivery is currently direct-peer-only. An attachment can only be delivered to a node that is (a) a direct, mutually-dialed peer of the sender and (b) directly connected to a node that holds the blob. There is no multi-hop delivery and no delivery via a relay peer other than a direct holder — even though the distribution document (CRDT metadata) gossips transitively across the whole mesh.

This is an upstream design limitation in peat-protocol (targeting) and peat-mesh (blob fetch), not a peat-node bug. peat-node only consumes these APIs. Filing here as the consumer-side tracking issue; implementation lands upstream, one PR per repo.

Evidence

Targeting — peat-protocol resolve_targets (storage/file_distribution.rs). Every distribution scope resolves through the sender's known_peers() (direct dials only):

• AllNodes → known_peers().fmt_short()
• Nodes { node_ids } → requested ids filtered to known_peers
• Formation → known_peers today, with a literal // TODO: Query formation membership from Automerge documents
• Capable → known_peers

A node reachable only transitively (2+ hops) receives the distribution document (gossip is transitive) but is never in target_nodes, so its receive watcher correctly skips it.

Fetch — peat-mesh fetch_blob (storage/iroh_blob_store.rs). The receiver pulls bytes only from its own known_peers. A blob_peer_index exists but is populated from prior direct fetches, not discovery. With no connected peer it errors ("Remote fetch requires P2P connectivity"). There is no "ask the mesh who has blob X" and no store-and-forward relay. (iroh relay_urls is connection-level NAT traversal, not application-level blob multi-hop — the candidate provider set is still known_peers.)

Impact

• Operators must wire full bidirectional dials between every sender/receiver pair for attachments to work. Across NAT this silently breaks when only one dial direction succeeds: the document syncs over the working direction, but target_nodes (or the blob fetch) excludes the unreachable side, so the file never lands.
• Defeats the DDIL premise that a file written on one node should propagate through the mesh via whatever connectivity exists — not just to its direct neighbors.

Proposed direction (upstream)

1. peat-protocol — membership-based targeting. Resolve target_nodes against the gossiped / Automerge mesh-membership set, not the sender's direct known_peers. The existing Formation TODO is the natural seam.
2. peat-mesh — provider discovery and/or store-and-forward. Either query the mesh for blob providers ("who has blob X?") or have intermediary peers cache and re-provide, so any connected peer holding the blob can serve it — not only a direct holder.

Per-repo work

• peat-protocol: membership-based resolve_targets
• peat-mesh: blob provider discovery / store-and-forward relay
• peat-node: integration + a multi-hop (3-node, non-adjacent sender/receiver) delivery functest once the upstream pieces land

Related

• Observability that makes the current direct-peer requirement diagnosable (peer-status heartbeat + full sync-stack logging): feat(node): peer-status heartbeat + log the full sync stack #169

[kitplummer]

Refinement: separate the "bidirectional-dial" footgun from true multi-hop

Investigating the transport layer clarified that there are two distinct problems here, and the first is much cheaper to fix:

1. Adjacent peers require a two-way dial (low-effort fix). A single iroh QUIC connection is bidirectional and is reused by the accept path — the bytes only ever need one connection. But peat-mesh's known_peers is populated only on outbound dial: add_peer is called solely from the connect paths (peer_connector.rs:140, and peat-node's connect_peer). The iroh accept loop does not register the inbound peer into known_peers. Consequence: when A dials B but B does not dial A, B can target/fetch from A, but A's known_peers stays empty — so A can't target B and can't fetch from B. Operators must wire PEAT_NODE_PEERS on both sides for one connection.

Proposed fix: register the inbound peer into the blob store's known_peers from the accept path. A single dial (one direction) then suffices for both targeting and fetch between directly-connected peers. This removes the most common silent-failure mode (one-directional config across NAT) without any of the multi-hop machinery below.

2. Non-adjacent (true multi-hop) delivery (larger fix). Even with (1), a node 2+ hops away — connected transitively but never directly — still can't be targeted (membership) or served the blob (provider discovery). That's the resolve_targets membership change + fetch_blob provider-discovery/store-and-forward work described above.

Suggest sequencing (1) first as a standalone peat-mesh PR — it's small, independently shippable, and fixes the symptom most deployments actually hit.

Updated per-repo work

• peat-mesh: register inbound-accepted peers into known_peers (fixes the two-way-dial requirement for adjacent peers) — do first
• peat-protocol: membership-based resolve_targets (multi-hop targeting)
• peat-mesh: blob provider discovery / store-and-forward relay (multi-hop fetch)
• peat-node: 3-node non-adjacent delivery functest once the above land

[kitplummer]

Adjacent-peer fix opened: defenseunicorns/peat-mesh#261 — SyncProtocolHandler::accept now registers inbound-authenticated peers into known_peers, so a single dial suffices between directly-connected peers. Includes a regression test (inbound_dialed_peer_is_registered_in_known_peers) verified to fail without the fix and pass with it.

Remaining (true multi-hop, non-adjacent): membership-based resolve_targets + blob provider discovery / store-and-forward — still open.

Once peat-mesh cuts rc.43, peat-node bumps its dependency pin to pick this up.

[kitplummer]

Design decision: multi-hop blob delivery (discovery + store-and-forward)

Decided (with @kitplummer): build both halves.

• Discovery = "who holds blob X" → directs a fetch to the nearest reachable holder and surfaces holders a node isn't directly peered with.
• Store-and-forward = "the pathway to convergence" → when no holder is currently reachable (partition / NAT / multi-hop-only), intermediaries cache + re-provide so a holder becomes reachable hop-by-hop.

Neither suffices alone: discovery can't fetch from an unreachable holder; forwarding without discovery is blind flooding. Together → epidemic convergence bounded by GC.

What already exists (so we don't rebuild it)

• fetch_blob already tries all known_peers (not just the origin) and prioritizes via a local BlobPeerIndex (blob_to_peers / peer_to_blobs).
• A fetched blob is cached locally and auto-served over iroh-blobs' ALPN — so "store" and "serve" are free once a node has the bytes.
• The gap is purely: (a) BlobPeerIndex is local (populated only from prior direct fetches), and (b) nothing makes a non-targeted node fetch to bridge a gap.

Mechanism

Discovery — mesh-gossiped provider index. A node announces itself as a holder of blob X when it acquires X (local import/add, or a successful fetch_blob). Holders propagate mesh-wide and merge as a CRDT (add-wins set of holder EndpointId+addr, with tombstones for GC). fetch_blob consults the now-mesh-wide index, adds holder addresses to memory_lookup, and dials the nearest reachable holder (iroh relay/holepunch permitting) — not just current direct peers.

Substrate (leaning): reuse the existing Automerge sync as a reserved internal document (e.g. a __blob_providers collection excluded from observe/query). It gossips transitively for free over existing connections and merges as a CRDT; the existing tombstone/TTL/GC machinery handles holder removal. Alternative if reserved docs can't be cleanly hidden from the consumer's data model: a dedicated announce over the sync ALPN. Resolved in Slice 1.

Store-and-forward — opportunistic, bounded, GC-on-convergence. Because every fetcher announces itself, the blob spreads holder→neighbor automatically along any path of willing nodes. To bridge gaps that only route through a non-targeted node, such a node may opportunistically fetch + cache + announce for an active, not-yet-converged distribution, bounded by a relay-cache size/TTL cap. Convergence signal = target set ⊆ holder set; once converged, relays GC the cached copy to reclaim space.

Layering / repo split (per this issue)

• peat-mesh (this work): gossiped provider index, directed fetch_blob, opportunistic relay cache + GC. General — benefits any blob, not just attachments.
• peat-protocol: membership-based resolve_targets (target the gossiped mesh, not direct dials). Required for the far node to be targeted so its watcher fetches.
• Both are needed end-to-end: no point targeting a node that can't fetch, nor bridging to a node that's never targeted.

Staged delivery (one peat-mesh PR each, each independently testable)

1. Gossiped provider index + directed fetch. Holder-announce-on-acquire; fetch_blob consults the mesh-wide index + memory_lookup. Test: 3-node line A–B–C, C discovers and fetches X that originated on A (via B as an announced holder). This alone delivers most multi-hop value via announce-on-fetch chaining.
2. Opportunistic relay for non-targeted bridge nodes (bounded cache). Test: A and C non-adjacent, only-path through non-targeted B; B bridges.
3. GC-on-convergence + cache bounds. Test: relay drops the blob once all targets report holding; size/TTL caps enforced.

Safety: relay cache is bounded (size + TTL) and only active for in-flight distributions; converged blobs are GC'd. No unbounded flooding.

[kitplummer]

Slice 1 substrate decided: dedicated peat/blob-announce/1 ALPN (not an extension of the Automerge sync message types).

Rationale: the sync wire decoder **hard-errors** on an unrecognized SyncMessageType tag (automerge_sync.rs: bail!("Unknown sync message type")), so adding a new tag would break sync against any older node during a mixed-version rollout. A separate ALPN is compat-safe — iroh ALPN negotiation means a node that doesn't speak peat/blob-announce/1 simply never opens the announce stream (graceful degradation to today's direct-peer behavior), and it isolates all new protocol code from the proven sync path. Follows the existing SyncProtocolHandler / enrollment-ALPN pattern.

Announce payload: {blob_hash, holder_endpoint_id, holder_addr}; on receipt → populate BlobPeerIndex + memory_lookup, then bounded transitive re-broadcast (seen-set/TTL). fetch_blob already consumes BlobPeerIndex, so directed multi-hop fetch falls out.

[kitplummer]

Slice 1/3 shipped: defenseunicorns/peat-mesh#262 — provider gossip (discovery + announce-on-fetch).

peat/blob-announce/1 ALPN carries 'who holds blob X' across the mesh into the BlobPeerIndex that fetch_blob already consults; a node announces after each successful fetch, so a blob converges along connected chains. Re-broadcast is limited to newly-learned holdings (TTL-bounded) so the epidemic terminates. Real-iroh e2e proves an announce lands the holder in a peer's peers_with_blob.

Remaining:

• Slice 2 — opportunistic relay so non-targeted bridge nodes carry a blob across a gap.
• Slice 3 — GC-on-convergence + relay cache bounds.
• peat-protocol — membership-based resolve_targets (so far nodes get targeted).
• peat-node — 3-node non-adjacent delivery functest.

[kitplummer]

Slice 1 merged → peat-mesh cc64ec5 (#262), approved + all checks green (incl. P2P Functional Tests). Provider gossip (discovery + announce-on-fetch) is now on peat-mesh main.

Remaining:

• peat-protocol — membership-based resolve_targets (target the gossiped mesh, not just direct peers). This is the critical-path trigger: until a far node appears in a distribution's target_nodes, its receive watcher never calls fetch_blob, so Slice 1's announce-on-fetch chain never starts — multi-hop can't begin regardless of the fetch-side work.
• Slice 2 (peat-mesh) — opportunistic relay so a non-targeted node bridges a gap. Narrower than first thought: for AllNodes/whole-mesh targeting (once membership-based) every node is a target, so Slice 1 already converges; Slice 2 only matters for subset-scoped distributions whose only path runs through a non-target.
• Slice 3 (peat-mesh) — GC-on-convergence + relay cache bounds.

Recommend doing the peat-protocol targeting next (it's what makes multi-hop actually fire end-to-end), then Slice 2/3 as refinements.