Skip to content

feat(attachments): mirror outbox layout in inbox + richer transaction logging#173

Merged
kitplummer merged 4 commits into
mainfrom
feat/attachment-transaction-logging
Jun 20, 2026
Merged

feat(attachments): mirror outbox layout in inbox + richer transaction logging#173
kitplummer merged 4 commits into
mainfrom
feat/attachment-transaction-logging

Conversation

@kitplummer

Copy link
Copy Markdown
Collaborator

Surfaced from a live two-node run: the receive log carried only distribution_id/blob_hash/size — no filename, no post-write validation.

  • Receive (inbox sink): info log with filename, bytes, blob_hash, target path; validates the on-disk copy size against the distribution's blob_size before the tmp→target rename. On mismatch, drops the partial + returns Err so the watcher retries (no short file published). Content integrity is already iroh-content-verified on fetch — this is the local-write completeness gate.
  • Send (outbox watcher): auto-distribute log now includes bytes + sha256 alongside filename + distribution_id.

Build + 59 attachment unit tests green.

…te validation

Both sides now describe the transfer by filename, and the receiver validates the
written file before publishing it:

- inbox sink (receive): log at info with filename, bytes, blob_hash, and target
  path; validate the on-disk copy's size against the distribution's blob_size
  before the tmp->target rename. On mismatch, drop the partial and return Err so
  the receive watcher retries instead of publishing a short file. (Content is
  already content-hash-verified by iroh on fetch — this is the local-write
  completeness gate.)
- outbox watcher (send): auto-distribute log now includes bytes + sha256
  alongside filename + distribution_id.

Surfaced while watching a live two-node run: the prior receive log carried only
distribution_id/blob_hash/size — no filename, no post-write check.

@peat-bot peat-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peat QA Review (SHA: 18ebdfd)

No findings.

Scope check: change is confined to src/attachments/inbox.rs, src/attachments/outbox.rs, and CHANGELOG.md. None of the gated surfaces are touched — no proto/sidecar.proto, no Connect RPC additions, no src/crypto.rs, no peat-mesh pin bump, no chart/peat-node/ or zarf.yaml, no src/watcher.rs, no non-Tokio runtime, no new cross-repo imports.

Receive-side post-write validation is correctly placed at the filesystem inbox sink (local-write completeness gate), not at the transport layer — content-hash verification remains upstream in iroh, as the inline comment notes. The Err-return-on-mismatch path relies on peat_protocol::storage::ReceiveSink::deliver's documented retry contract (the receive watcher owns retry per the module header), which matches the change's intent of "drop the partial, let the watcher retry" rather than publishing a short file.

Idiomatic Rust: anyhow::bail! for the failure path, let _ = tokio::fs::remove_file(&tmp).await for best-effort cleanup of the partial (correct — the bail is the load-bearing signal), no unwrap()/clone() overuse, no secret material in the new log fields (distribution_id, filename, bytes, blob_hash, sha256 hex are all non-sensitive transfer metadata). hex is already a direct dependency, so hex::encode(&sha256) adds no new crate.

Log-level lift from debug!info! on the receive path is per-distribution (not per-byte / per-chunk), so the operator-visibility win does not produce a meaningful log-volume regression at sidecar scale.

- Banner: first log line reports peat-node's version + the resolved
  peat-mesh/peat-protocol/peat-schema versions (extracted from Cargo.lock by
  build.rs via cargo:rustc-env). Confirms exactly which build + RC stack a
  container runs, from the top of the logs.
- --print-config / PEAT_NODE_PRINT_CONFIG: opt-in dump of the full resolved
  configuration at startup, with shared_key + encryption_key redacted.

Folded into the observability PR alongside the attachment transaction logging.

@peat-bot peat-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peat QA Review (SHA: b1e1b23)

Incremental review since 18ebdfd.

No findings.

Scope check: delta is confined to build.rs, src/main.rs, and CHANGELOG.md. None of the gated surfaces are touched — no proto/sidecar.proto, no Connect RPC additions, no src/crypto.rs, no peat-mesh pin bump (only a build-time read of the resolved version from Cargo.lock for telemetry), no chart/peat-node/ or zarf.yaml, no src/watcher.rs, no non-Tokio runtime, no new cross-repo imports.

Redaction completeness on --print-config: the two secret-bearing Args fields — shared_key and encryption_key — are both rewritten on the cloned struct before the info!("…{redacted:#?}") formatter runs. The Command::DeriveId arm also carries a shared_key, but its if let Some(Command::DeriveId { … }) = &args.command short-circuit returns before the print_config block, so the subcommand's shared key cannot leak via this path. TLS fields (agent_tls_cert/_key/_ca) are PathBuf, not contents. The flag is opt-in (default_value = "false") and gated behind both --print-config and PEAT_NODE_PRINT_CONFIG — operator-conscious.

build.rs::locked_version: the name = "<crate>" anchor matches only [[package]] headers — Cargo.lock's dependencies array uses bare string entries ("peat-mesh"), not name = …, so cross-section matches are not possible. unwrap_or_default() on read_to_string keeps the build resilient if Cargo.lock is unreadable; missing crates resolve to "unknown" rather than panicking. cargo:rerun-if-changed=Cargo.lock correctly invalidates the captured env vars on dependency bumps. Build-time only — no runtime exposure.

Idiomatic Rust: env!() macros for compile-time captures, args.clone()-before-mutating-redaction is the natural pattern for a Debug snapshot, no unwrap() on user-reachable inputs, if let Some(idx) = … / if let Some(line) = … rather than .unwrap() chains. Log-volume cost is one extra info! per process start (banner) plus the opt-in dump — no per-request impact.

Prior review (18ebdfd) was "No findings" — nothing to re-verify.

Received attachments were landing at {inbox}/{distribution_id}/{basename},
losing the original filename behind a UUID dir and flattening any
subdirectories. Mirror the sender's outbox layout instead: a file dropped at
outbox/sub/report.pdf now lands at inbox/sub/report.pdf with its name and
subdirs intact, and re-delivery of the same path overwrites (latest-wins).

- ingest: build_blob_metadata carries the full relative_path, not the basename
- inbox: inbox_relpath replaces inbox_filename — resolves the sender-provided
  name to a path relative to the inbox root, preserving subdirectories
- inbox: deliver writes at inbox/<relative_path> (parent dirs created, tmp+
  rename in the parent), dropping the distribution_id wrapper dir
- already_delivered keys on the resolved target path

Path-traversal hardening preserved and tightened: only Normal path components
are accepted, so an absolute path or one containing '..' can never escape the
inbox — such a name falls back to a flat {distribution_id}.bin at the root.
New tests cover subdir preservation, traversal/absolute rejection, the flat
fallback on a hostile name, and the size-mismatch bail.

@peat-bot peat-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peat QA Review (SHA: 1fededf)

Incremental review since b1e1b23.

No findings.

Scope check: the single new commit (fix(attachments): write inbox files at original relative path) is confined to src/attachments/inbox.rs and src/attachments/ingest.rs. None of the gated surfaces are touched — no proto/sidecar.proto, no Connect RPC additions or signature changes, no src/crypto.rs, no peat-mesh pin bump, no chart/peat-node/ or zarf.yaml, no src/watcher.rs TLS paths, no non-Tokio runtime, no new cross-repo imports.

Path-traversal guard (inbox_relpath): only Component::Normal is accepted; ParentDir, RootDir, and Prefix all short-circuit to None, and the caller falls back to a flat <distribution_id>.bin at the inbox root — inbox_root.join(...) cannot escape because no .. or absolute components ever reach it. CurDir is silently dropped, and an all-./ name folds to an empty PathBuf which is filtered to None. The unit tests cover the three meaningful traversal vectors (../../etc/passwd, /etc/passwd, a/../../b), the empty/missing/./ fallback cases, and the end-to-end deliver_traversal_name_stays_inside_inbox integration check that confirms a hostile ../../../../tmp/pwned lands at inbox/dist-evil.bin.

Post-write validation tightened from "trust the tokio::fs::copy byte-count" to a true metadata(&tmp).await?.len() re-read before the publishing rename; on mismatch the partial is removed and the watcher retries. The size check is the only completeness gate needed here — iroh content-verifies the bytes on fetch.

Behavioral change to flag for awareness, not as a finding: re-delivery is now latest-wins on the mirrored path (previously namespaced under {distribution_id}/). The PR documents this explicitly and adds an idempotency assertion that the partial marker is cleaned up on overwrite.

Test coverage tracks the surface change: the removed inbox_filename_* tests are replaced by relpath_* tests at the new boundary, and the already_delivered_* tests now key on the mirrored path. Subdirectory creation via create_dir_all is exercised by deliver_mirrors_relative_path_and_is_idempotent.

Idiomatic Rust: Component match with explicit reject-on-non-Normal is the standard Rust path-sanitisation pattern, unwrap_or_else(|| ...) for the fallback PathBuf is correct (lazy allocation), and as_deref()? / ? propagation in inbox_relpath is clean. No unwrap() chains on user-reachable inputs.

Prior review (b1e1b23) was "No findings" — nothing to re-verify.

The delivery tests still expected the old {inbox}/{distribution_id}/{name}
layout and broke once attachments started landing at inbox/<relative_path>.

- attachment-delivery-compose.sh: poll inbox/payload.bin, not inbox/$DIST/...
- attachments_e2e_test: await_inbox_file walks the inbox tree by content
  (new find_matching_file helper) instead of keying on the distribution_id;
  the NodeList negative-case scans C's whole inbox for the payload bytes

Verified: all 4 two-node e2e tests pass, and the compose delivery script
passes against the fix image (2 MiB payload, byte-identical, ~2s).

@peat-bot peat-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peat QA Review (SHA: 409a546)

Incremental review since 1fededf.

No findings.

Scope check: the single new commit (test(attachments): assert mirrored inbox layout (no distribution_id dir)) is confined to tests/attachments_e2e_test.rs and test/attachment-delivery-compose.sh — both test-only. No gated surfaces touched: no proto/sidecar.proto, no Connect RPC additions or signature changes, no src/crypto.rs, no peat-mesh pin bump, no chart/peat-node/ or zarf.yaml, no src/watcher.rs TLS paths, no non-Tokio runtime, no cross-repo imports.

Test alignment is correct for the inbox layout change shipped in 1fededf: attachments now land at inbox/<relative_path> (no distribution_id wrapper dir), so the prior await_inbox_file(inbox, distribution_id, expected, deadline) signature and the bash poll path $WORK/inbox/$DIST/payload.bin were stale. The new helper drops the distribution_id parameter and walks the inbox tree by content via find_matching_file, which is the right shape now that the on-disk path no longer encodes the distribution id.

find_matching_file is sound: synchronous std::fs is fine in a test helper (explicitly justified in the doc comment), the stack-based walk avoids recursion bounds concerns, .flatten() on read_dir entries skips unreadable entries cleanly, and dotfile-prefix filtering preserves the in-flight .partial marker exclusion. Byte-exact equality (actual == expected) is a strictly stronger signal than the prior size-match heuristic.

Negative-case tightening in node_list_scope_only_delivers_to_listed_nodes: the assertion that C must NOT receive is now a byte-exact scan over C's entire inbox tree rather than a size check inside the (no-longer-existing) dist_dir. Stronger signal, correct shape post-layout-change. The change also preserves the contract-shape assertion that SendAttachments returns a distributionId even though the value is no longer needed for the on-disk lookup — appropriate.

Idiomatic Rust: let Ok(...) = ... else { continue }, let Some(found) = ... early-return, .is_some_and for the dotfile predicate, and entries.flatten() are all standard 2024-edition patterns. No unwrap() on user-reachable inputs; no clone-fights with the borrow checker.

Surface-tier coverage unchanged on the Connect RPC layer (no new RPCs, no signature changes). The e2e file exercises SendAttachments / GetAttachmentDistribution end-to-end across two nodes as before; only the on-disk verification path was updated.

PR-description claim ("all 4 two-node e2e tests pass, and the compose delivery script passes against the fix image") is consistent with the diff — the test helper is the only mechanical block that needed updating for the layout change.

Prior review (1fededf) was "No findings" — nothing to re-verify.

@kitplummer kitplummer changed the title feat(attachments): richer transaction logging + receive-side post-write validation feat(attachments): mirror outbox layout in inbox + richer transaction logging Jun 20, 2026
@kitplummer kitplummer merged commit 48e63bf into main Jun 20, 2026
21 of 22 checks passed
@kitplummer kitplummer deleted the feat/attachment-transaction-logging branch June 20, 2026 04:10
@kitplummer kitplummer mentioned this pull request Jun 20, 2026
3 tasks
kitplummer added a commit that referenced this pull request Jun 20, 2026
Headline: received attachments now keep their original filename and folder
layout — the inbox mirrors the sender's outbox path (inbox/<relative_path>)
instead of burying files under inbox/<distribution_id>/<basename> (#173).

Also ships in this cut (previously [Unreleased]): startup version banner
(peat-node + peat-mesh/peat-protocol/peat-schema), --print-config /
PEAT_NODE_PRINT_CONFIG, and richer two-sided attachment transaction logging
with receive-side post-write size validation.

No dependency change vs 0.4.7 (peat-mesh rc.43 / peat-protocol rc.26).
kitplummer added a commit that referenced this pull request Jun 21, 2026
…x layout (#173)

#173 changed the production FilesystemInboxSink to write delivered blobs at
{inbox}/{relative_path} (mirroring the sender's outbox layout), but the
peat-cli InboxSink and the e2e TestInboxSink still nested every delivery
under {inbox}/{distribution_id}/{filename}. A peat-cli receiver and a
peat-node receiver therefore disagreed on where files land.

Bring both sinks into lockstep with production:
- peat-cli InboxSink: resolve a sanitised inbox-relative path (subdirs
  preserved, absolute/.. rejected -> {distribution_id}.bin fallback),
  create nested parents, add the post-write size-validation guard.
- e2e TestInboxSink: same path resolution + already_delivered keyed on the
  mirrored path.
- Update the two e2e scenarios to assert the mirrored path and that no
  {distribution_id}/ dir is created.
- Fix stale doc comments: attach.rs module + watch help, main.rs
  --attachment-inbox.

Regression tests (peat-cli lib): deliver_mirrors_relative_path_not_distribution_id_dir,
deliver_traversal_name_falls_back_inside_inbox, deliver_size_mismatch_bails_without_publishing,
plus inbox_relpath_* unit coverage. Adds chrono as a dev-dependency for the
DistributionDocument fixture.

Verified: fmt, clippy -D warnings, cargo test --workspace (the two
real-iroh attach e2e scenarios now assert the mirrored layout end-to-end).
kitplummer added a commit that referenced this pull request Jun 22, 2026
…#173) + docs/version-minimums (#175)

* docs(attachments): correct inbox layout to mirrored outbox path (#173) + version minimums

The example README described delivered files landing at
inbox/{distribution_id}/{filename}. As of v0.4.8 (#173) the inbox mirrors
the sender's outbox layout instead (outbox/sub/f -> inbox/sub/f, original
name, latest-wins), with an unsafe-name fallback to {distribution_id}.bin
at the inbox root. Validated end-to-end across two arm64 hosts, both directions.

- README: rewrite the three stale {distribution_id} path claims; add a
  per-feature version-minimums table (PRD-006 v0.2.0, reliable delivery
  v0.3.0, derive-id v0.4.4, outbox-watch v0.4.5, mirrored layout v0.4.8).
- Bump all three example compose pins v0.4.5 -> v0.4.8 so they exhibit the
  documented layout and match the validated run.

Does NOT touch the peat-cli InboxSink / e2e TestInboxSink / main.rs doc
comment, which still describe+implement the old layout (tracked separately).

* fix(attachments): align peat-cli + test inbox sinks to mirrored outbox layout (#173)

#173 changed the production FilesystemInboxSink to write delivered blobs at
{inbox}/{relative_path} (mirroring the sender's outbox layout), but the
peat-cli InboxSink and the e2e TestInboxSink still nested every delivery
under {inbox}/{distribution_id}/{filename}. A peat-cli receiver and a
peat-node receiver therefore disagreed on where files land.

Bring both sinks into lockstep with production:
- peat-cli InboxSink: resolve a sanitised inbox-relative path (subdirs
  preserved, absolute/.. rejected -> {distribution_id}.bin fallback),
  create nested parents, add the post-write size-validation guard.
- e2e TestInboxSink: same path resolution + already_delivered keyed on the
  mirrored path.
- Update the two e2e scenarios to assert the mirrored path and that no
  {distribution_id}/ dir is created.
- Fix stale doc comments: attach.rs module + watch help, main.rs
  --attachment-inbox.

Regression tests (peat-cli lib): deliver_mirrors_relative_path_not_distribution_id_dir,
deliver_traversal_name_falls_back_inside_inbox, deliver_size_mismatch_bails_without_publishing,
plus inbox_relpath_* unit coverage. Adds chrono as a dev-dependency for the
DistributionDocument fixture.

Verified: fmt, clippy -D warnings, cargo test --workspace (the two
real-iroh attach e2e scenarios now assert the mirrored layout end-to-end).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants