Skip to content

Add bounded streaming multipart staging and upload sinks #202

Description

@lukewilliamboswell

Summary

Build the multipart follow-up explicitly deferred by #185.

The host should stream multipart/form-data, capture small declared fields in bounded memory, and stage declared file fields under a writable-root capability. Roc receives metadata and opaque staged-file handles rather than every file byte.

Illustrative shape:

upload = request.body().stage_multipart!({
    root: upload_root,
    allowed_file_fields: ["avatar", "attachment"],
    allowed_value_fields: ["caption"],
    limits: {
        max_parts: 8,
        max_files: 2,
        max_file_bytes: 10 * 1024 * 1024,
        max_part_header_bytes: 8 * 1024,
        max_field_bytes: 16 * 1024,
        max_total_field_bytes: 32 * 1024,
    },
})?

upload.files.get(0)?.publish!({
    relative: destination,
    mode: CreateNew,
})?

Exact types should be designed during implementation.

Why this belongs in the host

A streaming boundary parser sits directly between an untrusted HTTP transport and host files. Keeping the transfer in the host preserves backpressure, bounds memory, enables secure staging, and prevents every file byte crossing into Roc.

Unlike arbitrary callbacks, the application supplies a finite declarative plan before the transfer. This follows design.md's request-body sink model.

Required contract

  • Require a valid multipart/form-data content type and boundary.
  • Enforce independent limits for total body bytes, parts, files, per-file bytes, part headers, individual value fields, and aggregate in-memory fields.
  • Preserve field order and duplicate names.
  • Return text/value fields as bytes unless their character decoding is unambiguous.
  • Treat client filenames and claimed content types as untrusted metadata only.
  • Never use a client filename as a filesystem path.
  • Stage files securely and exclusively on the selected writable root's filesystem.
  • Return opaque staged handles whose final release removes unpublished files.
  • Publish only to validated relative paths with the existing CreateNew semantics.
  • On malformed input, disconnect, timeout, shutdown, saturation, or limit failure, remove every staged file.
  • Do not promise atomic publication across multiple files. Each publication result is explicit.
  • Reject nested multipart in the first version.
  • Track active multipart sinks, staged files, and high-water resource use.

Staging before publication permits the application to inspect bounded form fields and metadata before selecting final destinations without installing a callback into Roc.

Tests

  • Boundary fragments split across every possible transport chunk position.
  • Empty, quoted, and maximum-length boundaries.
  • Repeated fields and repeated file names.
  • Header-count/byte, part-count, file-count, and size limits.
  • Malformed closing boundaries and truncated bodies.
  • Disconnect, timeout, and shutdown at each stage.
  • Cleanup when handles are dropped or publication fails.
  • Traversal and client-filename attacks.
  • Parser fuzzing and a corpus of real browser multipart bodies.
  • Zero leaked files, permits, buffers, or handles.

Non-goals

  • MIME-type trust or file-signature validation
  • Antivirus, sandboxing, or content disarm
  • Arbitrary application callbacks while parsing
  • Client-selected filesystem destinations
  • Unbounded text-field materialization
  • Atomic multi-file transactions
  • Background or resumable uploads

Acceptance criteria

  • Declared multipart files stream to secure staging without materializing file bodies in Roc.
  • Every parser and storage dimension has a finite limit.
  • Unpublished handles clean themselves up on all paths.
  • Publication reuses writable-root and safe-relative-path guarantees.
  • The parser has cross-platform, fragmentation, and fuzz coverage.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions