v0.35.1

Note

This release was brought to you by the Shipyard team.

What's Changed

Added

• new span for the handleIncoming bitswap client getter plus events when blocks are received.
• mark opentelemetry spans, span attributes, and span events as being used by ProbeLab's analysis scripts

Changed

• upgrade to go-dsqueue v0.1.0 - Fixes batch reuse that could cause panic.

Fixed

• gateway: Fixed duplicate peer IDs appearing in retrieval timeout error messages
• bitswap/client: fix tracing by using context to pass trace and retrieval state to session #1059
  • bitswap/client: propagate trace state when calling GetBlocks #1060
• bitswap/network/httpnet: improved error detection on HTTP and block fetches:
  • Do not attempt to GET a test CID if the endpoint returns 429 to the test HEAD request.
  • Unify error parsing and handling of http statues and content.

Full Changelog: v0.35.0...v0.35.1

v0.35.0

Note

This release was brought to you by the Shipyard team.

What's Changed

Added

• pinning/pinner: Added CheckIfPinnedWithType method to Pinner interface for efficient type-specific pin checks with optional name loading (#1035)
  • Enables checking specific pin types (recursive, direct, indirect) without loading all pins
  • Optional includeNames parameter controls whether pin names are loaded from datastore
  • CheckIfPinned now delegates to CheckIfPinnedWithType for consistency
• gateway: Enhanced error handling and UX for timeouts:
  • Added retrieval state tracking for timeout diagnostics. When retrieval timeouts occur, the error messages now include detailed information about which phase failed (path resolution, provider discovery, connecting, or data retrieval) and provider statistics including failed peer IDs #1015 #1023
  • Added Config.DiagnosticServiceURL to configure a CID retrievability diagnostic service. When set, 504 Gateway Timeout errors show a "Check CID retrievability" button linking to the service with ?cid=<failed-cid> #1023
  • Improved 504 error pages with "Retry" button, diagnostic service integration, and clear indication when timeout occurs on sub-resource vs root CID #1023
• gateway: Added Config.MaxRangeRequestFileSize to protect against CDN issues with large file range requests. When set to a non-zero value, range requests for files larger than this limit return HTTP 501 Not Implemented with a suggestion to use verifiable block requests (application/vnd.ipld.raw) instead. This provides protection against Cloudflare's issue where range requests for files over 5GiB are silently ignored, causing excess bandwidth consumption and billing

Changed

• routing/http: ✨ Delegated Routing V1 HTTP endpoints now return 200 with empty results instead of 404 when no records are found, per IPIP-513 (#1024)
  • Server endpoints (/routing/v1/providers/{cid}, /routing/v1/peers/{peer-id}, /routing/v1/ipns/{name}) return HTTP 200 with empty JSON arrays or appropriate content types for empty results
  • Client maintains backward compatibility by treating both 200 with empty results and 404 as "no records found"
  • IPNS endpoint distinguishes between valid records (Content-Type: application/vnd.ipfs.ipns-record) and no record found (any other content type)
• verifcid: 🛠 Enhanced Allowlist interface with per-hash size limits (#1018)
  • Expanded Allowlist interface with MinDigestSize(code uint64) and MaxDigestSize(code uint64) methods for per-hash function size validation
  • Added public constants: DefaultMinDigestSize (20 bytes), DefaultMaxDigestSize (128 bytes for cryptographic hashes), and DefaultMaxIdentityDigestSize (128 bytes for identity CIDs)
  • DefaultAllowlist implementation now uses these constants and supports different size limits per hash type
  • Renamed errors for clarity: Added ErrDigestTooSmall and ErrDigestTooLarge as the new primary errors
  • ErrBelowMinimumHashLength and ErrAboveMaximumHashLength remain as deprecated aliases pointing to the new errors
• bitswap: Updated to use verifcid.DefaultMaxDigestSize for MaximumHashLength constant
  • The default MaximumAllowedCid limit for incoming CIDs can be adjusted using bitswap.MaxCidSize or server.MaxCidSize options
• 🛠 bitswap/client: The RebroadcastDelay option now takes a time.Duration value. This is a potentially BREAKING CHANGE. The time-varying functionality of delay.Delay was never used, so it was replaced with a fixed duration value. This also removes the github.com/ipfs/go-ipfs-delay dependency.
• filestore: Support providing filestore-blocks. A new provider.MultihashProvider parameter has been added to filestore.New(). When used, the blocks handled by the Filestore's FileManager will be provided on write (Put and PutMany).

Removed

• provider: Provide() calls are replaced with StartProviding() to benefit from the Reprovide Sweep improvement. See kubo#10834 and kad-dht#1095.

Fixed

• routing/http/client:
  • Fixed off-by-one error in routing_http_client_length metric - the metric now correctly reports 0 for empty results instead of 1
  • Added metrics for IPNS operations (GetIPNS and PutIPNS) - these now report latency, status code, and result count (0 or 1 for GetIPNS)
  • Added simple counter metrics to avoid confusing histogram bucket math:
    • routing_http_client_requests_total - total requests including errors
    • routing_http_client_positive_responses_total - requests that returned at least 1 result
• ipld/unixfs/mod:
  • DagModifier now correctly preserves raw node codec when modifying data under the chunker threshold, instead of incorrectly forcing everything to dag-pb
  • DagModifier prevents creation of identity CIDs exceeding verifcid.DefaultMaxIdentityDigestSize limit when modifying data, automatically switching to proper cryptographic hash while preserving small identity CIDs
  • DagModifier now supports appending data to a RawNode by automatically converting it into a UnixFS file structure where the original RawNode becomes the first leaf block, fixing previously impossible append operations that would fail with "expected protobuf dag node" errors
• mfs:
  • Files with identity CIDs now properly inherit full CID prefix from parent directories (version, codec, hash type, length), not just hash type (#1018)

Security

• verifcid: Now enforces maximum size limit of 128 bytes for identity CIDs to prevent abuse (#1018, ipfs/specs#512).
  • 🛠 Attempts to read CIDs with identity multihash digests longer than DefaultMaxIdentityDigestSize will now produce ErrDigestTooLarge error.
  • Identity CIDs can inline data directly, and without a size limit, they could embed arbitrary amounts of data. Limiting the size also protects gateways from poorly written clients that might send absurdly big data to the gateway encoded as identity CIDs only to retrieve it back. Note that identity CIDs do not provide integrity verification, making them vulnerable to bit flips. They should only be used in controlled contexts like raw leaves of a larger DAG. The limit is explicitly defined as DefaultMaxIdentityDigestSize (128 bytes).

Full Changelog: v0.34.0...v0.35.0

v0.34.0

What's Changed

Added

• autoconf: Client library for fetching, caching and expanding IPFS network configurations using "auto" placeholders
• gateway: Added configurable limits for gateway resource protection:
  • Config.RetrievalTimeout: Maximum duration between writes of non-empty data to HTTP response body (default: 30s). Returns 504 Gateway Timeout when gateway cannot retrieve content within this period.
  • Config.MaxConcurrentRequests: Limits concurrent HTTP requests (default: 4096, suitable for most deployments). Returns 429 Too Many Requests with 60s Retry-After header when exceeded. To restore previous unlimited behavior set both RetrievalTimeout and MaxConcurrentRequests to 0.

    [!IMPORTANT]
    If your gateway returns many HTTP 429 responses while having available resources (CPU, memory), increase MaxConcurrentRequests. If experiencing high load or resource exhaustion, decrease it. See the MaxConcurrentRequests godoc for detailed tuning guidance.

  • Config.MetricsRegistry: Optional Prometheus registry for metrics isolation. When nil, uses the default global registry. Useful for testing and deployments with multiple gateway instances.
  • New middleware with Prometheus metrics:
    • ipfs_http_gw_concurrent_requests: Gauge tracking number of concurrent requests
    • ipfs_http_gw_responses_total{code}: Counter for all HTTP responses by status code
    • ipfs_http_gw_retrieval_timeouts_total{code,truncated}: Counter for retrieval timeout events with details on truncation
• namesys/IPNSPublisher: option to PublishOptions that allows for setting a custom sequence number for the IPNS record with proper validation to prevent unintentional replay attacks. #962
• blockstore: Added ValidatingBlockstore wrapper. This replaces the HashOnRead blockstore API.

Changed

• bitswap/network: The connection event manager now has a SetListeners method. Both bsnet and httpnet now have options to provide the ConnectionEventManager during New(...). This allows sharing the connection event manager when using both. The connection manager SHOULD be shared when using both networks with the network.Router utility.
• provider: Distribute the responsability of providing new blocks to the places that play a role in the different providing strategies #976. Refactor the logic to perform Provides, when the component has been given a provider:
  • Remove providing.Exchange
  • Provide directly from Blockstore when provider is set (via Option).
  • Provide directly from pinner/merkledag on dag traversal when provider is set (via Option).
  • Provide from MFS whenever there is a call to DAGService.Add and provider is set (via constructor param).
• upgrade to go-libp2p v0.43.0
• replace uber-go/multierr with errors.Join #996
• updated Go in go.mod to 1.24.0 #999

Removed

• blockstore: Removed HashOnRead API. This is a potentially BREAKING CHANGE for any users of the HashOnRead API. Use the ValidatingBlocksore instead.
  • The HashOnRead function was also removed from Filestore. To use this functionality, provide a ValidatingBlockstore when creating a new Filestore.

Full Changelog: v0.33.1...v0.34.0

This release was brought to you by the Shipyard team.

v0.33.1

What's Changed

Added

• provider: Add ability to clear provide queue #978

Changed

• bitswap/network: The connection event manager now has a SetListeners method. Both bsnet and httpnet now have options to provide the ConnectionEventManager during New(...). This allows sharing the connection event manager when using both. The connection manager SHOULD be shared when using both networks with the network.Router utility.
• bootstrap: Relay-only peers (with /p2p-circuit/ addresses) are now filtered out when selecting backup bootstrap peers to improve reliability.
• bitswap/client: disable use of traceability block #956
• bitswap/httpnet: start pinging before signaling Connected 982
• upgrade to go-libp2p v0.42.1

Fixed

• bitswap: fix an issue where boxo silently stops making http retrieval requests. #981, #980, #979 and #984 (writeup), #986.

Full Changelog: v0.33.0...v0.33.1

This release was brought to you by the Shipyard team.

v0.33.0

What's Changed

Added

• bitswap/network/httpnet: New WithMetricsLabelsForEndpoints allows defining which hosts/endpoints can be used for labelling metrics that support such label. '*' enables this for all endpoints receiving HTTP requests, but may cause metric cardinality growth when too many endpoints exist. These labels allow tracking, for example, number or requests per response status AND endpoint used. Non-labelled request hosts are labelled with same value: other.

Changed

• DontHaveTimeoutConfig's default MinTimeout is changed from 0 to 50ms #959 #965.
• upgrade to go-libp2p v0.42.0

Fixed

• bitswap/client: Fix sending extra wants #968 + #975
• routing/http/client: Improve URL handling for delegated routing endpoints #971

Security

• fix panic when incoming Bitswap protobuf message does not contain Wantlist #961

Full Changelog: v0.32.0...v0.33.0

This release was brought to you by the Shipyard team.

v0.32.0

What's Changed

Added

• provider includes metrics on the number of keys provided so far(reprovider_provide_count) and the number of keys reprovided so far (reprovider_reprovide_count) #944
• bitswap/client: New metrics:
  • ipfs_bitswap_wanthaves_broadcast: Count of want-haves broadcasts
  • ipfs_bitswap_haves_received: Count of total have responses
  • ipfs_bitswap_bcast_skips_total{: Count of broadcasts skipped as part of spam reduction logic (see "Changed" below)
  • ipfs_bitswap_unique_blocks_received: Count of non-duplicate blocks recieved

Changed

• provider: previously, the code in this module was logging to reprovider.simple, provider.batched and provider.queue facilities. They have now been consolidated in a single provider logging facility, along with some adjustments to logging levels and extra debug statements.
• bitswap/client: Added an opt-in ability to reduce bitswap broadcast volume by limiting broadcasts to peers that have previously responded as having wanted blocks and peers on local network. The following bitswap client options are available to configure the behavior of broadcast reduction:
  • BroadcastControlEnable enables or disables broadcast reduction logic. Setting this to false restores the previous broadcast behavior of sending broadcasts to all peers, and ignores all other BroadcastControl options. Default is false (disabled).
  • BroadcastControlMaxPeers sets a hard limit on the number of peers to send broadcasts to. A value of 0 means no broadcasts are sent. A value of -1 means there is no limit. Default is -1 (unlimited).
  • BroadcastControlLocalPeers enables or disables broadcast control for peers on the local network. If false, then always broadcast to peers on the local network. If true, apply broadcast control to local peers. Default is false (always broadcast to local peers).
  • BroadcastControlPeeredPeers enables or disables broadcast control for peers configured for peering. If false, then always broadcast to peers configured for peering. If true, apply broadcast control to peered peers. Default is false (always broadcast to peered peers).
  • BroadcastControlMaxRandomPeers sets the number of peers to broadcast to anyway, even though broadcast control logic has determined that they are not broadcast targets. Setting this to a non-zero value ensures at least this number of random peers receives a broadcast. This may be helpful in cases where peers that are not receiving broadcasts may have wanted blocks. Default is 0 (no random broadcasts).
  • BroadcastControlSendToPendingPeers enables or disables sending broadcasts to any peers to which there is a pending message to send. When true (enabled), this sends broadcasts to many more peers, but does so in a way that does not increase the number of separate broadcast messages. There is still the increased cost of the recipients having to process and respond to the broadcasts. Default is false.

Removed

• bitswap/server do not allow override of peer ledger with WithPeerLedger #938

Fixed

• gateway: Fixed suffix range-requests and updated tests to gateway-conformance v0.8 #922

Full Changelog: v0.31.0...v0.32.0

This release was brought to you by the Shipyard team.

v0.31.0

What's Changed

Added

• bitswap/httpnet: New option WithMaxDontHaveErrors(int) (defaults to 100)
  will trigger disconnections from bitswap peers that cannot provide any of
  the blocks that are requested for the given number of requests in a
  row.. This is meant to limit bitswap HTTP-based optimistic requests for
  blocks to discovered endpoints, which were before considered permanently
  peered upon discovery.

Changed

• upgrade to go-libp2p-kad-dht v0.33.1
• deprecated WithPeerLedger option for bitswap server. Will remove option in next release. See issue #928
• gateway: update backend car traversal to use go-car/v2 #925
  • This change may prohibit fetching DAGs from CAR files that have no roots.

Full Changelog: v0.30.0...v0.31.0

This release was brought to you by the Shipyard team.

v0.30.0

What's Changed

Added

• Control over UnixFS DAG Width
  • We have made some changes to allow setting custom max width of UnixFS DAGs. This enables users to produce DAGs that follow current and future community conventions (see the related discussion).
  • ipld/unixfs: DagModifier now allows specifying file DAG Width (MaxLinks) #898
  • ipld/unixfs/io/directory: We have made some changes to unixfs directory tooling #906
    • We have exposed creator methods for new BasicDirectory and HAMTDirectory, that complement the existing NewDirectory() which creates dynamic directories.
    • We have added WithCidBuilder(...), WithMaxLinks(...), WithMaxHAMTFanout(...) and WithStat(...) as options to these new methods so that empty directories can be initilized as wished from the get-go.
    • WithMaxLinks(...) and WithMaxHAMTFanout(...) are new options that allow to set a limit to the number of children that a directory DAG node can have. For details on what they exactly do for each of the directory type, please check the documentation.
  • mfs supports as well the new MaxLinks and MaxHAMTFanout options. They have been made part of the MkdirOptions object and the methods NewEmptyDirectory() and NewEmptyRoot() have been added to facilitate the initialization of MFS objects. #906
• provider: added support for walking partial DAGs in offline mode #905
  • a KeyChanFunc that traverses DAGs from a given root (NewDAGProvider).
  • a KeyChanFunc that buffers all the CIDs in memory from another KeyChanFunc (NewBufferedProvider).
  • fetcher/impl/blockservice: new option SkipNotFound for the IPLD fetcher. It will skip not found nodes when traversing the DAG. This allows offline traversal of DAGs when using, for example, an offline blockservice.
  • This enables use case of providing lazy-loaded, partially local DAGs (like ipfs files in Kubo's MFS implementation, see kubo#10386)
• gateway: generated HTML with UnixFS directory listings now include a button for copying CIDs of child entities #899
• bitswap/server: Add ability to enable/disable bitswap server using WithServerEnabled bitswap option (#911)[https://github.com//pull/911]

Changed

• upgrade to go-libp2p v0.41.1
• bitswap/network: Add a new requests_in_flight metric gauge that measures how many bitswap streams are being written or read at a given time.
• improve speed of data onboarding by batching/bufering provider queue writes #888
• provider: providing queue is now independent from reprovides, speeding up initial provides #907
  • renamed provider.ReproviderStats.TotalProvides => provider.ReproviderStats.TotalReprovides
  • renamed provider.ReproviderStats.AvgProvideDuration => provider.ReproviderStats.AvgReprovideDuration
• provider/queue deduplicates CIDs #910

Fixed

• gateway: query parameters are now supported and preserved in redirects triggered by a _redirects file #886
• provider: adjusted first reprovide timing after node reboot #890
• gateway: validate configuration and warn when UseSubdomains=true is used with IP-based hostnames #903

Full Changelog: v0.29.1...v0.30.0

This release was brought to you by the Shipyard team.

v0.29.1

What's Changed

Changed

• bitswap/httpnet: do not follow redirects #878
• provider: provider helper that buffers results to fix issues with slow re-providing #870

Fixed

• provider: don't reprovide if reprovideInterval is set to 0 #871
• gateway: Skip DNSLink lookup for IP addresses to avoid unnecessary DNS queries #880
• bitswap/client: Fix unintentional ignoring DontHaveTimeoutConfig #872

Full Changelog: v0.29.0...v0.29.1

v0.29.0

What's Changed

Added

• feat(bitswap/client): MinTimeout for DontHaveTimeoutConfig #865
• ✨ httpnet: Transparent HTTP-block retrieval support over Trustless Gateways [#747]((#747):
  • Complements Bitswap as a block-retrieval mechanism, implementing bitswap/network.
  • Understands peers found in provider records with /.../http endpoints (trustless gateway).
  • Treats them as "Bitswap" peers, except instead of using Bitswap it makes HTTP/2 requests to discover (HEAD) and retrieve (GET) individual blocks (?format=raw).
  • A bitswap/network proxy implementation allows co-existance with standard bitswap/network/bsnet.
  • httpnet is not enabled by default. Upstream implementations may use it by modifying how they create the Bitswap network and initialize bitswap.

Changed

• ipns: The DefaultRecordTTL changed from 1h to 5m #859
• upgrade to go-libp2p v0.41.0
• upgrade to go-libp2p-kad-dht [v0.30.2](github.com/libp2p/go-libp2p-kad-dht v0.30.2)
• upgrade to go-datastore v0.8.2 - includes API updates and removes go-process
• bitswap/client reduce lock scope of PeerManageer to help performance #860
• NewFromIpfsHost() and the rest of the BitSwapNetwork implementation has moved from github.com/ipfs/boxo/bitswap/network to github.com/ipfs/boxo/bitswap/network/bsnet.

Removed

• Removed dependency on github.com/hashicorp/go-multierror so that boxo only depends on one multi-error package, go.uber.org/multierr. #867

Full Changelog: v0.28.0...v0.29.0

This release was brought to you by the Shipyard team.