Merge master into nix-next#1717
Open
github-actions[bot] wants to merge 110 commits into
Open
Conversation
Update `harmonia`; deduplicate various cargo output hashes
Prepare for reuse by pulling process-management and polling logic out of `QueueRunnerBuildOne` into standalone modules: - `ProcessGroup.pm`: reusable labelled process group with `spawn`, `pump_logs`, and ordered teardown. - `wait_for_builds` in `QueueRunnerContext`: polling loop that checks the queue runner REST API until all build IDs are inactive. - `start_queue_runner` and `start_nix_daemon` in `QueueRunnerContext`: extracted from `QueueRunnerBuildOne` so future test helpers can start queue infrastructure without duplicating the setup. `QueueRunnerBuildOne` is rewritten to use these shared helpers. The queue-runner harness is still registered manually because `start_queue_runner` returns pre-bound sockets and a separate daemon harness that don't fit `ProcessGroup::spawn`'s simple model.
hydra-tests: Extract `ProcessGroup` and shared test helpers
Apply [semantic line breaks](https://sembr.org/) (one sentence per line) across all markdown files in the manual and repository. This makes diffs easier to review since changing one sentence no longer reflows an entire paragraph. Also add `CONTRIBUTING.md` documenting the convention, and clean up various pandoc artifacts along the way: escaped quotes (`\'`), escaped underscores (`\_`), broken multi-line list items, and non-breaking spaces.
Reformat markdown to use semantic line breaks
Forgot to do this when bumping harmonia. The commented stuff will return when we merge `nix-next`.
`buildproducts.path` can contain a sub-path below a store output (e.g. `/nix/store/…-nix-manual-2.31.4/share/doc/nix/manual`), but `OwnedBuildProduct::parse_paths` was calling `StoreDir::parse` which only accepts bare store paths. Any path with a `/` after the store path name failed with `invalid store path / symbol`, breaking the cached-build fast path for those builds. Move `RelativeStorePath` from the queue-runner into a new `store-path-utils` crate so the DB crate can use it directly. Change `OwnedBuildProduct`'s default type parameter from `StorePath` to `RelativeStorePath` and have `parse_path()` call `RelativeStorePath::from_path`, which correctly splits the full path into a `StorePath` base and a relative suffix.
When `create_step` encounters an existing step (`is_new=false`), it now re-checks whether the step's outputs have appeared in the local nix store since the step was first created. Without this check, builds whose outputs became available between poll cycles (e.g. via concurrent builds, substitution, or external upload) would get stuck in an infinite re-load loop: the DB says `finished=0`, the step already exists in memory so `create_build` inserts the build but never routes it to `handle_cached_build`. The fix adds two checks before returning `Valid(step)`: 1. If the step is already marked finished, return `None` immediately. 2. Query missing outputs from the local store; if all are present, mark the step finished, insert into `finished_drvs`, and return `None` so `create_build` routes to `handle_cached_build`.
Nix writes `Compression: bzip2` in narinfo files (see `src/libstore/nar-info.cc`), but hydra's parser only accepted `bz2` (the file extension, not the compression name). This caused errors like `invalid value for Compression: bzip2` when reading old narinfo entries from `cache.nixos.org`, making the queue runner treat those store paths as missing and rebuild them. Fix `FromStr` to accept `bzip2`, and fix `as_str` to write `bzip2` (matching nix). The file extension method `ext()` correctly uses `.nar.bz2` and is unchanged. Examples of affected narinfo files: - https://cache.nixos.org/2crhfrb8pqy61d32jaajpw33jjmny6xz.narinfo - https://cache.nixos.org/9fggf2bx5p29da89jrbmdh2fcd68m1mx.narinfo - https://cache.nixos.org/0xs59grdbgz7z2ci2ff4yfwb6jha1qaa.narinfo
The architecture doc was sitting outside `src/`, presumably because it was mostly notes and not yet cleaned up enough to be part of the user-facing docs. We will do that clean up next, but before we do that, move the file and expose it, so the subsequent diff is readable and not confused by the move.
- Organize components by deployment topology: master machine vs builder machines vs destination store, reflecting which components share a Nix store and which don't - Add `hydra-builder`, `hydra-notify`, `hydra-eval-jobset` — components that weren't documented before - Update `hydra-queue-runner` language from C++ to Rust - Add source layout and build system overview - Add database schema section with pointers to the SQL and migration docs - Update FAQ: both Nix and Hydra now have experimental dynamic derivations - Fix typos (`hydra-build-producs`, `obsolute`)
Add updated architecture section to the manual
Pick up latest changes from nix-community/harmonia.
Both `hydra-builder` and `hydra-queue-runner` had near-identical `build.rs` files that compiled the same `.proto` file and generated the same version constant. This consolidates that into a single `hydra-proto` crate with feature flags: - `client` — generates tonic client stubs (used by `hydra-builder`) - `server` — generates tonic server traits (used by `hydra-queue-runner`) - `db` — `From` impls bridging proto enums to `db::models` types Proto ↔ native type conversions are also consolidated here: - `ProtoStorePath` moved from `shared::proto`, now depends on `harmonia-store-core` directly rather than `nix-utils` - `Pressure`/`PressureState` — both components now use the generated proto types directly, eliminating duplicate structs and field-by-field conversion boilerplate - `PresignedUploadOpts`, `ConfigUpdate` — queue-runner now uses the proto types directly instead of maintaining trivial local wrappers - `BuildResultState` — restructured as a nested enum wrapping `hydra_proto::BuildResultState` with extra `Aborted`/`Cancelled` variants for queue-runner-internal states - `RelativeStorePath` — new proto message type making the store path vs sub-path split explicit on the wire; `From`/`TryFrom` conversions in `hydra-proto` bridge to `store_path_utils::RelativeStorePath` - `BuildProduct.path` changed from opaque `string` to structured `RelativeStorePath` message Also: - Groups all local workspace crates under `[workspace.dependencies]` with `workspace = true` references instead of relative paths - Updates the architecture doc with descriptions of each shared crate
Extract hydra-proto crate for shared `gRPC` code generation
Introduces `store.proto` (`nix.store.v1` package) containing types that are generic to any Nix store, not specific to Hydra: - `StorePath`, `StorePaths`, `RelativeStorePath` - `ValidPathInfo` — store path with NAR hash/size, references, deriver, CA - `NarInfo` — wraps `ValidPathInfo` with cache-specific fields (url, compression, file hash/size) `streaming.proto` (`runner.v1`) imports these and uses them for `BuildMessage.drv`, `BuildProduct.path`, `PresignedUploadComplete`, etc. `PresignedUploadComplete` is restructured to contain a `NarInfo` message instead of flattening all fields, making the layering explicit.
Split `.proto` into generic Nix store types and Hydra-specific types
The `binary-cache` crate had its own `NarInfo` struct that duplicated
what harmonia now provides as `harmonia_store_nar_info::NarInfo`. This
replaces it with the upstream type, which also brings in
`harmonia-store-path-info` for `ValidPathInfo`/`UnkeyedValidPathInfo`.
The local struct was flat; the harmonia type is nested:
`NarInfo { path, info: UnkeyedNarInfo { info: UnkeyedValidPathInfo, url, compression, ... } }`.
All construction and field access sites are updated accordingly.
`From<harmonia_store_nar_info::NarInfo>` and `TryFrom<proto::NarInfo>`
conversions are added in the proto crate so both components can convert
between native and wire representations without manual field-by-field
marshalling. Round-trip unit tests verify the conversion is lossless.
Also adds `store_dir` field to proto `ValidPathInfo` to match the
harmonia type. The queue-runner now asserts the store dir matches
rather than silently overriding it.
Replace local `NarInfo` with `harmonia-store-nar-info`
Consumers now import `StorePath`, `StoreDir`, `OutputName`, etc. directly from `harmonia-store-core` rather than through `nix-utils` re-exports. This makes the actual dependency on harmonia explicit at each use site. `nix-utils` retains its own local functionality (`parse_store_path`, `BaseStore`, `LocalStore`, `realise_drv`, etc.) but no longer acts as a forwarding layer for harmonia types.
Remove harmonia re-exports from `nix-utils`
Replace stringly-typed fields in `store.proto` with proper message types: - `Hash` — algorithm enum + raw digest bytes - `Signature` — key name + signature string - `ContentAddress` — method enum + hash `UnkeyedValidPathInfo.nar_hash` is now `Hash` (not string), `signatures` is `repeated Signature` (not `repeated string`), `ca` is `optional ContentAddress` (not `optional string`). `UnkeyedNarInfo.file_hash`/`file_size` renamed to `download_hash`/`download_size` with proper `Hash` type. All `From`/`TryFrom` impls use `&` for harmonia→proto direction (no need to consume the source) and by-value for proto→harmonia (proto types are consumed). Round-trip tests cover every layer.
Use structured proto types for `Hash`, `Signature`, and `ContentAddress`
Move each test from the monolithic `nixos-tests.nix` into its own file under `nixos-tests/`, with shared VM configuration extracted to `common.nix`. No functional changes to existing tests.
Split NixOS tests into separate files
Bumps [nix](https://github.com/NixOS/nix) from `24a0724` to `2208f98`. - [Commits](NixOS/nix@24a0724...2208f98) --- updated-dependencies: - dependency-name: nix dependency-version: 2208f98581bdc36f08f135d38bad30a798b34a4b dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
build(deps): bump nix from `24a0724` to `2208f98`
build(deps): bump nixpkgs from `41a7bd5` to `f58ee5f`
hydra-tests: Add basic test for logging
Harmonia's ATerm parser validates that CAFixed output paths match the content hash. The placeholder `aaaa...` was accepted by the older parser but is now rejected with `CAFixed output path mismatch`. Use the correct base32 hash that sha256:deadbeef... produces for name "test-src".
Introduce typed error enums so callers can distinguish error kinds
(e.g. DB infrastructure failure vs bad data) without string matching.
This lays the groundwork for the future error handling cleanups, by
allowing us to see more clearly what changes are made.
db crate:
- New `db::Error` enum: `Sql` (infrastructure) vs `Data(DataError)`
- `DataError` has `StorePath`, `StorePathName`, `IntConversion`, and
`BuildProductMissingPath { build_id, productnr }` variants
- `BuildProductRow` now carries `build` and `productnr` from the query
so error messages identify the offending row
- Removed `anyhow` dependency
nix-utils crate:
- Replaced `Error::Anyhow` catch-all with `DerivationParse` and
`StorePathName` variants
- Fixed CA fixed-output test to use correct output path
- Removed `anyhow` dependency
store-transfer crate:
- New `Error` enum: `Store`, `Grpc`, `Io`, `Protocol`, `Join`
- `ProtocolError` sub-enum: `EmptyStream`, `MissingHeader`,
`MissingGrpcField`, `InvalidPathInfo`, `TruncatedNar`
- Removed `anyhow` dependency
nix-support crate:
- `anyhow::Result` → `std::io::Result` (all errors are IO)
- Removed `anyhow` dependency
hydra-tracing crate:
- `anyhow::Result` → `Result<_, Box<dyn Error>>` (init-only)
- Removed `anyhow` dependency
binary-cache crate:
- `InvalidS3Scheme` and `InvalidCompression` error structs with
named `got` field replace `String` error variants
- `UrlParseError` variants renamed for clarity
proto crate:
- `NarInfoConvertError` now implements `std::error::Error`
Also adds `sqlx-prepare.sh` for regenerating the sqlx offline cache.
Replace `anyhow` with structured error types in support crates
chore: cleanup clippy lints
Bumps [nix](https://github.com/NixOS/nix) from `2208f98` to `8294f1c`. - [Commits](NixOS/nix@2208f98...8294f1c) --- updated-dependencies: - dependency-name: nix dependency-version: 8294f1c1fda17f87e65723ac9348395d2f07513c dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Remove instances of to_string_lossy
Bump harmonia
Add bidirectional conversions between `harmonia_utils_hash::Algorithm` and `nix::store::v1::hash::Algorithm`, then refactor the `Hash` `From`/`TryFrom` impls to delegate to them instead of duplicating the match inline. This is how it should have been all long --- the `From`/`TryFrom` impls should always do one thing and rely on other impls modularly --- but I didn't notice before we were missing this factoring out. Note: prost stores proto enum fields as `i32` and generates its own Rust enum; there's no way to tell prost-build to use a pre-existing enum type for a field. These `From` impls are the standard bridge between the generated proto enum and harmonia's domain type.
hydra-proto: add `Algorithm` `From` impls, simplify Hash conversions
build(deps): bump nix from `2208f98` to `8294f1c`
flake: bump nixpkgs to release-25.11
…eruns on early exit
Although 20-byte debug IDs are the most common, a number of lengths are valid. Instead of ignoring all non-20-byte debug IDs, accept all names that would be used by elfutils (of libelf fame) Co-authored-by: Amaan Qureshi <git@amaanq.com>
…injection feat(queue-runner): implement concurrent build injection and faster reruns on early exit
binary-cache: accept more debug ids
from_utf8_lossy is rarely what one wants, as it silently discards errors and can cause corruption when going from raw bytes to string and back.
binary-cache: remove String::from_utf8_lossy
Bumps harmonia to current main, which adds `write_derivation` for writing a resolved derivation to the store over the daemon protocol.
Print error traceback in JobFailure
deps: bump harmonia
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated PR to keep
nix-nextin sync withmaster.