Repository navigation
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
August 24, 2026 20:14
ee193f4 to
447e45c
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
3 times, most recently
from
September 11, 2026 11:20
c0ff943 to
ab87573
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
September 11, 2026 14:23
ab87573 to
ce0fde6
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
15 times, most recently
from
September 15, 2026 21:22
be37170 to
a65d90f
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
September 16, 2026 18:41
a65d90f to
2a0badd
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
September 16, 2026 19:45
2a0badd to
64ea1b5
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
2 times, most recently
from
October 2, 2026 08:57
acd5568 to
6ec5d6a
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
October 8, 2026 08:20
6bc783a to
dfbb2b2
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
October 9, 2026 07:38
dfbb2b2 to
7eeda1c
Compare
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
October 14, 2026 13:21
7eeda1c to
f58d3d8
Compare
toddbaert
pushed a commit
to open-feature/spec
that referenced
this pull request
Oct 14, 2026
> **Part of [#417](#417 This PR is the **assets** — the files four conformance implementations execute. It stacks on [#440](#440), which configures the releases that give these assets a tag. The normative description of them, Appendix F, is [#439](#439), stacked on this. [#425](#425) adds the machine-readable report. > > This PR previously carried the appendix too. It was split so the assets can land while the prose is still being reviewed. ## What The artifacts a provider conformance suite runs against: | path | what it is | |---|---| | `assets/provider-tck/gherkin/` | six feature files — evaluation, errors, events, lifecycle, reason, metadata | | `assets/provider-tck/flags/canonical-flags.json` | the flag set every scenario assumes | | `assets/provider-tck/openapi/control-api.yaml` | the HTTP surface a backend under test must expose | | `assets/provider-tck/go.mod`, `embed.go` | a nested Go module exposing the artifacts as an `embed.FS` | | `release-please-config.json`, `.release-please-manifest.json` | registers the assets as a releasable component; the general setup is #440 | Marked **experimental**: the scenario set is a representative subset, and the capability vocabulary may still change. ## Why this is split out Four language implementations already run these files, and all four are currently pinned to an **unmerged branch tip**. Every rebase here invalidates them — it has happened three times in two weeks. Nothing about those pins becomes real until the assets are on `main` with a tag, so landing the executable half first is worth more than landing it together with prose that will take longer to review. Appendix F is the same work, reviewed on its own terms. ## Why a Go module is in here A Go module is distributed as a zip of the VCS tree, in which a git submodule is only a gitlink and its files are absent. The other three languages build from a working tree and keep using the submodule, but a Go suite would have to vendor a copy of every artifact and police it against drift. `embed.go` removes the copy: a consumer pins a revision, the checksum database makes it immutable, and the embedded bytes are the same bytes every other language reads out of the submodule. `go.mod` and `embed.go` are inert for every non-Go consumer. Precedent: `open-feature/flagd-schemas`. The release config tags the assets as their own component, `specification/assets/provider-tck/vX.Y.Z` — the shape `go get` needs. Without it a Go consumer can only name a pseudo-version. ## Releasing the assets The general release-please setup is [#440](#440), which this stacks on. What this PR adds is the part that describes *these* assets: a second package, so they are tagged `specification/assets/provider-tck/vX.Y.Z`. That tag shape is what `go get` needs. Without it a Go consumer can only name a pseudo-version, which is what the four conformance implementations have had to do while this branch was unmerged, and it has cost several rounds of re-pinning. The specification package sets `exclude-paths` for this directory, so an asset-only change no longer cuts a specification release, and `separate-pull-requests` keeps the two from sharing one release pull request. ## The capability vocabulary lives in the README, for now A scenario exercising an optional part of the contract carries a tag; a provider declares what it supports, and a scenario gated on an undeclared tag is reported **skipped, with the reason** — never as passed. A suite that quietly goes green on scenarios it did not run is worse than no suite at all. That rule and the tag table are in `assets/provider-tck/README.md` so the tags in the feature files are not left undefined while the appendix is in review. #439 moves them into the appendix and leaves a pointer behind, so they end up with one home rather than two that can disagree. Worth stating plainly: **withholding a tag is not an admission of a defect.** `@numeric-coercion` borrows its rule from flagd's coercion ADR, and `@string-typing` / `@fully-typed-values` sit on a question the specification does not answer at all — [#430](#430) and [#433](#433). ## Decisions worth reviewing - **`assets/provider-tck/` rather than beside the SDK Gherkin**, because `assets/gherkin/evaluation.feature` already exists and tests an SDK, not a provider. This answers #417's Q1 — the subdirectory is forced rather than chosen. - **`.gitattributes` normalises these to LF.** Four languages consume them byte for byte and copy them verbatim into published artifacts, so a Windows checkout must not package differently from a Linux one. - **`missing-flag` must not exist** in the flag set — its absence is what the `FLAG_NOT_FOUND` scenario tests. The other four load-bearing properties of the flag set are in the README; each one, if seeded wrong, turns a scenario green or red for the wrong reason in every language at once. - **No container restarts.** Outages are produced through the control API, because orchestrators reassign host ports on restart and every provider pointed at the old one then fails looking flaky. ## Implementations All four execute exactly these files, against flagd (both resolvers), OFREP and Flagsmith: | | suite | flagd | OFREP | Flagsmith | |---|---|---|---|---| | Java | [#1830](open-feature/java-sdk-contrib#1830) | [#1847](open-feature/java-sdk-contrib#1847) | [#1840](open-feature/java-sdk-contrib#1840) | [#1849](open-feature/java-sdk-contrib#1849) | | Go | [#940](open-feature/go-sdk-contrib#940) | [#941](open-feature/go-sdk-contrib#941) | [#942](open-feature/go-sdk-contrib#942) | [#959](open-feature/go-sdk-contrib#959) | | JavaScript | [#1606](open-feature/js-sdk-contrib#1606) | [#1608](open-feature/js-sdk-contrib#1608) | [#1612](open-feature/js-sdk-contrib#1612) | [#1623](open-feature/js-sdk-contrib#1623) | | Python | [#409](open-feature/python-sdk-contrib#409) | [#411](open-feature/python-sdk-contrib#411) | [#414](open-feature/python-sdk-contrib#414) | — | ## What running them has found - **flagd silently narrows a float flag to an integer**, Go and Java, both resolvers, no error code — [flagd#1996](open-feature/flagd#1996) - **flagd reported ready before flags were evaluable**, on 100% of starts — [flagd#2047](open-feature/flagd#2047), fixed in flagd v0.17.0 - **The Java SDK's `MultiProvider` swallows child provider events** — [java-sdk#1882](open-feature/java-sdk#1882) - **The Python flagd provider widens a boolean to a float** — [python-sdk-contrib#417](open-feature/python-sdk-contrib#417) - **The JavaScript Flagsmith provider stringifies unconditionally** where its other paths correctly report a mismatch — visible only because three languages ran identical scenarios against one backend - two specification gaps, raised rather than decided here: [#430](#430), [#433](#433) --------- Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
flagd resolves flags two quite different ways, and they are separately conformant: any difference between the two results is a difference an application would see when it switches resolver. Two spec files rather than one, because jest-cucumber accumulates step definitions in module state and two runProviderTck calls in one file would make every step ambiguous. The TCK drives flagd-testbed's launchpad through the standardised control API, which the launchpad already implements, so neither the testbed nor the existing e2e suites in this directory change. The stack is started once per suite and never restarted; isolation comes from the control API instead. Both resolvers declare Events, Stale, ConfigurationChange, Object and UnavailableInit. The Stale declaration is the notable one: in Go the RPC resolver never emits PROVIDER_STALE while in-process does (open-feature/go-sdk-contrib#939), but here both report a lost connection through the same disconnectCallback seam -- grpc-service.ts:274 and grpc-fetch.ts:197 -- and the single handler behind it, flagd-provider.ts:130-148, emits PROVIDER_STALE before escalating to PROVIDER_ERROR. The contract is implemented once, in the provider, so it cannot differ between resolvers. StrictNumericTyping is left undeclared, as it must be for every JavaScript provider: the language has no integer type, so the scenario is unsatisfiable by construction rather than by defect. Targeting and Caching are undeclared because no scenario carries their tags yet. Closes #1607 Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Comments only. Both suites already omit the capability, and for a reason the rename does not change: JavaScript has no integer type, so no provider written in it can answer either half of the coercion contract. Worth noting what that means here as against the other languages. Go and Java withhold this capability for flagd because the provider narrows 0.5 to 0 with no error -- a defect, tracked as open-feature/flagd#1996. These suites omit it because the question cannot be asked at all. Same omission, unrelated causes, and the conformance report distinguishes them: a JavaScript report carries it under `declaration.notApplicable` with the language as the reason, rather than leaving it merely undeclared. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The e2e target carried `"parallel": false`, which the `@nx/jest:jest` executor has no such option for -- `parallel` belongs to `nx:run-commands`, which the neighbouring `generate` and `pullSpec` targets do use. Unknown executor options are dropped silently, so the declaration read as serial and behaved as the default: jest spread the four e2e spec files across workers. `runInBand` is the option that means what this one was trying to say. Serial execution matters here beyond speed, because each spec file brings up its own flagd-testbed Compose stack. Four at once is four sets of containers competing for the same host resources, and -- until the following commit -- for the same flags directory. Worth knowing while reading this: `--testPathPattern` no longer filters either. jest 30 renamed it to `--testPathPatterns`, and the executor schema followed, so passing the old singular name selects nothing and silently runs everything. Nothing in the repository passes it; it is a trap for whoever next tries to run one spec from the command line. The target's own `--testFile` is unaffected. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Both resolvers left `@lifecycle` undeclared, which skipped the whole of `lifecycle.feature` -- six scenarios that Java runs against this same provider. Nothing about flagd justified the omission: `initialize` awaits `connect`, which awaits `waitForReady` on the gRPC channel (grpc-service.ts:194-202) and rejects when the backend is unreachable, so both terminal outcomes are observable rather than synthesised by the SDK. That is exactly what the tag asserts, and it is the harder claim of the two -- `@events` alone can be satisfied vacuously, because every SDK synthesises PROVIDER_READY for a provider that does nothing on startup. In-process earns it more plainly still: it syncs the whole ruleset before reporting ready. `@reinitialization` is deliberately NOT declared, and this is the half worth reading. Requirement 2.5.2 says a provider SHOULD revert to its uninitialized state after shutdown, and its supporting text says "some providers MAY allow reinitialization from this state" -- permitted, not required. flagd declines it: `onClose` delegates to `disconnect`, which calls `close()` on the client (grpc-service.ts:140-143 for RPC, grpc-fetch.ts:95-99 for in-process) and nothing constructs a new one. So the reuse scenario is reported as skipped, and NO known deviation is recorded against it. An earlier pass at this recorded one, which would have published a specification-sanctioned choice as a flagd defect -- the scenario was untagged at the time, and the reflex was to treat a failing scenario as evidence of a bug rather than to go and read the requirement. The rule this cost: find the numbered requirement before recording a deviation or withholding a capability, because a failing scenario is not on its own evidence of anything. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Both resolvers now declare eight capabilities rather than six, and both declarations are made on a run rather than on a reading of the provider. @targeting is the one worth having. flagd-testbed already serves targeting-key-flag -- it was added upstream in February 2024 and the testbed's own targeting.feature uses the same flag, variants and uuid -- so nothing had to be seeded and no image bump was needed. All three scenarios pass on both resolvers: a matching context resolves "hit", a non-matching one resolves "miss", and no context at all resolves "miss" without erroring. That last one is not a formality for this provider, which has two quite separate paths to a targeting decision: RPC asks flagd to evaluate the rule, in-process syncs the ruleset and evaluates it locally in flagd-core. The two agreeing is the finding, the same way it is for staleness. @Variants is the honest half. Seven of the eight gated rows pass on both resolvers. The eighth asks for large-integer-flag, which this testbed image does not serve at all (flagd-testbed#392) -- the same missing flag that already fails the mandatory 2^31 - 1 scenario here, and the reason LargeIntegers stays undeclared. No knownDeviation is recorded for it: a deviation claims the provider gets something wrong, and what is missing is a flag in the backend's set. Withdrawing @Variants instead would claim flagd has no variant concept, which the other seven rows disprove. @caching stays undeclared and is now the only reserved name left, so the comments that said "Targeting and Caching because no scenario carries their tags yet" no longer described anything. Measured over 104 scenario instances, 52 per resolver: 4 failed, 10 skipped, 90 passed. Both failures per resolver are the missing large-integer-flag. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Nothing had to be seeded. The four flags are flagd-testbed's own, from
flags/disabled-flags.json, and the launchpad already serves them under the
`default` configuration, which is why Appendix F took its names and values from
there.
Both resolvers pass all four rows, and they reach the answer by different
routes. In-process syncs the ruleset and evaluates locally, so the caller's
default is in hand when the state is read: flagd-core returns it directly
(flagd-core.ts:176-181). RPC is the interesting half, because the evaluation is
remote and the request never carried a default -- flagd answers with the type's
zero value, `{"value":"0","reason":"DISABLED","variant":""}` for
disabled-integer-flag -- and the provider substitutes the caller's default
client-side when the variant is empty and the reason is DEFAULT or DISABLED
(grpc-service.ts:306-312). Without that substitution three of the four rows
would have failed on the value alone, which is what makes the declaration a
statement about this provider rather than about flagd.
The two resolvers agreeing is the same finding as for @Variants and @targeting:
an application switching resolver sees the same four values.
Measured, not read. Each suite is 56 scenarios, 5 skipped, 49 passed, 2 failed
-- and both failures are the pre-existing testbed gap: `large-integer-flag` is
absent from the image (flagd-testbed#392), which fails the mandatory 2^31 - 1
scenario and the one @Variants row that asks for the same flag. Neither is new
and neither is the provider's.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The adoption hand-rolled its container lifecycle: `FlagdComposeContainer.build()`
at module scope, a `beforeAll` to start it, a guarded `afterAll` to stop it, a
port lookup per resolver, and an `HttpControl` built from a thunk over the
launchpad's mapped port. Every one of those is now the suite's job, so all of it
goes: `runContainerizedProviderTck` is handed a Compose file, the two ports the
two resolvers connect to, and a factory that reads the endpoint it discovers.
The Compose file is this adoption's own, at `src/e2e/tck/docker-compose.yaml`,
and it is deliberately not `test-harness/docker-compose.yaml` next door. Three
differences, each of which is the point:
- **No `/flags` bind mount.** The launchpad writes the configuration it is
asked for into the container's own `/flags` from the rawflags baked into the
testbed image, so the mount buys nothing and costs isolation: the
neighbouring e2e suites share one host directory through it, and with two
containers up each rewrites the other's flags. That is why this branch
previously grew a per-instance temp directory -- and why that commit is gone
rather than kept. Removing the shared resource beats scheduling around it,
and a conformance suite whose failures read as provider defects
(`error-code was 'FLAG_NOT_FOUND', expected 'TYPE_MISMATCH'` on a flag that
exists) is the worst possible thing to leave to a scheduling guarantee.
`flagdComposeContainer.ts` is therefore untouched by this branch again: the
pre-existing e2e suites still import it, so it cannot be deleted, but the
conformance suites no longer go near it.
- **No environment variables.** The image tag is pinned in the file rather
than read from `test-harness/version.txt`. Every capability the two suites
declare rests on a run against v3.10.1 specifically, so a bump is a
deliberate act with a result to record, not something a dependency update
does quietly.
- **No host port bindings.** The suite requires dynamically mapped ports and
discovers them after startup.
One Compose file for both suites, as in Java: 8013 is RPC evaluation and 8015
the in-process sync stream, both served by the one testbed service, so each
suite declares only the port its resolver uses.
`retryGracePeriod` stops being a per-suite knob. Both resolvers passed 30, and
they have to: the grace period is emitted from the shared provider layer rather
than from either transport, which is the same reason both declare `@stale`.
Measured rather than asserted, on v3.10.1, under node 24:
- flagd-rpc: 49 passed, 5 skipped, 2 failed.
- flagd-in-process: 49 passed, 5 skipped, 2 failed.
Identical, which is the finding the two suites exist to produce. Both failures
are the two already recorded against the testbed image rather than against the
provider -- the mandatory 32-bit precision scenario and the `max-int32` row of
the `@variants` outline, both wanting a `large-integer-flag` that v3.10.1 does
not serve (open-feature/flagd-testbed#392). The `@stale`, `@lifecycle` and
`@unavailable` scenarios all pass, which is what establishes that the suite's
await-ready on the control API is sufficient without the post-command settle
delay that was dropped.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The suites are Docker-gated and run by hand before merge, not on every push. They were not: this repository's CI has an `e2e` job running `npm run e2e`, which is `nx run-many --all --target=e2e`, and libs/providers/flagd/src/e2e/jest.config.ts has no testMatch, so Jest's default swept up anything named *.spec.ts under src/e2e -- including two conformance suites that pull a backend image and, against the pinned testbed, fail two scenarios for a missing flag (flagd-testbed#392). CI would have gone red on this branch for a reason that is not about this provider. So the boundary is now a directory rather than a filename. The suites, their shared body and their Compose file live in src/e2e/tck with a Jest config of their own and an nx target of their own: npx nx tck providers-flagd src/e2e/jest.config.ts ignores that directory and loses the @openfeature/tck path mapping it no longer needs, so this branch's footprint on the pre-existing e2e config is one ignore pattern. The suites keep runInBand, and the target drops the flagd-core pullTestHarness dependency that `e2e` needs, because the image tag is pinned in the Compose file instead of read from the submodule. The provider README says all of this, because an exclusion nobody wrote down is indistinguishable from an oversight -- which is exactly how this went unnoticed. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The reasoning now lives in Appendix F's "Running the suite in CI", promoted there because four READMEs is where this kind of decision had already drifted into several answers. What stays here is the mechanism, which is this repository's alone: the suites live in src/e2e/tck/ behind a Jest project of their own, src/e2e/jest.config.ts ignores that directory, and `npx nx tck providers-flagd` is the only way in. The first mistake Appendix F names was made here, so it is named here too: the suites once sat in src/e2e/tests/, where the pre-existing e2e target's default testMatch swept them up, and that target is a CI job in this repository. The directory boundary exists because of it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… runs The `tck` target depended on `generate`, `pullTestHarness` and `buildValidators` but not on `tck:pullSpec`, so nothing in the graph checked the `open-feature/spec` submodule out. The library's own `test` target has that dependency; this one did not, and the difference is invisible until the pin moves. Found by it happening. Rebasing this branch onto a base that re-pinned the submodule moved the *gitlink* and left the submodule's working tree on the previous revision, because that is all a gitlink is. The suite then ran the old feature files: a new capability was declared, the feature file carrying it was not on disk, and the run went green having tested nine scenarios fewer than it claimed -- with no count to compare against, because an out-of-date asset set is internally consistent with itself. The same thing bit the Python suite on the same pin. A green run that tested less than it claimed is the one failure mode a conformance suite must not have, so this is a dependency in the graph rather than a line in the README. The README says why it is there. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
flagd reports the standard resolution vocabulary with the standard meanings, so both suites say so and reason.feature checks the claim. All nine scenarios run in each -- the tag composes with @targeting and @disabled-flags and both are already declared here -- and all nine pass in both: STATIC for the four rule-less flags, ERROR for the unknown flag and the type mismatch, TARGETING_MATCH for the matching key and DEFAULT for the miss, DISABLED for the disabled flag. STATIC for a rule-less flag is the row the specification genuinely leaves open -- types.md types DEFAULT as "no dynamic evaluation occurred or dynamic evaluation yielded no result", which a rule-less flag satisfies as readily -- so it is measured rather than assumed. flagd answers STATIC and DEFAULT in exactly the two places Appendix F separates them. The two resolvers agreeing is the finding rather than a copy: RPC passes the reason through from flagd's evaluation response, in-process gets it from flagd-core evaluating the synced ruleset, and an application switching resolver sees the same reason as well as the same value. That is what makes a reason worth building telemetry on. `npx nx tck providers-flagd`: 130 scenarios, 116 passed, 10 skipped, 4 failed -- eighteen more than before the pin, nine per resolver, none of them skipped. The four failures are unchanged and pre-existing: the testbed image does not serve large-integer-flag (flagd-testbed#392), which fails the mandatory 32-bit precision scenario and one @Variants row in each resolver. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Both suites carried a comment explaining that JavaScript has no integer type and so cannot be asked @numeric-coercion. The library refuses the capability centrally now, and gates its scenarios whatever a suite declares, so the comment documents a decision these files no longer make -- and a per-adoption restatement of a language property is the thing the central refusal replaces. Neither suite ever named the capability, so nothing about what runs changes. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ot a hedge The suite already left the capability undeclared and recorded no deviation, because the missing flag is the testbed's gap rather than the provider's. What it did not say is which rule makes that the right shape. Appendix F now states it: the unit of a declaration decision is the scenario, not the tag, and @large-integers gates exactly one -- so a backend without its flag leaves nothing about the capability establishable and withholding is correct. @numeric-coercion would go the other way for the same reason, gating three, which is why the two absences are formally different things and should not be read as one. The second consequence is the one worth writing down here: an omission for a backend gap is temporary in a way the others in this file are not. Caching is reserved until the specification writes a scenario, Reinitialization is a sanctioned choice this provider makes -- neither expires on its own. @large-integers goes away when the testbed serves the flag, so the note names what closes it. Comment only. No capability list, no deviation and no assertion changes. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
`nx.json` sets `passWithNoTests: true` in the `@nx/jest:jest` target defaults, which the `tck` target inherits like any other Jest target. So the one command documented for running the conformance suite exits 0 having collected zero scenarios if the spec files move, are renamed, or fall under an ignore pattern -- green because nothing ran, which is the failure this suite exists to make impossible for a provider and had wired into its own entry point. The default is right for the rest of the workspace, where a project may legitimately have no tests yet. It is wrong here: this target has exactly one Jest project and it is the conformance suite. Overriding it in the target's own options is the narrowest fix. `cache: true` is inherited too, so a previous result can be replayed rather than re-run. That is left alone -- the inputs include the spec assets, and a replay of a genuine pass is not the same hazard -- but it is worth knowing when someone reports that the suite "passed". Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ide it `src/e2e/tck/` becomes `src/tck/`, a sibling of `src/e2e/` rather than a child of it. Nesting said the conformance suite is a kind of e2e test, which is what the separate `tck` target exists to deny. The two answer different questions -- the e2e suite tests this provider against flagd's own harness, the conformance suite tests it against the OpenFeature provider contract -- and they mean different things by failure: an e2e suite is expected green, while a conformance suite fails scenarios by design wherever a `knownDeviation` is declared. Selection stops being a convention as a result. It used to depend on `src/e2e/jest.config.ts` remembering `testPathIgnorePatterns: ['<rootDir>/tck/']`; now a file is in `src/tck/` or it is not. That ignore pattern is deleted and the unit config ignores `/src/tck/` instead, so neither `nx test providers-flagd` nor `nx e2e providers-flagd` can collect these suites. `passWithNoTests: false` on the `tck` target is what makes a wrong glob after the move fail loudly rather than go green having collected nothing; it was set for that reason and it is load-bearing here. `tckSuite.ts` becomes `suite.ts`: in a directory called `tck` the prefix said it twice. The two spec files already dropped their `tck-` prefix when they moved out of `src/e2e/tests/`, where it had been the selector. The move costs one thing, which is the interesting part. `tsconfig.lib.json` excluded `./src/e2e` wholesale, so it covered the adoption by accident; with the suite outside that directory, `suite.ts` fell into the library build and the build failed with thirteen errors, none of them in the adoption's own code. A library build compiles what gets published, so it sets `types: []` and `module: ES6`, while the conformance harness is a test library whose entry point calls `describe`, `test` and `jest`. `./src/tck` is therefore excluded explicitly, and `tsconfig.spec.json` includes it explicitly -- which, unlike the old `./src/e2e` entry, scopes a future `tsc --noEmit -p tsconfig.spec.json` check to the conformance suite alone rather than to the pre-existing e2e files beside it. Same scenarios, same results, a different directory: `npx nx tck providers-flagd` reports 130 scenarios, 116 passed, 10 skipped, 4 failed, unchanged. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The comment explained that the conformance suites are not in this directory. A note in one suite's configuration about a suite that is somewhere else is the thing the move was meant to remove, not something the move should add -- the directory boundary is the explanation. This leaves the branch touching nothing under src/e2e at all, which is the correct diff for a change that moved files out of it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… is owned
Two things, because both are the same mistake seen from different angles: a
fact restated in a second place drifts from the first.
The Compose file moves to libs/shared/tck-backend/docker-compose.yaml and is
shared with the OFREP adoption. Stripped of comments the two files it
replaces were the same six lines of YAML with a different port list, and a
per-project copy is how two suites come to ask the same questions of two
different backends -- which is exactly what happened across the four
languages, this one having drifted to flagd-testbed v3.10.1 while Go, Java
and Python sit on v3.8.0. The shared file exposes 8013, 8015, 8016 and 8080;
each suite asks the harness only for the ports it uses, and the rest cost
nothing because there are no host bindings. Both tck targets name the file in
their inputs, since it is outside the project root and nx's default
{projectRoot}/**/* would not invalidate a cached run when it changed.
The file itself keeps only what is a trap in this file: that it is not the
flagd-core test-harness Compose file the e2e suites use, which bind-mounts
${FLAGS_DIR} into a shared host directory and runs an Envoy sidecar. The
testbed's missing flags and the dynamic-port convention are
open-feature/flagd-testbed#392's and Appendix F's respectively, and are
linked rather than re-explained.
The suite files keep every per-capability reason with its evidence -- the
DISABLED wire response and the grpc-service.ts:306-312 substitution that
earns @disabled-flags for a resolver that evaluates remotely, the single
disconnectCallback seam that makes @Stale identical in both resolvers, the
testbed gap that makes @large-integers temporary rather than permanent -- and
lose the citations of what Appendix F's rules are, the comparisons with Go's
two resolvers, and the directory-layout argument that the provider README
already makes.
Comments, Markdown and one file move; no behaviour change and no count moves.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Appendix F moved four scenarios out of the mandatory type-mismatch matrix and behind a capability tag, on the grounds that a backend storing flag values as strings satisfies the string accessor for every flag and has no mismatch to report. That tag has since been split in two: @string-typing keeps the boolean and integer rows a partially typed store can still answer, and @fully-typed-values holds the float and structured ones. flagd is fully typed, so both resolvers declare both tags and all four scenarios keep running. Measured rather than assumed, and by two different routes. RPC reads the type off flagd's evaluation response; in-process reads it off the synced flag definition and evaluates in flagd-core. Both refuse the string accessor for all four, named rather than inferred from a total: A non-string flag is not returned as its string representation [boolean-flag] A non-string flag is not returned as its string representation [integer-flag] A float flag is not returned as its string representation A structured flag is not returned as its JSON text Eight passes across the two resolvers, and each run's declaration line carries @fully-typed-values and @string-typing both. Both tags had to be named explicitly, because this adoption declares by an allow-list rather than by everything-except. A capability registered upstream is undeclared here until it is written down, so taking the split without adding the new half would have turned the float and structured scenarios from passes into skips -- a quieter regression than a failure, because the results would simply stop asserting a property flagd demonstrably has, and nothing would say so. Leaving either tag undeclared would have been the cheaper change and the wrong one. Withholding is for a backend that cannot be asked the question, not for one that answers it. Totals are unchanged at 65 scenarios per resolver -- errors.feature still holds 19, four of them now gated by two tags instead of one -- with the same five skips and the same two pre-existing failures in each, both from flagd-testbed not serving large-integer-flag (open-feature/flagd-testbed#392). Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
v3.10.3 is the first testbed release carrying flagd v0.17.0, which includes "wait for flag state before reporting ready" (open-feature/flagd#2050, closing open-feature/flagd#2047). That defect returned 200 from /readyz while the flag store was still empty, on 100% of starts, and is the cause behind the intermittent FLAG_NOT_FOUND runs recorded in these suites. v3.10.2 looked current but shipped flagd v0.16.3; the base-image bump landed after that tag was cut. This also picks up the TLS SANs fix (flagd-testbed#421). Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The adoption restated facts other repositories own: specification rationale for optional capabilities, what the TCK library's vocabulary and gates mean, what flagd-testbed seeds, and the review history of these files. Kept each capability's own reasoning, the measured tallies, and the notes that stop a timeout or an ignore pattern being simplified back into a bug. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
force-pushed
the
feat/provider-tck-flagd
branch
from
October 14, 2026 14:31
f58d3d8 to
d6c15fc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flagd provider under the conformance suite, both resolvers, in
libs/providers/flagd/src/tck/.Two spec files, because jest-cucumber accumulates step definitions in module state and two
runContainerizedProviderTckcalls in one file would make every step ambiguous.The directory is a sibling of
src/e2e/, not a subdirectory of it. The suite used to be excludedfrom the e2e run by one Jest config remembering to ignore a subdirectory of another; that line is
deleted, because a file is in
src/tck/or it is not. The unit config gained/src/tck/intestPathIgnorePatterns, which was not optional — without itnx test providers-flagdwould collectthese specs and start Docker in the unit run.
What it declares
Both resolvers declare the same twelve:
@events,@lifecycle,@stale,@configuration-change,@object,@unavailable,@variants,@targeting,@disabled-flags,@standard-reasons,@string-typingand@fully-typed-values.The last two hold in both resolvers — RPC reads the type off the evaluation response, in-process off
the synced definition via flagd-core, and both refuse the string accessor for a non-string flag. All
four scenarios were verified as passing by name, not inferred from totals.
@numeric-coercionis not withheld here — it is refused by the implementation for everyJavaScript provider.
typeof 10andtypeof 0.5are both'number', so requestingfloat-flagasan integer is indistinguishable from requesting it as a float and no provider in this language can
satisfy the scenario. Naming it fails at configuration time with that reason, and the scenario is
skipped with a message saying the SDK cannot ask the question — not that this provider declined.
The finding: no Go-style asymmetry here
Go's flagd RPC resolver never emits
PROVIDER_STALEwhile its in-process resolver does(go-sdk-contrib#939), so the Go adoption
withholds
@stalefor RPC. JavaScript has no such split, and the reason is structural rather thanlucky.
Neither resolver decides anything about staleness. Both report a lost connection through a
disconnectCallback—service/grpc/grpc-service.tsandservice/in-process/grpc/grpc-fetch.ts—and the provider owns the whole transition in one place,
flagd-provider.ts: emitStale, arm atimer, escalate to
ErrorwhenretryGracePeriodexpires. Because the contract exists once, the tworesolvers cannot diverge. That is the shape Go needs, and it has been noted on go-sdk-contrib#939.
The backend stack
One Compose file serves both conformance suites in this repository, at
libs/shared/tck-backend/docker-compose.yaml: the unmodifiedflagd-testbedimage, servicebackend, publishing 8013, 8015, 8016 and 8080 with no host port bindings — the suite requiresdynamically mapped ports and discovers them after startup. Each suite asks the harness only for the
ports it needs, so the extra ones cost nothing.
It used to be one file per adoption. Four hand-maintained copies across the four languages had
drifted on the image tag: this repository was on
v3.10.1while Go, Java and Python sat onv3.8.0,so suites whose results are only comparable if they asked the same backend were asking two.
Resolved by aligning the other three up to
v3.10.1, the current release — this repository's tagis unchanged and the tallies below still stand on the image they were measured against.
Moving the file out of the project root opened a cache hole, which is fixed here. Both
tcktargets inherit
inputs: ["default", ...], where nx'sdefaultis{projectRoot}/**/*— so aCompose file outside the project would not have invalidated the cache, and changing the image tag
could have returned a cached green against the previous image without starting it. Both targets now
name
{workspaceRoot}/libs/shared/tck-backend/**/*explicitly. The accepted cost: nx's project graphcannot see a
join()path as an edge, sonx affectedwill not select these targets — harmless onlybecause nothing runs
nx affected --target=tck.Running it, and the tally
130 scenarios over the two resolvers: 116 pass, 10 skip, 4 fail. The four are the
large-integer-flagfixture gap thatflagd-testbed#392 fills — two per resolver,
the mandatory precision scenario and the
max-int32@variantsrow — not provider defects, andneither gets a deviation entry.
A red run with many more than four is this repo's own flakiness rather than the provider or the
backend. Two causes, both local and both worth knowing: Jest does not exit after a containerised run
here, so an orphaned worker keeps its testbed stack alive — two such processes from "completed"
runs were found still holding stacks, 11 and 22 minutes old — and under that contention the
launchpad's
POST /startanswers before the flags are evaluable. A control run at the pre-move tipfailed 43 of 130 on the same machine, worse than anything after, which is how we know the move did not
cause it.
ps aux | grep jestand kill them between runs.One deliberate omission: no
jest.retryTimeson these specs, though the neighbouring flagd e2e suitesuse 3. A conformance result that only holds on the third attempt is not a conformance result, so
contention surfaces as a failure rather than being retried away.
Worth filing against the JS flagd provider
service/in-process/file/file-fetch.tsnever receives or calls
disconnectCallback— itsconnectignores the reconnect callback — so aprovider using
offlineFlagSourcePathemits neitherPROVIDER_STALEnor a reconnectPROVIDER_READY. That may be intended for a file source, but it is undocumented and invisible.retryGracePeriod:configuration.tsdocuments@default 5;constants.tssetsDEFAULT_RETRY_GRACE_PERIOD = 10. Small, but it is the knob deciding how long aprovider stays STALE before claiming ERROR.