Skip to content

test(flagd): run the provider conformance suite against both resolvers - #1608

Draft
aepfli wants to merge 20 commits into
feat/provider-tckfrom
feat/provider-tck-flagd
Draft

aepfli wants to merge 20 commits into
feat/provider-tckfrom
feat/provider-tck-flagd

Conversation

@aepfli

@aepfli aepfli commented Aug 24, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1606. Base is feat/provider-tck, so this diff shows only the new work. Review
#1606 first — it describes the suite, the adoption API and the capability vocabulary, and none of
that is repeated here.

Part of #1607 and open-feature/spec#417.

The flagd provider under the conformance suite, both resolvers, in libs/providers/flagd/src/tck/.

Two spec files, because jest-cucumber accumulates step definitions in module state and two
runContainerizedProviderTck calls in one file would make every step ambiguous.

The directory is a sibling of src/e2e/, not a subdirectory of it. The suite used to be excluded
from the e2e run by one Jest config remembering to ignore a subdirectory of another; that line is
deleted, because a file is in src/tck/ or it is not. The unit config gained /src/tck/ in
testPathIgnorePatterns, which was not optional — without it nx test providers-flagd would collect
these specs and start Docker in the unit run.

What it declares

Both resolvers declare the same twelve: @events, @lifecycle, @stale, @configuration-change,
@object, @unavailable, @variants, @targeting, @disabled-flags, @standard-reasons,
@string-typing and @fully-typed-values.

The last two hold in both resolvers — RPC reads the type off the evaluation response, in-process off
the synced definition via flagd-core, and both refuse the string accessor for a non-string flag. All
four scenarios were verified as passing by name, not inferred from totals.

@numeric-coercion is not withheld here — it is refused by the implementation for every
JavaScript provider. typeof 10 and typeof 0.5 are both 'number', so requesting float-flag as
an integer is indistinguishable from requesting it as a float and no provider in this language can
satisfy the scenario. Naming it fails at configuration time with that reason, and the scenario is
skipped with a message saying the SDK cannot ask the question — not that this provider declined.

The finding: no Go-style asymmetry here

Go's flagd RPC resolver never emits PROVIDER_STALE while its in-process resolver does
(go-sdk-contrib#939), so the Go adoption
withholds @stale for RPC. JavaScript has no such split, and the reason is structural rather than
lucky.

Neither resolver decides anything about staleness. Both report a lost connection through a
disconnectCallback — service/grpc/grpc-service.ts and service/in-process/grpc/grpc-fetch.ts —
and the provider owns the whole transition in one place, flagd-provider.ts: emit Stale, arm a
timer, escalate to Error when retryGracePeriod expires. Because the contract exists once, the two
resolvers cannot diverge. That is the shape Go needs, and it has been noted on go-sdk-contrib#939.

The backend stack

One Compose file serves both conformance suites in this repository, at
libs/shared/tck-backend/docker-compose.yaml: the unmodified flagd-testbed image, service
backend, publishing 8013, 8015, 8016 and 8080 with no host port bindings — the suite requires
dynamically mapped ports and discovers them after startup. Each suite asks the harness only for the
ports it needs, so the extra ones cost nothing.

It used to be one file per adoption. Four hand-maintained copies across the four languages had
drifted on the image tag: this repository was on v3.10.1 while Go, Java and Python sat on v3.8.0,
so suites whose results are only comparable if they asked the same backend were asking two.
Resolved by aligning the other three up to v3.10.1, the current release — this repository's tag
is unchanged and the tallies below still stand on the image they were measured against.

Moving the file out of the project root opened a cache hole, which is fixed here. Both tck
targets inherit inputs: ["default", ...], where nx's default is {projectRoot}/**/* — so a
Compose file outside the project would not have invalidated the cache, and changing the image tag
could have returned a cached green against the previous image without starting it. Both targets now
name {workspaceRoot}/libs/shared/tck-backend/**/* explicitly. The accepted cost: nx's project graph
cannot see a join() path as an edge, so nx affected will not select these targets — harmless only
because nothing runs nx affected --target=tck.

Running it, and the tally

npx nx tck providers-flagd

130 scenarios over the two resolvers: 116 pass, 10 skip, 4 fail. The four are the
large-integer-flag fixture gap that
flagd-testbed#392 fills — two per resolver,
the mandatory precision scenario and the max-int32 @variants row — not provider defects, and
neither gets a deviation entry.

A red run with many more than four is this repo's own flakiness rather than the provider or the
backend. Two causes, both local and both worth knowing: Jest does not exit after a containerised run
here
, so an orphaned worker keeps its testbed stack alive — two such processes from "completed"
runs were found still holding stacks, 11 and 22 minutes old — and under that contention the
launchpad's POST /start answers before the flags are evaluable. A control run at the pre-move tip
failed 43 of 130 on the same machine, worse than anything after, which is how we know the move did not
cause it. ps aux | grep jest and kill them between runs.

One deliberate omission: no jest.retryTimes on these specs, though the neighbouring flagd e2e suites
use 3. A conformance result that only holds on the third attempt is not a conformance result, so
contention surfaces as a failure rather than being retried away.

Worth filing against the JS flagd provider

  1. The offline/file mode has no staleness contract at all. service/in-process/file/file-fetch.ts
    never receives or calls disconnectCallback — its connect ignores the reconnect callback — so a
    provider using offlineFlagSourcePath emits neither PROVIDER_STALE nor a reconnect
    PROVIDER_READY. That may be intended for a file source, but it is undocumented and invisible.
  2. Doc/code mismatch on retryGracePeriod: configuration.ts documents @default 5;
    constants.ts sets DEFAULT_RETRY_GRACE_PERIOD = 10. Small, but it is the knob deciding how long a
    provider stays STALE before claiming ERROR.

@coderabbitai

coderabbitai Bot commented Aug 24, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch from 2a0badd to 64ea1b5 Compare September 16, 2026 19:45
@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch 2 times, most recently from acd5568 to 6ec5d6a Compare October 2, 2026 08:57
@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch from 6bc783a to dfbb2b2 Compare October 8, 2026 08:20
@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch from dfbb2b2 to 7eeda1c Compare October 9, 2026 07:38
@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch from 7eeda1c to f58d3d8 Compare October 14, 2026 13:21
toddbaert pushed a commit to open-feature/spec that referenced this pull request Oct 14, 2026
> **Part of [#417](#417
This PR is the **assets** — the files four conformance implementations
execute. It stacks on
[#440](#440), which configures
the releases that give these assets a tag. The normative description of
them, Appendix F, is
[#439](#439), stacked on this.
[#425](#425) adds the
machine-readable report.
>
> This PR previously carried the appendix too. It was split so the
assets can land while the prose is still being reviewed.

## What

The artifacts a provider conformance suite runs against:

| path | what it is |
|---|---|
| `assets/provider-tck/gherkin/` | six feature files — evaluation,
errors, events, lifecycle, reason, metadata |
| `assets/provider-tck/flags/canonical-flags.json` | the flag set every
scenario assumes |
| `assets/provider-tck/openapi/control-api.yaml` | the HTTP surface a
backend under test must expose |
| `assets/provider-tck/go.mod`, `embed.go` | a nested Go module exposing
the artifacts as an `embed.FS` |
| `release-please-config.json`, `.release-please-manifest.json` |
registers the assets as a releasable component; the general setup is
#440 |

Marked **experimental**: the scenario set is a representative subset,
and the capability vocabulary may still change.

## Why this is split out

Four language implementations already run these files, and all four are
currently pinned to an **unmerged branch tip**. Every rebase here
invalidates them — it has happened three times in two weeks. Nothing
about those pins becomes real until the assets are on `main` with a tag,
so landing the executable half first is worth more than landing it
together with prose that will take longer to review.

Appendix F is the same work, reviewed on its own terms.

## Why a Go module is in here

A Go module is distributed as a zip of the VCS tree, in which a git
submodule is only a gitlink and its files are absent. The other three
languages build from a working tree and keep using the submodule, but a
Go suite would have to vendor a copy of every artifact and police it
against drift. `embed.go` removes the copy: a consumer pins a revision,
the checksum database makes it immutable, and the embedded bytes are the
same bytes every other language reads out of the submodule. `go.mod` and
`embed.go` are inert for every non-Go consumer. Precedent:
`open-feature/flagd-schemas`.

The release config tags the assets as their own component,
`specification/assets/provider-tck/vX.Y.Z` — the shape `go get` needs.
Without it a Go consumer can only name a pseudo-version.

## Releasing the assets

The general release-please setup is
[#440](#440), which this stacks
on. What this PR adds is the part that describes *these* assets: a
second package, so they are tagged
`specification/assets/provider-tck/vX.Y.Z`.

That tag shape is what `go get` needs. Without it a Go consumer can only
name a pseudo-version, which is what the four conformance
implementations have had to do while this branch was unmerged, and it
has cost several rounds of re-pinning.

The specification package sets `exclude-paths` for this directory, so an
asset-only change no longer cuts a specification release, and
`separate-pull-requests` keeps the two from sharing one release pull
request.

## The capability vocabulary lives in the README, for now

A scenario exercising an optional part of the contract carries a tag; a
provider declares what it supports, and a scenario gated on an
undeclared tag is reported **skipped, with the reason** — never as
passed. A suite that quietly goes green on scenarios it did not run is
worse than no suite at all.

That rule and the tag table are in `assets/provider-tck/README.md` so
the tags in the feature files are not left undefined while the appendix
is in review. #439 moves them into the appendix and leaves a pointer
behind, so they end up with one home rather than two that can disagree.

Worth stating plainly: **withholding a tag is not an admission of a
defect.** `@numeric-coercion` borrows its rule from flagd's coercion
ADR, and `@string-typing` / `@fully-typed-values` sit on a question the
specification does not answer at all —
[#430](#430) and
[#433](#433).

## Decisions worth reviewing

- **`assets/provider-tck/` rather than beside the SDK Gherkin**, because
`assets/gherkin/evaluation.feature` already exists and tests an SDK, not
a provider. This answers #417's Q1 — the subdirectory is forced rather
than chosen.
- **`.gitattributes` normalises these to LF.** Four languages consume
them byte for byte and copy them verbatim into published artifacts, so a
Windows checkout must not package differently from a Linux one.
- **`missing-flag` must not exist** in the flag set — its absence is
what the `FLAG_NOT_FOUND` scenario tests. The other four load-bearing
properties of the flag set are in the README; each one, if seeded wrong,
turns a scenario green or red for the wrong reason in every language at
once.
- **No container restarts.** Outages are produced through the control
API, because orchestrators reassign host ports on restart and every
provider pointed at the old one then fails looking flaky.

## Implementations

All four execute exactly these files, against flagd (both resolvers),
OFREP and Flagsmith:

| | suite | flagd | OFREP | Flagsmith |
|---|---|---|---|---|
| Java |
[#1830](open-feature/java-sdk-contrib#1830) |
[#1847](open-feature/java-sdk-contrib#1847) |
[#1840](open-feature/java-sdk-contrib#1840) |
[#1849](open-feature/java-sdk-contrib#1849) |
| Go | [#940](open-feature/go-sdk-contrib#940) |
[#941](open-feature/go-sdk-contrib#941) |
[#942](open-feature/go-sdk-contrib#942) |
[#959](open-feature/go-sdk-contrib#959) |
| JavaScript |
[#1606](open-feature/js-sdk-contrib#1606) |
[#1608](open-feature/js-sdk-contrib#1608) |
[#1612](open-feature/js-sdk-contrib#1612) |
[#1623](open-feature/js-sdk-contrib#1623) |
| Python |
[#409](open-feature/python-sdk-contrib#409) |
[#411](open-feature/python-sdk-contrib#411) |
[#414](open-feature/python-sdk-contrib#414) | —
|

## What running them has found

- **flagd silently narrows a float flag to an integer**, Go and Java,
both resolvers, no error code —
[flagd#1996](open-feature/flagd#1996)
- **flagd reported ready before flags were evaluable**, on 100% of
starts —
[flagd#2047](open-feature/flagd#2047), fixed
in flagd v0.17.0
- **The Java SDK's `MultiProvider` swallows child provider events** —
[java-sdk#1882](open-feature/java-sdk#1882)
- **The Python flagd provider widens a boolean to a float** —
[python-sdk-contrib#417](open-feature/python-sdk-contrib#417)
- **The JavaScript Flagsmith provider stringifies unconditionally**
where its other paths correctly report a mismatch — visible only because
three languages ran identical scenarios against one backend
- two specification gaps, raised rather than decided here:
[#430](#430),
[#433](#433)

---------

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli added 20 commits October 14, 2026 16:29
flagd resolves flags two quite different ways, and they are separately
conformant: any difference between the two results is a difference an
application would see when it switches resolver. Two spec files rather than
one, because jest-cucumber accumulates step definitions in module state and two
runProviderTck calls in one file would make every step ambiguous.

The TCK drives flagd-testbed's launchpad through the standardised control API,
which the launchpad already implements, so neither the testbed nor the existing
e2e suites in this directory change. The stack is started once per suite and
never restarted; isolation comes from the control API instead.

Both resolvers declare Events, Stale, ConfigurationChange, Object and
UnavailableInit. The Stale declaration is the notable one: in Go the RPC
resolver never emits PROVIDER_STALE while in-process does
(open-feature/go-sdk-contrib#939), but here both report a lost connection
through the same disconnectCallback seam -- grpc-service.ts:274 and
grpc-fetch.ts:197 -- and the single handler behind it, flagd-provider.ts:130-148,
emits PROVIDER_STALE before escalating to PROVIDER_ERROR. The contract is
implemented once, in the provider, so it cannot differ between resolvers.

StrictNumericTyping is left undeclared, as it must be for every JavaScript
provider: the language has no integer type, so the scenario is unsatisfiable by
construction rather than by defect. Targeting and Caching are undeclared
because no scenario carries their tags yet.

Closes #1607

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Comments only. Both suites already omit the capability, and for a reason the
rename does not change: JavaScript has no integer type, so no provider written in
it can answer either half of the coercion contract.

Worth noting what that means here as against the other languages. Go and Java
withhold this capability for flagd because the provider narrows 0.5 to 0 with no
error -- a defect, tracked as open-feature/flagd#1996. These suites omit it
because the question cannot be asked at all. Same omission, unrelated causes, and
the conformance report distinguishes them: a JavaScript report carries it under
`declaration.notApplicable` with the language as the reason, rather than leaving
it merely undeclared.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The e2e target carried `"parallel": false`, which the `@nx/jest:jest`
executor has no such option for -- `parallel` belongs to
`nx:run-commands`, which the neighbouring `generate` and `pullSpec`
targets do use. Unknown executor options are dropped silently, so the
declaration read as serial and behaved as the default: jest spread the
four e2e spec files across workers.

`runInBand` is the option that means what this one was trying to say.

Serial execution matters here beyond speed, because each spec file brings
up its own flagd-testbed Compose stack. Four at once is four sets of
containers competing for the same host resources, and -- until the
following commit -- for the same flags directory.

Worth knowing while reading this: `--testPathPattern` no longer filters
either. jest 30 renamed it to `--testPathPatterns`, and the executor
schema followed, so passing the old singular name selects nothing and
silently runs everything. Nothing in the repository passes it; it is a
trap for whoever next tries to run one spec from the command line. The
target's own `--testFile` is unaffected.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Both resolvers left `@lifecycle` undeclared, which skipped the whole of
`lifecycle.feature` -- six scenarios that Java runs against this same
provider. Nothing about flagd justified the omission: `initialize` awaits
`connect`, which awaits `waitForReady` on the gRPC channel
(grpc-service.ts:194-202) and rejects when the backend is unreachable, so
both terminal outcomes are observable rather than synthesised by the SDK.
That is exactly what the tag asserts, and it is the harder claim of the
two -- `@events` alone can be satisfied vacuously, because every SDK
synthesises PROVIDER_READY for a provider that does nothing on startup.

In-process earns it more plainly still: it syncs the whole ruleset before
reporting ready.

`@reinitialization` is deliberately NOT declared, and this is the half
worth reading. Requirement 2.5.2 says a provider SHOULD revert to its
uninitialized state after shutdown, and its supporting text says "some
providers MAY allow reinitialization from this state" -- permitted, not
required. flagd declines it: `onClose` delegates to `disconnect`, which
calls `close()` on the client (grpc-service.ts:140-143 for RPC,
grpc-fetch.ts:95-99 for in-process) and nothing constructs a new one.

So the reuse scenario is reported as skipped, and NO known deviation is
recorded against it. An earlier pass at this recorded one, which would have
published a specification-sanctioned choice as a flagd defect -- the
scenario was untagged at the time, and the reflex was to treat a failing
scenario as evidence of a bug rather than to go and read the requirement.
The rule this cost: find the numbered requirement before recording a
deviation or withholding a capability, because a failing scenario is not on
its own evidence of anything.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Both resolvers now declare eight capabilities rather than six, and both
declarations are made on a run rather than on a reading of the provider.

@targeting is the one worth having. flagd-testbed already serves
targeting-key-flag -- it was added upstream in February 2024 and the testbed's
own targeting.feature uses the same flag, variants and uuid -- so nothing had to
be seeded and no image bump was needed. All three scenarios pass on both
resolvers: a matching context resolves "hit", a non-matching one resolves
"miss", and no context at all resolves "miss" without erroring. That last one is
not a formality for this provider, which has two quite separate paths to a
targeting decision: RPC asks flagd to evaluate the rule, in-process syncs the
ruleset and evaluates it locally in flagd-core. The two agreeing is the finding,
the same way it is for staleness.

@Variants is the honest half. Seven of the eight gated rows pass on both
resolvers. The eighth asks for large-integer-flag, which this testbed image does
not serve at all (flagd-testbed#392) -- the same missing flag that already fails
the mandatory 2^31 - 1 scenario here, and the reason LargeIntegers stays
undeclared. No knownDeviation is recorded for it: a deviation claims the
provider gets something wrong, and what is missing is a flag in the backend's
set. Withdrawing @Variants instead would claim flagd has no variant concept,
which the other seven rows disprove.

@caching stays undeclared and is now the only reserved name left, so the
comments that said "Targeting and Caching because no scenario carries their tags
yet" no longer described anything.

Measured over 104 scenario instances, 52 per resolver:
4 failed, 10 skipped, 90 passed. Both failures per resolver are the missing
large-integer-flag.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Nothing had to be seeded. The four flags are flagd-testbed's own, from
flags/disabled-flags.json, and the launchpad already serves them under the
`default` configuration, which is why Appendix F took its names and values from
there.

Both resolvers pass all four rows, and they reach the answer by different
routes. In-process syncs the ruleset and evaluates locally, so the caller's
default is in hand when the state is read: flagd-core returns it directly
(flagd-core.ts:176-181). RPC is the interesting half, because the evaluation is
remote and the request never carried a default -- flagd answers with the type's
zero value, `{"value":"0","reason":"DISABLED","variant":""}` for
disabled-integer-flag -- and the provider substitutes the caller's default
client-side when the variant is empty and the reason is DEFAULT or DISABLED
(grpc-service.ts:306-312). Without that substitution three of the four rows
would have failed on the value alone, which is what makes the declaration a
statement about this provider rather than about flagd.

The two resolvers agreeing is the same finding as for @Variants and @targeting:
an application switching resolver sees the same four values.

Measured, not read. Each suite is 56 scenarios, 5 skipped, 49 passed, 2 failed
-- and both failures are the pre-existing testbed gap: `large-integer-flag` is
absent from the image (flagd-testbed#392), which fails the mandatory 2^31 - 1
scenario and the one @Variants row that asks for the same flag. Neither is new
and neither is the provider's.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The adoption hand-rolled its container lifecycle: `FlagdComposeContainer.build()`
at module scope, a `beforeAll` to start it, a guarded `afterAll` to stop it, a
port lookup per resolver, and an `HttpControl` built from a thunk over the
launchpad's mapped port. Every one of those is now the suite's job, so all of it
goes: `runContainerizedProviderTck` is handed a Compose file, the two ports the
two resolvers connect to, and a factory that reads the endpoint it discovers.

The Compose file is this adoption's own, at `src/e2e/tck/docker-compose.yaml`,
and it is deliberately not `test-harness/docker-compose.yaml` next door. Three
differences, each of which is the point:

  - **No `/flags` bind mount.** The launchpad writes the configuration it is
    asked for into the container's own `/flags` from the rawflags baked into the
    testbed image, so the mount buys nothing and costs isolation: the
    neighbouring e2e suites share one host directory through it, and with two
    containers up each rewrites the other's flags. That is why this branch
    previously grew a per-instance temp directory -- and why that commit is gone
    rather than kept. Removing the shared resource beats scheduling around it,
    and a conformance suite whose failures read as provider defects
    (`error-code was 'FLAG_NOT_FOUND', expected 'TYPE_MISMATCH'` on a flag that
    exists) is the worst possible thing to leave to a scheduling guarantee.
    `flagdComposeContainer.ts` is therefore untouched by this branch again: the
    pre-existing e2e suites still import it, so it cannot be deleted, but the
    conformance suites no longer go near it.
  - **No environment variables.** The image tag is pinned in the file rather
    than read from `test-harness/version.txt`. Every capability the two suites
    declare rests on a run against v3.10.1 specifically, so a bump is a
    deliberate act with a result to record, not something a dependency update
    does quietly.
  - **No host port bindings.** The suite requires dynamically mapped ports and
    discovers them after startup.

One Compose file for both suites, as in Java: 8013 is RPC evaluation and 8015
the in-process sync stream, both served by the one testbed service, so each
suite declares only the port its resolver uses.

`retryGracePeriod` stops being a per-suite knob. Both resolvers passed 30, and
they have to: the grace period is emitted from the shared provider layer rather
than from either transport, which is the same reason both declare `@stale`.

Measured rather than asserted, on v3.10.1, under node 24:

  - flagd-rpc: 49 passed, 5 skipped, 2 failed.
  - flagd-in-process: 49 passed, 5 skipped, 2 failed.

Identical, which is the finding the two suites exist to produce. Both failures
are the two already recorded against the testbed image rather than against the
provider -- the mandatory 32-bit precision scenario and the `max-int32` row of
the `@variants` outline, both wanting a `large-integer-flag` that v3.10.1 does
not serve (open-feature/flagd-testbed#392). The `@stale`, `@lifecycle` and
`@unavailable` scenarios all pass, which is what establishes that the suite's
await-ready on the control API is sufficient without the post-command settle
delay that was dropped.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The suites are Docker-gated and run by hand before merge, not on every
push. They were not: this repository's CI has an `e2e` job running
`npm run e2e`, which is `nx run-many --all --target=e2e`, and
libs/providers/flagd/src/e2e/jest.config.ts has no testMatch, so Jest's
default swept up anything named *.spec.ts under src/e2e -- including two
conformance suites that pull a backend image and, against the pinned
testbed, fail two scenarios for a missing flag (flagd-testbed#392). CI
would have gone red on this branch for a reason that is not about this
provider.

So the boundary is now a directory rather than a filename. The suites,
their shared body and their Compose file live in src/e2e/tck with a Jest
config of their own and an nx target of their own:

  npx nx tck providers-flagd

src/e2e/jest.config.ts ignores that directory and loses the
@openfeature/tck path mapping it no longer needs, so this branch's
footprint on the pre-existing e2e config is one ignore pattern. The
suites keep runInBand, and the target drops the flagd-core
pullTestHarness dependency that `e2e` needs, because the image tag is
pinned in the Compose file instead of read from the submodule.

The provider README says all of this, because an exclusion nobody wrote
down is indistinguishable from an oversight -- which is exactly how this
went unnoticed.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The reasoning now lives in Appendix F's "Running the suite in CI", promoted there
because four READMEs is where this kind of decision had already drifted into
several answers. What stays here is the mechanism, which is this repository's
alone: the suites live in src/e2e/tck/ behind a Jest project of their own,
src/e2e/jest.config.ts ignores that directory, and `npx nx tck providers-flagd`
is the only way in.

The first mistake Appendix F names was made here, so it is named here too: the
suites once sat in src/e2e/tests/, where the pre-existing e2e target's default
testMatch swept them up, and that target is a CI job in this repository. The
directory boundary exists because of it.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… runs

The `tck` target depended on `generate`, `pullTestHarness` and
`buildValidators` but not on `tck:pullSpec`, so nothing in the graph checked the
`open-feature/spec` submodule out. The library's own `test` target has that
dependency; this one did not, and the difference is invisible until the pin
moves.

Found by it happening. Rebasing this branch onto a base that re-pinned the
submodule moved the *gitlink* and left the submodule's working tree on the
previous revision, because that is all a gitlink is. The suite then ran the old
feature files: a new capability was declared, the feature file carrying it was
not on disk, and the run went green having tested nine scenarios fewer than it
claimed -- with no count to compare against, because an out-of-date asset set is
internally consistent with itself. The same thing bit the Python suite on the
same pin.

A green run that tested less than it claimed is the one failure mode a
conformance suite must not have, so this is a dependency in the graph rather
than a line in the README. The README says why it is there.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
flagd reports the standard resolution vocabulary with the standard meanings, so
both suites say so and reason.feature checks the claim. All nine scenarios run
in each -- the tag composes with @targeting and @disabled-flags and both are
already declared here -- and all nine pass in both: STATIC for the four
rule-less flags, ERROR for the unknown flag and the type mismatch,
TARGETING_MATCH for the matching key and DEFAULT for the miss, DISABLED for the
disabled flag.

STATIC for a rule-less flag is the row the specification genuinely leaves open
-- types.md types DEFAULT as "no dynamic evaluation occurred or dynamic
evaluation yielded no result", which a rule-less flag satisfies as readily -- so
it is measured rather than assumed. flagd answers STATIC and DEFAULT in exactly
the two places Appendix F separates them.

The two resolvers agreeing is the finding rather than a copy: RPC passes the
reason through from flagd's evaluation response, in-process gets it from
flagd-core evaluating the synced ruleset, and an application switching resolver
sees the same reason as well as the same value. That is what makes a reason
worth building telemetry on.

`npx nx tck providers-flagd`: 130 scenarios, 116 passed, 10 skipped, 4 failed --
eighteen more than before the pin, nine per resolver, none of them skipped. The
four failures are unchanged and pre-existing: the testbed image does not serve
large-integer-flag (flagd-testbed#392), which fails the mandatory 32-bit
precision scenario and one @Variants row in each resolver.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Both suites carried a comment explaining that JavaScript has no integer type
and so cannot be asked @numeric-coercion. The library refuses the capability
centrally now, and gates its scenarios whatever a suite declares, so the comment
documents a decision these files no longer make -- and a per-adoption
restatement of a language property is the thing the central refusal replaces.
Neither suite ever named the capability, so nothing about what runs changes.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ot a hedge

The suite already left the capability undeclared and recorded no deviation,
because the missing flag is the testbed's gap rather than the provider's. What it
did not say is which rule makes that the right shape. Appendix F now states it:
the unit of a declaration decision is the scenario, not the tag, and
@large-integers gates exactly one -- so a backend without its flag leaves nothing
about the capability establishable and withholding is correct. @numeric-coercion
would go the other way for the same reason, gating three, which is why the two
absences are formally different things and should not be read as one.

The second consequence is the one worth writing down here: an omission for a
backend gap is temporary in a way the others in this file are not. Caching is
reserved until the specification writes a scenario, Reinitialization is a
sanctioned choice this provider makes -- neither expires on its own.
@large-integers goes away when the testbed serves the flag, so the note names
what closes it.

Comment only. No capability list, no deviation and no assertion changes.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
`nx.json` sets `passWithNoTests: true` in the `@nx/jest:jest` target defaults,
which the `tck` target inherits like any other Jest target. So the one command
documented for running the conformance suite exits 0 having collected zero
scenarios if the spec files move, are renamed, or fall under an ignore pattern
-- green because nothing ran, which is the failure this suite exists to make
impossible for a provider and had wired into its own entry point.

The default is right for the rest of the workspace, where a project may
legitimately have no tests yet. It is wrong here: this target has exactly one
Jest project and it is the conformance suite. Overriding it in the target's own
options is the narrowest fix.

`cache: true` is inherited too, so a previous result can be replayed rather than
re-run. That is left alone -- the inputs include the spec assets, and a replay of
a genuine pass is not the same hazard -- but it is worth knowing when someone
reports that the suite "passed".

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ide it

`src/e2e/tck/` becomes `src/tck/`, a sibling of `src/e2e/` rather than a
child of it.

Nesting said the conformance suite is a kind of e2e test, which is what the
separate `tck` target exists to deny. The two answer different questions --
the e2e suite tests this provider against flagd's own harness, the
conformance suite tests it against the OpenFeature provider contract -- and
they mean different things by failure: an e2e suite is expected green, while
a conformance suite fails scenarios by design wherever a `knownDeviation`
is declared.

Selection stops being a convention as a result. It used to depend on
`src/e2e/jest.config.ts` remembering `testPathIgnorePatterns:
['<rootDir>/tck/']`; now a file is in `src/tck/` or it is not. That ignore
pattern is deleted and the unit config ignores `/src/tck/` instead, so
neither `nx test providers-flagd` nor `nx e2e providers-flagd` can collect
these suites. `passWithNoTests: false` on the `tck` target is what makes a
wrong glob after the move fail loudly rather than go green having collected
nothing; it was set for that reason and it is load-bearing here.

`tckSuite.ts` becomes `suite.ts`: in a directory called `tck` the prefix
said it twice. The two spec files already dropped their `tck-` prefix when
they moved out of `src/e2e/tests/`, where it had been the selector.

The move costs one thing, which is the interesting part. `tsconfig.lib.json`
excluded `./src/e2e` wholesale, so it covered the adoption by accident; with
the suite outside that directory, `suite.ts` fell into the library build and
the build failed with thirteen errors, none of them in the adoption's own
code. A library build compiles what gets published, so it sets `types: []`
and `module: ES6`, while the conformance harness is a test library whose
entry point calls `describe`, `test` and `jest`. `./src/tck` is therefore
excluded explicitly, and `tsconfig.spec.json` includes it explicitly --
which, unlike the old `./src/e2e` entry, scopes a future
`tsc --noEmit -p tsconfig.spec.json` check to the conformance suite alone
rather than to the pre-existing e2e files beside it.

Same scenarios, same results, a different directory: `npx nx tck
providers-flagd` reports 130 scenarios, 116 passed, 10 skipped, 4 failed,
unchanged.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The comment explained that the conformance suites are not in this directory. A
note in one suite's configuration about a suite that is somewhere else is the
thing the move was meant to remove, not something the move should add -- the
directory boundary is the explanation.

This leaves the branch touching nothing under src/e2e at all, which is the
correct diff for a change that moved files out of it.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… is owned

Two things, because both are the same mistake seen from different angles: a
fact restated in a second place drifts from the first.

The Compose file moves to libs/shared/tck-backend/docker-compose.yaml and is
shared with the OFREP adoption. Stripped of comments the two files it
replaces were the same six lines of YAML with a different port list, and a
per-project copy is how two suites come to ask the same questions of two
different backends -- which is exactly what happened across the four
languages, this one having drifted to flagd-testbed v3.10.1 while Go, Java
and Python sit on v3.8.0. The shared file exposes 8013, 8015, 8016 and 8080;
each suite asks the harness only for the ports it uses, and the rest cost
nothing because there are no host bindings. Both tck targets name the file in
their inputs, since it is outside the project root and nx's default
{projectRoot}/**/* would not invalidate a cached run when it changed.

The file itself keeps only what is a trap in this file: that it is not the
flagd-core test-harness Compose file the e2e suites use, which bind-mounts
${FLAGS_DIR} into a shared host directory and runs an Envoy sidecar. The
testbed's missing flags and the dynamic-port convention are
open-feature/flagd-testbed#392's and Appendix F's respectively, and are
linked rather than re-explained.

The suite files keep every per-capability reason with its evidence -- the
DISABLED wire response and the grpc-service.ts:306-312 substitution that
earns @disabled-flags for a resolver that evaluates remotely, the single
disconnectCallback seam that makes @Stale identical in both resolvers, the
testbed gap that makes @large-integers temporary rather than permanent -- and
lose the citations of what Appendix F's rules are, the comparisons with Go's
two resolvers, and the directory-layout argument that the provider README
already makes.

Comments, Markdown and one file move; no behaviour change and no count moves.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Appendix F moved four scenarios out of the mandatory type-mismatch matrix and
behind a capability tag, on the grounds that a backend storing flag values as
strings satisfies the string accessor for every flag and has no mismatch to
report. That tag has since been split in two: @string-typing keeps the boolean
and integer rows a partially typed store can still answer, and
@fully-typed-values holds the float and structured ones. flagd is fully typed,
so both resolvers declare both tags and all four scenarios keep running.

Measured rather than assumed, and by two different routes. RPC reads the type
off flagd's evaluation response; in-process reads it off the synced flag
definition and evaluates in flagd-core. Both refuse the string accessor for all
four, named rather than inferred from a total:

  A non-string flag is not returned as its string representation  [boolean-flag]
  A non-string flag is not returned as its string representation  [integer-flag]
  A float flag is not returned as its string representation
  A structured flag is not returned as its JSON text

Eight passes across the two resolvers, and each run's declaration line carries
@fully-typed-values and @string-typing both.

Both tags had to be named explicitly, because this adoption declares by an
allow-list rather than by everything-except. A capability registered upstream is
undeclared here until it is written down, so taking the split without adding the
new half would have turned the float and structured scenarios from passes into
skips -- a quieter regression than a failure, because the results would simply
stop asserting a property flagd demonstrably has, and nothing would say so.

Leaving either tag undeclared would have been the cheaper change and the wrong
one. Withholding is for a backend that cannot be asked the question, not for one
that answers it.

Totals are unchanged at 65 scenarios per resolver -- errors.feature still holds
19, four of them now gated by two tags instead of one -- with the same five
skips and the same two pre-existing failures in each, both from flagd-testbed
not serving large-integer-flag (open-feature/flagd-testbed#392).

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
v3.10.3 is the first testbed release carrying flagd v0.17.0, which includes
"wait for flag state before reporting ready" (open-feature/flagd#2050, closing
open-feature/flagd#2047). That defect returned 200 from /readyz while the flag
store was still empty, on 100% of starts, and is the cause behind the
intermittent FLAG_NOT_FOUND runs recorded in these suites.

v3.10.2 looked current but shipped flagd v0.16.3; the base-image bump landed
after that tag was cut. This also picks up the TLS SANs fix (flagd-testbed#421).

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The adoption restated facts other repositories own: specification rationale for
optional capabilities, what the TCK library's vocabulary and gates mean, what
flagd-testbed seeds, and the review history of these files.

Kept each capability's own reasoning, the measured tallies, and the notes that
stop a timeout or an ignore pattern being simplified back into a bug.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch from f58d3d8 to d6c15fc Compare October 14, 2026 14:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants