Clarify validation layers and consider “Validate at every level possible”

[jmchilton]

Context

While reviewing the three Build one pages, the treatment of an “external check” turned out to be both more pervasive and less clearly modeled than expected:

• The Model places the external check outside the four structural parts.
• Guiding Principles folds it into “Deterministic Tools Do Deterministic Work.”
• Anatomy of an Instance treats the domain’s external check as a major extension-surface category.
• Setting up a Foundry makes building the external check a discrete implementation phase.
• One Worked Example: the Astro Stack explains why it has no fixed implementation row.

This may be pointing at a stronger and more general principle, tentatively:

Validate at every level possible.

This issue is deliberately a design question, not a settled rewrite.

Three validation levels to distinguish

1. Source validation

Does the knowledge base satisfy its authored contracts?

Examples:

• frontmatter and kind schemas
• resolvable references
• controlled vocabulary and registry membership
• license coherence
• generated catalogs and other self-documentation checks

2. Cast verification

Was the artifact built faithfully and reproducibly from the declared source?

Examples:

• deterministic output
• resolved reference and placement checks
• source and destination hashes
• provenance completeness
• drift detection and reproducible recasting

3. Domain evaluation

Does work performed with the artifact deserve trust in its domain?

Examples:

• Galaxy’s deterministic workflow validator
• statistical genomics’ empirical referee
• human review, proof tools, simulation, calibration, or negative controls

The first two appear intrinsic to the Foundry machinery. The third is domain-specific and may not admit one universal mechanism—but it may still express an important discipline: use the strongest independent validation the domain makes possible.

Questions to resolve

• Should “Validate at every level possible” become a guiding principle?
• Which validation levels are requirements of the pattern, and which are instance-specific?
• Is domain evaluation part of the extension surface, or simply downstream domain machinery?
• How should we preserve “doing does not certify itself” without implying that every domain has a deterministic oracle?
• Is “external check” still useful terminology, or does it obscure the three distinct concerns?
• Where does validation end and evaluation begin?

Likely documentation impact

Review and align:

• content/pattern/the-model.md
• content/pattern/guiding-principles.md
• content/pattern/principles-in-action.md
• content/pattern/anatomy-of-an-instance.md
• content/pattern/setting-up-a-foundry.md
• content/pattern/standing-up-a-foundry.md
• relevant case pages, especially the discussion of the pattern’s durable assets

Possible direction—not yet a decision

• Treat source validation and cast verification as shared Foundry responsibilities.
• Treat domain evaluation as a domain-owned responsibility whose mechanism and attainable strength vary.
• Frame the principle as applying validation at every boundary where a meaningful independent check exists.
• Show the three levels explicitly instead of using “external check” as one catch-all term.

Done when

• The three levels have clear names and boundaries.
• The Model and Guiding Principles agree about which guarantees belong to the pattern.
• The Build one pages distinguish shared validation machinery from domain-specific evaluation.
• Galaxy and statistical-genomics examples demonstrate the distinction without defining the abstraction by either implementation.