001 collection consistency audit

[RoddieKieley]

Summary

This was created for APPENG-4927, checking the consistency of the agentic-collections pack. It is very much DRAFT and needs review to see what is valuable information versus what might be valuable to actually commit to the project repository itself.

Pack(s) affected

• rh-sre
• rh-developer
• ocp-admin
• rh-virt
• rh-ai-engineer
• Other / repo-wide

Change type

• New skill
• New agent
• New pack
• Update existing skill / agent
• MCP server config (mcps.json)
• Docs / README
• CI / tooling

CLAUDE.md compliance

• Agents orchestrate skills; no direct MCP/tool calls in agents
• Skills are single-purpose task executors
• Skills encapsulate all tool access (MCP tools invoked only inside skills)
• Document consultation: file is read with the Read tool, then declared to the user
• No credentials hardcoded; env vars used via ${VAR} references
• Human-in-the-loop confirmation added for any destructive or critical operations

Validation

• make validate passes locally
• New/changed skills have valid YAML frontmatter (name, description)
• New/changed agents have valid YAML frontmatter (name, description)

[github-actions]

Gemini Code Review

1. Summary

This PR introduces a comprehensive "Collection Consistency Audit" framework to ensure metadata, versioning, style tokens, and documentation claims remain aligned across the repository. It adds Python-based audit scripts, updates CI workflows to enforce these checks, standardizes skill frontmatter (model and color fields) across several rh-sre skills, and introduces a canonical style token policy for the docs site.

2. Project Rules Compliance

• Frontmatter fields: The PR correctly updates several skills in rh-sre to include the mandatory model and color fields, and fixes a malformed frontmatter block in remediation/SKILL.md by removing an errant --- that was prematurely closing the YAML block. It also correctly updates the validators to enforce these fields.
• Build reminder: Since skills were modified (rh-sre/skills/*/SKILL.md), please ensure you run make validate locally to verify all changes pass the updated structure and design checks.

3. Code Quality Issues

The code quality is excellent. The audit framework is modular, well-structured, and relies on the standard library, avoiding unnecessary external dependencies.

One minor observation regarding repository hygiene:

• Committing Generated Reports: The PR commits reports/ci-violations.json, reports/consistency-audit.json, and reports/consistency-audit.md. Generated audit reports are typically ephemeral and shouldn't be tracked in version control, as they will cause constant merge conflicts and dirty working trees whenever metadata changes.

4. Suggestions

• Ignore Reports Directory: Consider removing the committed files in the reports/ directory from version control and adding the directory to .gitignore. The reports are generated dynamically in CI and locally, so tracking them in Git is unnecessary and prone to causing noise.

  # Add to .gitignore
  reports/

• Graceful File Handling: In scripts/consistency_audit_lib/checks/style_checks.py and icon_mapping_checks.py, the scripts assume files like docs/styles.css and docs/icons.json always exist. While true for the current repo state, adding a quick .exists() check before reading them (similar to how you handled docs/data.json) would make the audit script more robust against future structural changes.

5. Verdict

APPROVE — The PR is exceptionally well-designed, implements a highly valuable consistency framework, and correctly aligns existing skills with the project's frontmatter policies. Consider the suggestion regarding the reports/ directory before merging.

---

Automated review by Gemini 3.1 Pro | Rules: CLAUDE.md, SKILL_DESIGN_PRINCIPLES.md

[dmartinol]

Thank you for your first contribution!

Let's discuss offline how we can integrate these changes. Main concerns:

• Adoption of spec-kit which was not used before (at least document the install and howtos)
• The audit reported some issues that need to be addressed, which was the main goal of the task
• If we really want a continuous audit, shouldn't we integrate these new checks into the existing Make targets?

[dmartinol]

these plugin.json were deleted from the repo since when we adopted lola

[dmartinol]

on the other hand, we must have a collection.yaml file under the .catalog folder (but this is already checked by the collection_validate python script)

[dmartinol]

rh-support-engineer has been removed from the repo, where does this come from?

[dmartinol]

this is in the .gitignore list because it's only generated in the GH pages

# Generated Documentation
docs/data.json

[dmartinol]

not clear where the warns come from: we should add the explanation to the findings section.

[dmartinol]

probably CLM-001 is the only real issue, the other will disappear as we integrate the previous comments

[dmartinol]

this is not a read-only skill, so I think the color severity should be raised (maybe not red but at least yellow, WDYT?)

[dmartinol]

This is one thing I wanted to verify with this consistency audit: is it really worth to have model values different from inherit?

[dmartinol]

this can be completely removed IMO

[dmartinol]

should we restore this line?

[RoddieKieley]

* Adoption of `spec-kit` which was not used before (at least document the install and howtos)

The spec-kit is not itself included as it is in the .specify directory as well as skill entries that are in directories .claude/skills/speckit-*. That being said the result of using spec-kit which is the output found in the included specs directory. While that does create some noise for humans I have found it to be useful in providing context over time when you further use ai to do something and you are double digit specs into development within a project.

* The audit reported some issues that need to be addressed, which was the main goal of the task

Indeed there were some issues but unfortunately I took the title of APPENG-4927, "Consistency check of all collections (e.g. using same version, model, color, etc etc)" literally and took a pass to undo some of the changes.

That being said we should update the title to be action oriented, identify the changes required, and get them implemented.

* If we really want a continuous audit, shouldn't we integrate these new checks into the existing Make targets?

Those were some of the changes that were undone and not included. There were new make targets however I backed off to a check and providing an audit.

[dmartinol]

generally speaking, a great job in terms of problem analysis and 360-degree solution.
anyway, since we're planning to move the skill content to a different repo, or maybe even to separate repos, I'd avoid adding new cross-packs validations.
can we keep the Makefile and script changes for another run and just take the minimal changes to ensure consistent definition of the skills?
roughly, this means to keep all files apart from those under .cursor/, .github/, scripts/, specs/ and the Makefile.

[RoddieKieley]

anyway, since we're planning to move the skill content to a different repo, or maybe even to separate repos, I'd avoid adding new cross-packs validations.

ack

can we keep the Makefile and script changes for another run and just take the minimal changes to ensure consistent definition of the skills?

Yes, should be able to split up the work with the automated script updates separated from the actual content updates.

roughly, this means to keep all files apart from those under .cursor/, .github/, scripts/, specs/ and the Makefile.

Good guidance, will check.

[RoddieKieley]

@dmartinol Hopefully this is reduced to the valuable pieces that are able to be committed so that we can complete this PR #63 . I simply added another commit with the reduction, so the branch history should have the other content available when required.

[dmartinol]

Thank you! I will remove extra references to make validate-consistency-audit later