A write up of file structure for company-wide CUE validation schemas

[Amoenus]

Summary

This issue proposes the addition of a detailed write-up or documentation to the cue-by-example repository, focusing on the file structure for company-wide CUE validation schemas.

Given the extensive amount of configurations in organizations, having a clear understanding of how validation schemas (and potentially data files for generation) are structured would be immensely beneficial.

Detailed Description

Organizations begin adopting CUE for configuration and validation tasks. However, the lack of clear documentation on the recommended or standard file structure for validation schemas can lead to inconsistencies and inefficiencies. This documentation should ideally cover:

1. Best Practices for File Organization: Guidelines on how to structure validation schema files in a project, considering scalability and ease of maintenance.

2. Example Structures and Use Cases: Providing diverse templates and examples, including but not limited to:

   • Azure pipelines validation schemas.
   • Crossplane XRDs (Composite Resource Definitions) validation schemas.
   • Kubernetes configuration validation schemas.
   • Dockerfile and container configuration validation.
   • Terraform infrastructure as code validation schemas.
   • Ansible playbook validation schemas.
   • GitHub Actions workflow validation schemas.
   • Jenkins pipeline configuration validation.
   • Validation schemas for cloud services configurations (AWS, GCP, Azure).
   • Network configuration validation schemas for devices or services.
   • Database schema validation for SQL and NoSQL databases.
   • Application configuration files like JSON, YAML, or XML validation.
   • Environment variable and .env file validation schemas.
   • Validation for mobile app configuration files (iOS/Android).
   • Web server configuration validation (Nginx, Apache).

3. Integration Recommendations: Tips on integrating these schemas with existing CI/CD pipelines, focusing on automation and consistency checks.

[Amoenus]

Some exploration led me to this preliminary structure in a project. I am opting to put cue stuff in .cue folder.

.
├── .cue
│   ├── schemas
│   │   ├── ipAddress.cue
│   │   ├── dnsARecord.cue (uses `ipAddress.cue`)
│   │   ├── ingress.cue
│   ├── defaults
│   │   ├── default-resource1.cue
│   │   ├── default-resource2.cue
│   └── environments
│       ├── dev
│       │   ├── dev-resource1.cue
│       │   ├── dev-resource2.cue
│       └── prod
│           ├── prod-resource1.cue
│           ├── prod-resource2.cue
├── .devops
│   └── azure-pipelines.yml
├── .gitignore
├── README.md
├── renovate.json
└── example-app
    ├── .helmignore
    ├── Chart.lock
    ├── Chart.yaml
    ├── templates
    │   ├── template-resource1.yml (classic `helm` chart with variables)
    │   └── template-resource2.yml (classic `helm` chart with variables)
    └── values.yaml (Generated with `cue export`)

Baby-steps obviously, but would like to gather people's thoughts on this approach and how it could be expanded in the context and scope of the issue above

[Amoenus]

Expanding upon this idea to a more generic file structure:

Company-wide CUE Schemas Repository

company-cue-schemas/
├── schemas/
│   ├── core/
│   │   ├── dnsARecord/
│   │   │   ├── defaults.cue
│   │   │   └── dnsARecord.schema.cue
│   │   ├── ipAddress/
│   │   │   ├── defaults.cue
│   │   │   └── ipAddress.schema.cue
│   │   ├── keda/
│   │   │   ├── defaults.cue
│   │   │   └── keda.schema.cue
│   │   ├── kyverno/
│   │   │   ├── defaults.cue
│   │   │   └── kyverno.schema.cue
│   │   ├── cilium/
│   │   │  ├── defaults.cue
│   │   │  └── cilium.schema.cue
│   │   └── crossplaneCluster/
│   │       ├── defaults.cue
│   │       └── crossplaneCluster.schema.cue
│   ├── infrastructure/
│   │   └── clusterClaim/
│   │       ├── defaults.cue
│   │       └── clusterClaim.schema.cue
│   ├── pipelines/
│   │   ├── build/
│   │   │   ├── defaults.cue
│   │   │   └── build.schema.cue
│   │   ├── deploy/
│   │   │   ├── defaults.cue
│   │   │   └── deploy.schema.cue
│   │   └── shared/
│   │        ├── defaults.cue
│   │        └── shared.schema.cue
│   ├── composites/
│   │   ├── clusterClaim/
│   │   │   ├── defaults.cue
│   │   │   └── clusterClaim.schema.cue (imports schemas from company-cue-schemas/schemas/core/)
│   │   ├── frontend/
│   │   │   ├── defaults.cue
│   │   │   └── frontend.schema.cue (imports schemas from company-cue-schemas/schemas/core/)
│   │   └── backend/
│   │        ├── defaults.cue
│   │        └── backend.schema.cue (imports schemas from company-cue-schemas/schemas/core/)
├── generated/
│   └── composites/
│       ├── frontend.xrd.yaml (Generated Composite for frontend)
│       ├── backend.xrd.yaml (Generated Composite for backend)
│       └── crossplaneCluster.xrd.yaml (Generated Composite for crossplaneCluster)
└── README.md

App Infrastructure Repository (infra-repo):

infra-repo/
├── .cue/
│   ├── dev/
│   │   ├── frontend.cue (imports schemas from company-cue-schemas/schemas/infrastructure/frontend/)
│   │   ├── backend.cue (imports schemas from company-cue-schemas/schemas/infrastructure/backend/)
│   │   └── variables.cue (environment-specific variables for Azure Pipelines)
│   ├── prod/
│   │   ├── frontend.cue (imports schemas from company-cue-schemas/schemas/infrastructure/frontend/)
│   │   ├── backend.cue (imports schemas from company-cue-schemas/schemas/infrastructure/backend/)
│   │   └── variables.cue (environment-specific variables for Azure Pipelines)
│   └── shared/
│        ├── frontend.cue (default app/repo specific values, likely app name, etc)
│        ├── backend.cue (default app/repo specific values, likely app name, etc)
│        └── variables.cue (environment-specific variables for Azure Pipelines)
├── .devops/
│   ├── dev/
│   │   └── variables.yaml (Generated with `cue export` from dev/frontend.cue, dev/backend.cue, dev/variables.cue)
│   ├── prod/
│   │   └── variables.yaml (Generated with `cue export` from prod/frontend.cue, prod/backend.cue, prod/variables.cue)
│   ├── azure-pipelines-frontend.yml (Front-end deployment pipeline for all environments. Generated with `cue export` from dev/frontend.cue, prod/frontend.cue)
│   └── azure-pipelines-backend.yml (Back-end deployment pipeline for all environments. Generated with `cue export` from dev/backend.cue, prod/backend.cue)
├── .gitignore
└── README.md

App Front-end Repository (frontend-repo):

frontend-repo/
├── .cue/
│   ├── buildvalues.cue (Build specific values. Potentially imports schemas from company-cue-schema)
│   ├── someconfig.cue (Default configuration)
│   ├── someconfig.schema.cue (Repo-specific schema. May import from company-cue-schema)
│   └── build.cue (imports schemas from company-cue-schemas/schemas/pipelines)
├── src/ (Application implementation)
│   ├── someconfig.someextension (Generated with `cue export` from .cue/someconfig.cue)
│   └── //app implementation comment
├── .devops/
│   └── azure-pipelines.yaml (Environment-agnostic build pipeline. Generated with `cue export` from .cue/build.cue)
└── README.md

NOTE: Azure pipelines can be replaced with GitOps solution here.

App Back-end Repository (backend-repo):

frontend-repo/
├── .cue/
│   ├── buildvalues.cue (Build specific values. Potentially imports schemas from company-cue-schema)
│   ├── someconfig.cue (Default configuration)
│   ├── someconfig.schema.cue (Repo-specific schema. May import from company-cue-schema)
│   └── build.cue (imports schemas from company-cue-schemas/schemas/pipelines)
├── src/ (Application implementation)
│   ├── someconfig.someextension (Generated with `cue export` from .cue/someconfig.cue)
│   └── //app implementation comment
├── .devops/
│   └── azure-pipelines.yaml (Environment-agnostic build pipeline. Generated with `cue export` from .cue/build.cue)
└── README.md

NOTE: Azure pipelines can be replaced with GitOps solution here.