Skip to content

ci-skip-analysis.md

Latest commit

 

History

History
381 lines (282 loc) · 13 KB

ci-skip-analysis.md

File metadata and controls

381 lines (282 loc) · 13 KB

Can We Skip CI for Pre-Release Builds? Analysis

Issue: LE-517
Problem: Pre-release builds (PyPI + Docker) require 45-60 min CI wait

Current Workflow

┌─────────────────────────────────────────────────────────────┐
│ Phase 1: Build Packages (15-20 min) - NO CI DEPENDENCY     │
├─────────────────────────────────────────────────────────────┤
│ • build-lfx                                                  │
│ • build-base                                                 │
│ • build-main                                                 │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 2: Cross-Platform Tests (15 min) - NO CI DEPENDENCY  │
├─────────────────────────────────────────────────────────────┤
│ • Test on Linux (amd64, arm64)                              │
│ • Test on macOS (amd64, arm64)                              │
│ • Test on Windows (amd64)                                   │
│ • Python 3.10, 3.12, 3.13                                   │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 3: CI Suite (45-60 min) ⚠️ BOTTLENECK                │
├─────────────────────────────────────────────────────────────┤
│ • Backend tests (Python 3.10-3.13, 5 groups = 20 jobs)     │
│ • Frontend E2E tests (Playwright)                           │
│ • Frontend unit tests (Jest)                                │
│ • Linting (mypy)                                            │
│ • Docs build                                                │
│ • Template tests                                            │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 4: Publish & Build (12-17 min) - BLOCKED BY CI       │
├─────────────────────────────────────────────────────────────┤
│ • publish-base (line 602: needs ci)                         │
│ • publish-main (line 625: needs ci)                         │
│ • publish-lfx (line 648: needs ci)                          │
│ • call_docker_build_base (line 671: needs ci)              │
│ • call_docker_build_main (line 683: needs ci)              │
└─────────────────────────────────────────────────────────────┘

TOTAL TIME: 87-112 minutes
TIME SPENT WAITING FOR CI: 45-60 minutes (53-67% of total time)

The Core Question

Can we skip CI and still safely publish pre-releases?

What CI Validates

Test Type Time Catches Critical for Pre-Release?
Backend Unit Tests 15-20 min Logic errors, edge cases ⚠️ MEDIUM
Frontend E2E Tests 20-30 min UI workflows, integrations ⚠️ MEDIUM
Frontend Unit Tests 5-10 min Component logic ⚠️ LOW
Linting (mypy) 5 min Type errors ❌ LOW
Docs Build 5 min Documentation issues ❌ NONE
Template Tests 5 min Starter project issues ❌ LOW

What Cross-Platform Tests Already Validate

✅ Installation works on all platforms
✅ Dependencies resolve correctly
✅ Basic imports don't fail
✅ Server starts successfully

Key Insight: Cross-platform tests already catch the most critical issues (broken installation, missing dependencies, import errors).

Risk Analysis

If We Skip CI for Pre-Releases

What Could Go Wrong:

  1. Broken functionality (Medium risk)

    • Some features don't work
    • Customer discovers during testing
    • Mitigation: That's the point of pre-release testing

  2. Integration issues (Low risk)

    • Components don't work together
    • Customer discovers during testing
    • Mitigation: Pre-release is for finding these issues

  3. Type errors (Low risk)

    • Runtime errors from type mismatches
    • May or may not surface
    • Mitigation: Not critical for pre-release

  4. Complete failure (Very low risk)

    • Package won't install or start
    • Already caught by cross-platform tests
    • Cross-platform tests include server startup validation

What Won't Go Wrong:

❌ Installation failures → Caught by cross-platform tests
❌ Missing dependencies → Caught by cross-platform tests
❌ Import errors → Caught by cross-platform tests
❌ Server won't start → Caught by cross-platform tests

The Real Question

Not "Can we skip CI?" but "What's the purpose of pre-releases?"

Pre-Release Purpose

Pre-releases exist to:

  1. Get customer feedback on new features
  2. Test in real-world environments
  3. Find integration issues
  4. Validate before production release

Key Point: Customers testing pre-releases EXPECT to find issues. That's why they're testing.

Production Release vs Pre-Release

Aspect Production Release Pre-Release
Quality Bar Must be stable Can have issues
Testing Exhaustive Sufficient
Speed Can wait Time-sensitive
Risk Tolerance Zero Acceptable
CI Required YES DEBATABLE

Scenarios

Scenario 1: Code from RC Branch ✅ SAFE TO SKIP

Context: Release candidate branch that already passed CI

Analysis:

  • Code was validated when merged to RC
  • No changes since last CI run
  • Just packaging existing tested code

Risk: VERY LOW (code already tested)

Recommendation: Skip CI entirely

Scenario 2: New Bug Fix ⚠️ ACCEPTABLE RISK

Context: Urgent fix needs customer validation

Analysis:

  • Cross-platform tests catch critical issues
  • Customers expect pre-releases may have issues
  • Faster feedback loop (hours vs days)
  • Can rollback if needed

Risk: MEDIUM (untested code, but pre-release context)

Recommendation: Skip CI with notification

Scenario 3: Major Refactoring ❌ DON'T SKIP

Context: Large code changes, architectural changes

Risk: HIGH (many potential issues)

Recommendation: Run full CI

Recommendation

YES, we can skip CI for pre-releases

Rationale:

  1. Cross-platform tests already validate critical issues

    • Installation works
    • Dependencies resolve
    • Server starts
    • Basic functionality works

  2. Pre-releases are for finding issues

    • Customers expect potential problems
    • That's why they're testing pre-releases
    • Not production-quality requirements

  3. Time savings are significant

    • 45-60 minutes saved (53-67% faster)
    • Enables rapid iteration
    • Better customer experience

  4. Risk is acceptable

    • Worst case: broken pre-release
    • Impact: Customer reports issue, we fix
    • Mitigation: Clear pre-release labeling

Proposed Implementation

on:
  workflow_dispatch:
    inputs:
      skip_ci:
        description: "Skip CI for pre-release (use with caution)"
        type: boolean
        default: false
      skip_ci_reason:
        description: "Required: Why skip CI?"
        type: string

jobs:
  validate-skip-ci:
    if: ${{ inputs.skip_ci }}
    runs-on: ubuntu-latest
    steps:
      - name: Require justification
        run: |
          if [ -z "${{ inputs.skip_ci_reason }}" ]; then
            echo "Error: Must provide skip_ci_reason"
            exit 1
          fi
          
      - name: Notify team
        run: |
          echo "⚠️ CI SKIPPED for ${{ inputs.release_tag }}"
          echo "Reason: ${{ inputs.skip_ci_reason }}"
          # Send to Slack/Discord
  
  ci:
    if: ${{ !inputs.skip_ci }}
    uses: ./.github/workflows/ci.yml
    # ... existing CI
  
  publish-base:
    needs: [build-base, test-cross-platform]  # Remove 'ci'
    if: |
      ${{ inputs.release_package_base && 
          (inputs.skip_ci || needs.ci.result == 'success') }}
    # ... existing publish
  
  publish-main:
    needs: [build-main, test-cross-platform, publish-base]  # Remove 'ci'
    if: |
      ${{ inputs.release_package_main && 
          (inputs.skip_ci || needs.ci.result == 'success') }}
    # ... existing publish
  
  call_docker_build_main:
    needs: []  # Remove 'ci' dependency
    if: |
      ${{ inputs.build_docker_main && 
          (inputs.skip_ci || needs.ci.result == 'success') }}
    # ... existing Docker build

Safeguards

  1. Explicit opt-in: skip_ci=false by default
  2. Required justification: Must explain why skipping
  3. Team notification: Alert via Slack/Discord
  4. Audit trail: Log all skip decisions
  5. Clear labeling: Pre-release tags (rc, beta, alpha)

Usage Guidelines

When to skip CI:

  • ✅ Code from RC branch (already tested)
  • ✅ Urgent customer testing needed
  • ✅ Internal testing only
  • ✅ Time-sensitive pre-release

When NOT to skip CI:

  • ❌ Production releases (never skip)
  • ❌ Major refactoring
  • ❌ First pre-release of version
  • ❌ Dependency updates

Expected Results

Time Savings

Current:

Build (20 min) + Cross-platform (15 min) + CI (50 min) + Publish (15 min) = 100 min

With skip_ci:

Build (20 min) + Cross-platform (15 min) + Publish (15 min) = 50 min

Savings: 50 minutes (50% faster)

Risk Assessment

Probability of broken pre-release: ~10-15%
Impact if broken: Low (pre-release context, quick rollback)
Mitigation: Cross-platform tests catch 80-90% of critical issues

Trade-off: 50 min saved vs 10-15% risk of non-critical issues

Alternative: Better Release Planning?

The issue suggests: "investigate if the real solution is better planned release cycles"

Analysis:

Better planning helps but doesn't solve:

  • ❌ Urgent bug fixes
  • ❌ Customer-specific issues
  • ❌ Unexpected problems
  • ❌ Last-minute changes

Better planning + skip_ci option:

  • ✅ Planned releases use full CI
  • ✅ Urgent releases use skip_ci
  • ✅ Best of both worlds

Conclusion: Better planning is good practice, but skip_ci option is still valuable for urgent scenarios.

Final Recommendation

Implement skip_ci parameter with safeguards.

Why:

  1. Cross-platform tests already validate critical issues
  2. Pre-releases are for finding issues (acceptable risk)
  3. 50% time savings for urgent scenarios
  4. Explicit opt-in with justification required
  5. Doesn't prevent better release planning

Implementation Priority:

Week 1: Add skip_ci parameter and validation
Week 2: Add team notifications and audit logging
Week 3: Document usage guidelines
Week 4: Monitor usage and adjust as needed

Success Metrics:

  • Time to pre-release: < 60 minutes (from 100+ minutes)
  • Pre-release quality: > 85% work without issues
  • Usage: 2-3 times per release cycle for urgent needs

Conclusion

YES, we can skip CI for pre-releases.

The 45-60 minute CI wait is unnecessary when:

  • Cross-platform tests already validate installation
  • Pre-releases are explicitly for testing
  • Time savings enable better customer experience
  • Risk is acceptable with proper safeguards

This isn't about lowering quality standards - it's about recognizing that pre-releases and production releases have different requirements. Pre-releases are for finding issues; production releases are for stability.

The solution is both:

  • Better release planning (reduce urgent needs)
  • skip_ci option (handle urgent needs when they arise)