|
| 1 | +--- |
| 2 | +description: Profile memory and CPU usage during e2e tests and analyze results |
| 3 | +--- |
| 4 | + |
| 5 | +# Test Profiling |
| 6 | + |
| 7 | +Profile memory and CPU usage during e2e tests using the Go-based `bin/test-profile` tool. |
| 8 | + |
| 9 | +## Available Commands |
| 10 | + |
| 11 | +### /test-profile start [name] |
| 12 | +Start background profiling daemon. Collects heap/CPU profiles from operator-controller and catalogd every 10s. |
| 13 | +- Auto-stops after 3 consecutive collection failures (e.g., cluster teardown) |
| 14 | +- Use with any test command: `make test-e2e`, `make test-experimental-e2e`, etc. |
| 15 | +- Follow with `/test-profile stop` to generate analysis |
| 16 | + |
| 17 | +**Example:** |
| 18 | +```bash |
| 19 | +/test-profile start baseline |
| 20 | +make test-e2e |
| 21 | +/test-profile stop |
| 22 | +``` |
| 23 | + |
| 24 | +### /test-profile stop |
| 25 | +Stop profiling daemon and generate analysis report. Cleans up port-forwards and empty profiles. |
| 26 | + |
| 27 | +### /test-profile run [name] [test-target] |
| 28 | +Automated workflow: start test, profile until completion, analyze. |
| 29 | +- Default test-target: `test-e2e` |
| 30 | +- Other targets: `test-experimental-e2e`, `test-upgrade-e2e`, etc. |
| 31 | + |
| 32 | +**Example:** |
| 33 | +```bash |
| 34 | +/test-profile run baseline test-e2e |
| 35 | +``` |
| 36 | + |
| 37 | +### /test-profile analyze [name] |
| 38 | +Analyze existing profiles in `test-profiles/[name]/`. Generates markdown report with: |
| 39 | +- Memory/CPU growth patterns |
| 40 | +- Top allocators |
| 41 | +- OpenAPI, JSON, and cache hotspots |
| 42 | + |
| 43 | +### /test-profile compare [baseline] [optimized] |
| 44 | +Compare two test runs. Outputs to `test-profiles/comparisons/[baseline]-vs-[optimized].md` |
| 45 | + |
| 46 | +### /test-profile collect |
| 47 | +One-time snapshot of heap/CPU profiles from running pods. Saves to `test-profiles/manual/` |
| 48 | + |
| 49 | +## Implementation Steps |
| 50 | + |
| 51 | +When executing commands, I will: |
| 52 | + |
| 53 | +1. **Build tool**: `make build-test-profiler` (builds to `bin/test-profile`) |
| 54 | +2. **Execute command**: `./bin/test-profile [command] [args]` |
| 55 | +3. **For start/stop workflow**: Monitor daemon logs, handle errors gracefully |
| 56 | +4. **For run command**: Start test in background, monitor progress, analyze on completion |
| 57 | +5. **For analysis**: Present key findings from generated markdown reports |
| 58 | + |
| 59 | +## Configuration |
| 60 | + |
| 61 | +Environment variables (defaults shown): |
| 62 | +```bash |
| 63 | +TEST_PROFILE_COMPONENTS="operator-controller:olmv1-system:operator-controller-controller-manager:6060;catalogd:olmv1-system:catalogd-controller-manager:6060" |
| 64 | +TEST_PROFILE_INTERVAL=10 # seconds between collections |
| 65 | +TEST_PROFILE_CPU_DURATION=10 # CPU profiling duration |
| 66 | +TEST_PROFILE_MODE=both # both|heap|cpu |
| 67 | +TEST_PROFILE_DIR=./test-profiles |
| 68 | +TEST_PROFILE_TEST_TARGET=test-e2e |
| 69 | +``` |
| 70 | + |
| 71 | +## Output Structure |
| 72 | + |
| 73 | +``` |
| 74 | +test-profiles/ |
| 75 | +├── [name]/ |
| 76 | +│ ├── operator-controller/ |
| 77 | +│ │ ├── heap*.pprof |
| 78 | +│ │ └── cpu*.pprof |
| 79 | +│ ├── catalogd/ |
| 80 | +│ │ ├── heap*.pprof |
| 81 | +│ │ └── cpu*.pprof |
| 82 | +│ ├── profiler.log |
| 83 | +│ └── analysis.md |
| 84 | +└── comparisons/ |
| 85 | + └── [name1]-vs-[name2].md |
| 86 | +``` |
| 87 | + |
| 88 | +## Tool Location |
| 89 | + |
| 90 | +- Source: `hack/tools/test-profiling/` (Go-based CLI) |
| 91 | +- Binary: `bin/test-profile` |
| 92 | +- Make targets: `make build-test-profiler`, `make start-profiling/[name]`, `make stop-profiling` |
| 93 | + |
| 94 | +## Requirements |
| 95 | + |
| 96 | +- kubectl access to cluster |
| 97 | +- go tool pprof (for analysis) |
| 98 | +- Go 1.22+ (for building) |
0 commit comments