diff --git a/README.ja.md b/README.ja.md index 2b831bd..ef2b641 100644 --- a/README.ja.md +++ b/README.ja.md @@ -6,6 +6,8 @@ [English](README.md) · [한국어](README.ko.md) +**ステータス: Public Beta** + ## FlowMap とは FlowMap は Swift コードを解析し、ファイル・型・関数・呼び出し関係をワークスペース全体のグラフとして構築し、VS Code 上で視覚的に探索できる開発者ツールです。 @@ -119,6 +121,7 @@ FlowMap には、実際の Git コミットペアをリプレイし、graph diff - コミットリプレイ runner: `scripts/run_commit_replay.mjs` - シナリオ runner(合成回帰セット): `scripts/run_sample_scenarios.mjs` - リプレイ要約 builder: `scripts/build_replay_summary.mjs` +- 詳細検証レポート builder: `scripts/build_validation_detail.mjs` 例: @@ -133,6 +136,9 @@ node scripts/run_commit_replay.mjs \ - `reports/replay-validation-bundle.md` - `reports/replay-validation-bundle.json` +- `reports/sample-scenarios-report.json` +- `reports/public-beta-validation-detail.md` +- `reports/public-beta-validation-detail.json` 最新の統合メトリクス(`reports/replay-validation-bundle.md`): @@ -142,6 +148,17 @@ node scripts/run_commit_replay.mjs \ - Swift Detection Rate: 94.64% - Overall Match Rate: 97.13% +最新シナリオ回帰検証(`reports/sample-scenarios-report.json`): + +- シナリオ数: 60 +- Passed / Failed: 60 / 0 +- FP / FN 合計: 0 / 0 + +Public Beta 詳細検証(`reports/public-beta-validation-detail.md`): + +- 非パスケースをコミット単位で記録 +- Public beta gate: PASS + ## ライセンス FlowMap は source-available モデルで提供されています。 @@ -181,4 +198,8 @@ Issue と Pull Request を歓迎します。 ## ステータス -FlowMap は継続的に進化しています。現バージョンは完成済みプラットフォームではなく、初期段階の実用ツールです。フィードバックと貢献を歓迎します。 +FlowMap は現在 **Public Beta** です。 + +- コアのグラフ/差分パイプラインは外部テスト可能な安定度に到達しています。 +- 検証成果物は `reports/` に公開し、透明性を確保しています。 +- Swift 呼び出し解決の一部 edge case は継続改善中です。 diff --git a/README.ko.md b/README.ko.md index d0f0085..7b0199a 100644 --- a/README.ko.md +++ b/README.ko.md @@ -6,6 +6,8 @@ [English](README.md) · [日本語](README.ja.md) +**상태: Public Beta** + ## FlowMap이란? FlowMap은 Swift 코드를 파싱해 파일, 타입, 함수, 호출 관계를 워크스페이스 단위로 그래프화하고, VS Code에서 시각적으로 탐색할 수 있게 해주는 개발 도구입니다. @@ -119,6 +121,7 @@ FlowMap은 실제 Git 커밋 쌍을 리플레이해서 diff 검출 방향이 맞 - 커밋 리플레이 러너: `scripts/run_commit_replay.mjs` - 시나리오 러너(합성 회귀 세트): `scripts/run_sample_scenarios.mjs` - 리플레이 요약 빌더: `scripts/build_replay_summary.mjs` +- 상세 검증 리포트 빌더: `scripts/build_validation_detail.mjs` 예시: @@ -133,6 +136,9 @@ node scripts/run_commit_replay.mjs \ - `reports/replay-validation-bundle.md` - `reports/replay-validation-bundle.json` +- `reports/sample-scenarios-report.json` +- `reports/public-beta-validation-detail.md` +- `reports/public-beta-validation-detail.json` 최신 통합 지표(`reports/replay-validation-bundle.md`): @@ -142,6 +148,17 @@ node scripts/run_commit_replay.mjs \ - Swift Detection Rate: 94.64% - Overall Match Rate: 97.13% +최신 시나리오 회귀 검증(`reports/sample-scenarios-report.json`): + +- 시나리오 수: 60 +- 통과 / 실패: 60 / 0 +- FP / FN 합계: 0 / 0 + +퍼블릭 베타 상세 검증(`reports/public-beta-validation-detail.md`): + +- 비정상 케이스를 커밋 단위로 상세 기록 +- Public beta gate: PASS + ## 라이선스 FlowMap은 source-available 방식으로 제공됩니다. @@ -181,4 +198,8 @@ FlowMap은 source-available 방식으로 제공됩니다. ## 상태 -FlowMap은 빠르게 발전 중입니다. 현재 버전은 완성된 플랫폼이 아닌 초기 단계 도구입니다. 피드백과 기여를 환영합니다. +FlowMap은 이제 **Public Beta** 단계입니다. + +- 핵심 그래프/디프 파이프라인은 외부 테스트 가능한 수준으로 안정화되었습니다. +- 검증 산출물은 `reports/`에 공개하여 투명하게 확인할 수 있습니다. +- Swift 호출 해석의 일부 edge case는 지속적으로 개선 중입니다. diff --git a/README.md b/README.md index 8f661a6..3487efd 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ [한국어](README.ko.md) · [日本語](README.ja.md) +**Status: Public Beta** + ## What is FlowMap? FlowMap parses Swift code, builds a workspace-level graph of files, types, functions, and call relationships, and visualizes that graph inside VS Code. @@ -119,6 +121,7 @@ FlowMap includes automated replay validation that replays real Git commit pairs - Commit replay runner: `scripts/run_commit_replay.mjs` - Scenario runner (synthetic regression set): `scripts/run_sample_scenarios.mjs` - Replay summary builder: `scripts/build_replay_summary.mjs` +- Detailed validation builder: `scripts/build_validation_detail.mjs` Example: @@ -133,6 +136,9 @@ Current bundled validation output: - `reports/replay-validation-bundle.md` - `reports/replay-validation-bundle.json` +- `reports/sample-scenarios-report.json` +- `reports/public-beta-validation-detail.md` +- `reports/public-beta-validation-detail.json` Latest bundled metrics (`reports/replay-validation-bundle.md`): @@ -142,6 +148,17 @@ Latest bundled metrics (`reports/replay-validation-bundle.md`): - Swift Detection Rate: 94.64% - Overall Match Rate: 97.13% +Latest scenario regression (`reports/sample-scenarios-report.json`): + +- Scenario count: 60 +- Passed / Failed: 60 / 0 +- FP / FN total: 0 / 0 + +Detailed public beta validation (`reports/public-beta-validation-detail.md`): + +- Non-pass replay cases listed with commit-level detail +- Public beta gate: PASS + ## License FlowMap is source-available. @@ -181,4 +198,8 @@ Good areas to contribute: ## Status -FlowMap is actively evolving. The current version is a functional early-stage tool, not yet a finished platform. Feedback and contributions are encouraged. +FlowMap is now in **Public Beta**. + +- Core graph/diff pipeline is stable enough for external testing. +- Validation artifacts are published under `reports/` for transparent review. +- Some edge cases in Swift call resolution still exist and are tracked as ongoing improvements. diff --git a/reports/public-beta-validation-detail.json b/reports/public-beta-validation-detail.json new file mode 100644 index 0000000..98f9f26 --- /dev/null +++ b/reports/public-beta-validation-detail.json @@ -0,0 +1,324 @@ +{ + "created_at": "2026-03-07T19:30:58.208Z", + "mode": "public-beta-validation-detail", + "inputs": { + "replay_reports": [ + "/Users/seungminlee/Desktop/Development/FlowMap/reports/replay-heavy-swift-full-strict.json", + "/Users/seungminlee/Desktop/Development/FlowMap/reports/replay-flowmap-75.json", + "/Users/seungminlee/Desktop/Development/FlowMap/reports/replay-kdecoder.json" + ], + "scenario_report": "/Users/seungminlee/Desktop/Development/FlowMap/reports/sample-scenarios-report.json" + }, + "replay": { + "totals": { + "total_pairs": 209, + "swift_pairs": 112, + "non_swift_pairs": 97, + "tp": 106, + "tn": 97, + "fp": 0, + "fn": 6, + "pass_pairs": 203, + "warn_pairs": 6, + "failed_pairs": 0, + "error_pairs": 0 + }, + "metrics": { + "non_swift_fp_rate": 0, + "swift_detection_rate": 0.9464, + "overall_match_rate": 0.9713 + }, + "reports": [ + { + "report": "replay-heavy-swift-full-strict.json", + "repo": "flowmap-heavy-swift-repo", + "strict_swift": true, + "total_pairs": 120, + "swift_pairs": 100, + "non_swift_pairs": 20, + "confusion": { + "tp": 100, + "tn": 20, + "fp": 0, + "fn": 0 + }, + "pass_pairs": 120, + "warn_pairs": 0, + "failed_pairs": 0, + "error_pairs": 0, + "non_swift_fp_rate": 0, + "swift_detection_rate": 1, + "gate": { + "max_non_swift_fp_rate": 0, + "max_errors": 0, + "strict_swift_required": true + }, + "gate_passed": true + }, + { + "report": "replay-flowmap-75.json", + "repo": "FlowMap", + "strict_swift": false, + "total_pairs": 74, + "swift_pairs": 4, + "non_swift_pairs": 70, + "confusion": { + "tp": 4, + "tn": 70, + "fp": 0, + "fn": 0 + }, + "pass_pairs": 74, + "warn_pairs": 0, + "failed_pairs": 0, + "error_pairs": 0, + "non_swift_fp_rate": 0, + "swift_detection_rate": 1, + "gate": { + "max_non_swift_fp_rate": 0, + "max_errors": 0, + "strict_swift_required": false + }, + "gate_passed": true + }, + { + "report": "replay-kdecoder.json", + "repo": "KDecoder", + "strict_swift": false, + "total_pairs": 15, + "swift_pairs": 8, + "non_swift_pairs": 7, + "confusion": { + "tp": 2, + "tn": 7, + "fp": 0, + "fn": 6 + }, + "pass_pairs": 9, + "warn_pairs": 6, + "failed_pairs": 0, + "error_pairs": 0, + "non_swift_fp_rate": 0, + "swift_detection_rate": 0.25, + "gate": { + "max_non_swift_fp_rate": 0, + "max_errors": 0, + "strict_swift_required": false + }, + "gate_passed": true + } + ], + "non_pass_summary": { + "count": 6, + "by_status": { + "warn_swift_no_graph_change": 6 + } + }, + "non_pass_cases": [ + { + "repo": "KDecoder", + "report": "replay-kdecoder.json", + "level": "warn", + "status": "warn_swift_no_graph_change", + "prev": "ac97c6859b000be3a354dc3808745fb363f7abdb", + "curr": "0bbee279dc78c61e3ce1462ea9a90f05d4583840", + "commit": { + "short": "0bbee27", + "hash": "0bbee279dc78c61e3ce1462ea9a90f05d4583840", + "subject": "Bump version to v2.0.1", + "author": "adgk2349", + "date": "2026-03-02T23:51:44+09:00" + }, + "swift_changed": true, + "swift_files": [ + "KDecoder/View/ContentView.swift" + ], + "engine_changed": false, + "expected_changed": false, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + } + }, + { + "repo": "KDecoder", + "report": "replay-kdecoder.json", + "level": "warn", + "status": "warn_swift_no_graph_change", + "prev": "9472679eeda62d29422e2c28d37962b72cb7047e", + "curr": "dacbd2e2628f36c618f2cd1b04d3dc429bc1e5b5", + "commit": { + "short": "dacbd2e", + "hash": "dacbd2e2628f36c618f2cd1b04d3dc429bc1e5b5", + "subject": "Translate README to Korean, add quit button", + "author": "adgk2349", + "date": "2026-03-03T00:00:26+09:00" + }, + "swift_changed": true, + "swift_files": [ + "KDecoder/View/ContentView.swift" + ], + "engine_changed": false, + "expected_changed": false, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + } + }, + { + "repo": "KDecoder", + "report": "replay-kdecoder.json", + "level": "warn", + "status": "warn_swift_no_graph_change", + "prev": "dacbd2e2628f36c618f2cd1b04d3dc429bc1e5b5", + "curr": "32eb843be13f8a57db450371ba2b600273653546", + "commit": { + "short": "32eb843", + "hash": "32eb843be13f8a57db450371ba2b600273653546", + "subject": "Move quit button to bottom bar with visible glassEffect style", + "author": "adgk2349", + "date": "2026-03-03T00:05:11+09:00" + }, + "swift_changed": true, + "swift_files": [ + "KDecoder/View/ContentView.swift" + ], + "engine_changed": false, + "expected_changed": false, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + } + }, + { + "repo": "KDecoder", + "report": "replay-kdecoder.json", + "level": "warn", + "status": "warn_swift_no_graph_change", + "prev": "32eb843be13f8a57db450371ba2b600273653546", + "curr": "9ce4f79c4ef454433fb8e4d8b261834b010ef174", + "commit": { + "short": "9ce4f79", + "hash": "9ce4f79c4ef454433fb8e4d8b261834b010ef174", + "subject": "Rearrange layout: version beside title, quit top-right, checkbox bottom-right", + "author": "adgk2349", + "date": "2026-03-03T00:08:30+09:00" + }, + "swift_changed": true, + "swift_files": [ + "KDecoder/View/ContentView.swift" + ], + "engine_changed": false, + "expected_changed": false, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + } + }, + { + "repo": "KDecoder", + "report": "replay-kdecoder.json", + "level": "warn", + "status": "warn_swift_no_graph_change", + "prev": "34ae261cb34b953be0b96c8b94db7038d8b7d121", + "curr": "014d377246c4350361321f22246c16c17ee77fc6", + "commit": { + "short": "014d377", + "hash": "014d377246c4350361321f22246c16c17ee77fc6", + "subject": "Update repository links to reflect rename to KDecoder-for-Mac", + "author": "adgk2349", + "date": "2026-03-03T00:53:28+09:00" + }, + "swift_changed": true, + "swift_files": [ + "KDecoder/View/AboutView.swift" + ], + "engine_changed": false, + "expected_changed": false, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + } + }, + { + "repo": "KDecoder", + "report": "replay-kdecoder.json", + "level": "warn", + "status": "warn_swift_no_graph_change", + "prev": "014d377246c4350361321f22246c16c17ee77fc6", + "curr": "36d81f811a61593221eeac7de3994d5a34309871", + "commit": { + "short": "36d81f8", + "hash": "36d81f811a61593221eeac7de3994d5a34309871", + "subject": "Correct repository name to KDecoder_for_Mac in links", + "author": "adgk2349", + "date": "2026-03-03T00:55:20+09:00" + }, + "swift_changed": true, + "swift_files": [ + "KDecoder/View/AboutView.swift" + ], + "engine_changed": false, + "expected_changed": false, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + } + } + ] + }, + "sample_scenarios": { + "scenario_count": 60, + "passed": 60, + "failed": 0, + "pass_rate": 1, + "fp_total": 0, + "fn_total": 0, + "gate_passed": true + }, + "public_beta_gate": { + "thresholds": { + "replay_non_swift_fp_rate_max": 0.01, + "replay_swift_detection_rate_min": 0.9, + "replay_overall_match_rate_min": 0.95, + "replay_error_pairs_max": 0, + "sample_pass_rate_min": 1, + "sample_fp_total_max": 0, + "sample_fn_total_max": 0 + }, + "checks": { + "replay_non_swift_fp_rate_ok": true, + "replay_swift_detection_rate_ok": true, + "replay_overall_match_rate_ok": true, + "replay_error_pairs_ok": true, + "sample_pass_rate_ok": true, + "sample_fp_total_ok": true, + "sample_fn_total_ok": true + }, + "passed": true + } +} diff --git a/reports/public-beta-validation-detail.md b/reports/public-beta-validation-detail.md new file mode 100644 index 0000000..86dbf4f --- /dev/null +++ b/reports/public-beta-validation-detail.md @@ -0,0 +1,63 @@ +# FlowMap Public Beta Validation (Detailed) + +- Generated: 2026-03-07T19:30:58.208Z +- Replay reports: 3 +- Scenario report: sample-scenarios-report.json + +## Replay Aggregate + +- Total commit pairs: 209 +- Swift / Non-Swift pairs: 112 / 97 +- TP / TN / FP / FN: 106 / 97 / 0 / 6 +- Non-Swift FP Rate: 0% +- Swift Detection Rate: 94.64% +- Overall Match Rate: 97.13% + +## Replay Per Report + +| Repo | Strict | Pairs | Swift | Non-Swift | TP | TN | FP | FN | Warn | Fail | Error | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| flowmap-heavy-swift-repo | yes | 120 | 100 | 20 | 100 | 20 | 0 | 0 | 0 | 0 | 0 | +| FlowMap | no | 74 | 4 | 70 | 4 | 70 | 0 | 0 | 0 | 0 | 0 | +| KDecoder | no | 15 | 8 | 7 | 2 | 7 | 0 | 6 | 6 | 0 | 0 | + +## Non-Pass Cases (Replay) + +- Non-pass total: 6 +- warn_swift_no_graph_change: 6 + +| Repo | Status | Commit | Subject | Swift Files | Engine Changed | Expected Changed | +|---|---|---|---|---:|---:|---:| +| KDecoder | warn_swift_no_graph_change | 0bbee27 | Bump version to v2.0.1 | 1 | no | no | +| KDecoder | warn_swift_no_graph_change | dacbd2e | Translate README to Korean, add quit button | 1 | no | no | +| KDecoder | warn_swift_no_graph_change | 32eb843 | Move quit button to bottom bar with visible glassEffect style | 1 | no | no | +| KDecoder | warn_swift_no_graph_change | 9ce4f79 | Rearrange layout: version beside title, quit top-right, checkbox bottom-right | 1 | no | no | +| KDecoder | warn_swift_no_graph_change | 014d377 | Update repository links to reflect rename to KDecoder-for-Mac | 1 | no | no | +| KDecoder | warn_swift_no_graph_change | 36d81f8 | Correct repository name to KDecoder_for_Mac in links | 1 | no | no | + +## Sample Scenario Regression + +- Scenario count: 60 +- Passed / Failed: 60 / 0 +- Pass rate: 100% +- FP total: 0 +- FN total: 0 +- Gate: PASS + +## Public Beta Gate + +- replay_non_swift_fp_rate_ok: PASS +- replay_swift_detection_rate_ok: PASS +- replay_overall_match_rate_ok: PASS +- replay_error_pairs_ok: PASS +- sample_pass_rate_ok: PASS +- sample_fp_total_ok: PASS +- sample_fn_total_ok: PASS + +**Final Gate:** PASS + +## Notes + +- `warn_swift_no_graph_change` means Swift files changed but graph topology did not change. +- In non-strict mode, these warnings are tracked but do not fail the report. + diff --git a/reports/replay-validation-bundle.json b/reports/replay-validation-bundle.json index b63cb3c..198c9f0 100644 --- a/reports/replay-validation-bundle.json +++ b/reports/replay-validation-bundle.json @@ -1,5 +1,5 @@ { - "created_at": "2026-03-07T08:09:16.408Z", + "created_at": "2026-03-07T19:29:12.817Z", "total_reports": 3, "total_pairs": 209, "total_swift_pairs": 112, diff --git a/reports/replay-validation-bundle.md b/reports/replay-validation-bundle.md index 7c1f93b..9f94fc3 100644 --- a/reports/replay-validation-bundle.md +++ b/reports/replay-validation-bundle.md @@ -1,6 +1,6 @@ # Commit Replay Validation Summary -- Generated: 2026-03-07T08:09:16.408Z +- Generated: 2026-03-07T19:29:12.817Z - Reports merged: 3 - Total commit pairs: 209 - Swift / Non-Swift pairs: 112 / 97 diff --git a/reports/sample-scenarios-report.json b/reports/sample-scenarios-report.json new file mode 100644 index 0000000..0f280bb --- /dev/null +++ b/reports/sample-scenarios-report.json @@ -0,0 +1,986 @@ +{ + "created_at": "2026-03-07T19:29:18.530Z", + "tool": "FlowMap sample scenario runner", + "workspace": "/Users/seungminlee/Desktop/Development/FlowMap", + "binaries": { + "flowmap": "/Users/seungminlee/Desktop/Development/FlowMap/target/debug/flowmap", + "parser": "/Users/seungminlee/Desktop/Development/FlowMap/parsers/swift-ast/.build/debug/flowmap-swift-ast" + }, + "summary": { + "scenario_count": 60, + "passed": 60, + "failed": 0, + "pass_rate": 1, + "fp_total": 0, + "fn_total": 0, + "gate": { + "min_pass_rate": 1, + "max_failed_scenarios": 0, + "max_fp_total": 0, + "max_fn_total": 0 + }, + "gate_passed": true + }, + "results": [ + { + "id": "S01", + "name": "Clean workspace should have zero diff", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S02", + "name": "Add new utility file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 2, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S03", + "name": "Add new type file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S04", + "name": "Delete model file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 3, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S05", + "name": "Delete service file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 5, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 5, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S06", + "name": "String literal change should not change graph", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S07", + "name": "Trailing comment should not change graph", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S08", + "name": "Rename service method", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 1, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S09", + "name": "Add helper method to view model", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S10", + "name": "Remove method from view controller", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 1, + "changed_nodes": 2, + "added_edges": 0, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S11", + "name": "Add extra call in viewDidLoad", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 3, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S12", + "name": "Remove existing call in viewDidLoad", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 2, + "added_edges": 0, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S13", + "name": "Change call target fetchData -> refreshData", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S14", + "name": "Add protocol requirement", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S15", + "name": "Remove protocol requirement", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 1, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S16", + "name": "Change protocol signature", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 1, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S17", + "name": "Add nested model type", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S18", + "name": "Rename controller type", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 5, + "removed_nodes": 5, + "changed_nodes": 0, + "added_edges": 6, + "removed_edges": 6, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S19", + "name": "Move setupUI from controller to view model", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 1, + "changed_nodes": 2, + "added_edges": 1, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S20", + "name": "Add uppercase singleton-chain call", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 1, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S21", + "name": "Add duplicate call edge (should stay zero diff)", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S22", + "name": "Line shift should produce changed nodes", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 4, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S23", + "name": "Add new view model file and call it", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S24", + "name": "Delete controller file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 6, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 6, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S25", + "name": "Replace known call with unknown symbol", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S26", + "name": "Add file with unresolved call", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 2, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S27", + "name": "Rename file path (delete+add semantics)", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 5, + "removed_nodes": 5, + "changed_nodes": 0, + "added_edges": 5, + "removed_edges": 5, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S28", + "name": "Change service method signature", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 1, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S29", + "name": "Add nested state enum to controller", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 4, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S30", + "name": "Add one file and delete one file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 3, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S31", + "name": "Add utility file with two free functions", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S32", + "name": "Add core feature flag type file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 2, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S33", + "name": "Delete app file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 3, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S34", + "name": "Delete protocols file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 5, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 4, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S35", + "name": "Rename view model file path", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 5, + "removed_nodes": 5, + "changed_nodes": 0, + "added_edges": 5, + "removed_edges": 5, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S36", + "name": "Rename model type name", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 2, + "removed_nodes": 2, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S37", + "name": "Add helper method to service", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 1, + "added_edges": 1, + "removed_edges": 0, + "impact": 1 + }, + "failures": [] + }, + { + "id": "S38", + "name": "Inline authenticate and remove method", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 1, + "changed_nodes": 1, + "added_edges": 0, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S39", + "name": "Add top-level helper and call it from service", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 2, + "added_edges": 2, + "removed_edges": 0, + "impact": 1 + }, + "failures": [] + }, + { + "id": "S40", + "name": "Add top-level factory function to model file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S41", + "name": "EOF comment in controller should not change graph", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S42", + "name": "Insert blank line near top of view model", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 4, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S43", + "name": "Add enum-only file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 2, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S44", + "name": "Rewrite service file with identical content", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S45", + "name": "Add second protocol requirement", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S46", + "name": "Remove refreshData method in view model", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 1, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S47", + "name": "Add nested enum inside service type", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 3, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S48", + "name": "Change init parameter label in controller", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S49", + "name": "Rename service type", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 4, + "removed_nodes": 4, + "changed_nodes": 0, + "added_edges": 5, + "removed_edges": 5, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S50", + "name": "Add nested-folder helper file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S51", + "name": "Replace model file with new model", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 2, + "removed_nodes": 3, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S52", + "name": "Service string literal update only", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S53", + "name": "Add logDisconnect method and call", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 2, + "added_edges": 2, + "removed_edges": 0, + "impact": 1 + }, + "failures": [] + }, + { + "id": "S54", + "name": "Remove bare call in refreshData", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 0, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S55", + "name": "Change bare call fetchData to debugLog", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 1, + "added_edges": 2, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S56", + "name": "Add uppercase singleton chain call in view model", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 0, + "removed_nodes": 0, + "changed_nodes": 2, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S57", + "name": "Rename controller file path", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 6, + "removed_nodes": 6, + "changed_nodes": 0, + "added_edges": 6, + "removed_edges": 6, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S58", + "name": "Replace app file with alternate app file", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 3, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 2, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S59", + "name": "Add extra protocol in core", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 1, + "removed_nodes": 0, + "changed_nodes": 0, + "added_edges": 1, + "removed_edges": 0, + "impact": 0 + }, + "failures": [] + }, + { + "id": "S60", + "name": "Add file and change service signature together", + "pass": true, + "fp": 0, + "fn": 0, + "counts": { + "added_nodes": 3, + "removed_nodes": 1, + "changed_nodes": 0, + "added_edges": 2, + "removed_edges": 1, + "impact": 0 + }, + "failures": [] + } + ] +} diff --git a/scripts/build_validation_detail.mjs b/scripts/build_validation_detail.mjs new file mode 100644 index 0000000..74c0551 --- /dev/null +++ b/scripts/build_validation_detail.mjs @@ -0,0 +1,365 @@ +#!/usr/bin/env node + +import fs from "fs"; +import path from "path"; + +function fail(msg) { + console.error(`ERROR: ${msg}`); + process.exit(1); +} + +function parseArgs(argv) { + const out = { + replayReports: [], + scenarioReport: "", + jsonOut: "", + mdOut: "", + }; + + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--replay-report") { + const v = argv[++i]; + if (!v) fail("missing value for --replay-report"); + out.replayReports.push(path.resolve(v)); + continue; + } + if (a === "--scenario-report") { + const v = argv[++i]; + if (!v) fail("missing value for --scenario-report"); + out.scenarioReport = path.resolve(v); + continue; + } + if (a === "--json-out") { + const v = argv[++i]; + if (!v) fail("missing value for --json-out"); + out.jsonOut = path.resolve(v); + continue; + } + if (a === "--md-out") { + const v = argv[++i]; + if (!v) fail("missing value for --md-out"); + out.mdOut = path.resolve(v); + continue; + } + if (a === "--help" || a === "-h") { + console.log( + "Usage: node scripts/build_validation_detail.mjs --replay-report [--replay-report ...] --scenario-report --json-out --md-out ", + ); + process.exit(0); + } + fail(`unknown arg: ${a}`); + } + + if (out.replayReports.length === 0) { + fail("at least one --replay-report is required"); + } + if (!out.scenarioReport) { + fail("--scenario-report is required"); + } + if (!out.jsonOut) { + fail("--json-out is required"); + } + if (!out.mdOut) { + fail("--md-out is required"); + } + return out; +} + +function ensureFileExists(filePath, label) { + if (!fs.existsSync(filePath)) { + fail(`${label} not found: ${filePath}`); + } +} + +function ensureParent(filePath) { + fs.mkdirSync(path.dirname(filePath), { recursive: true }); +} + +function pct(numerator, denominator) { + if (!denominator) return 0; + return Number(((numerator / denominator) * 100).toFixed(2)); +} + +function repoLabel(repoPath) { + if (!repoPath || repoPath === "unknown") return "unknown"; + const normalized = String(repoPath).replace(/[\\\/]+$/, ""); + const label = path.basename(normalized); + return label || normalized; +} + +function toResultLevel(status) { + if (status === "pass") return "pass"; + if (String(status).startsWith("warn")) return "warn"; + if (status === "error") return "error"; + return "fail"; +} + +function main() { + const args = parseArgs(process.argv.slice(2)); + + const replayInputs = args.replayReports.map((p) => { + ensureFileExists(p, "replay report"); + return { path: p, data: JSON.parse(fs.readFileSync(p, "utf8")) }; + }); + ensureFileExists(args.scenarioReport, "scenario report"); + const scenario = JSON.parse(fs.readFileSync(args.scenarioReport, "utf8")); + + const replayRows = replayInputs.map(({ path: reportPath, data }) => { + const s = data.summary ?? {}; + const repo = repoLabel(data.config?.repo ?? "unknown"); + const nonPasses = (data.results ?? []) + .filter((r) => r.status !== "pass") + .map((r) => ({ + repo, + report: path.basename(reportPath), + level: toResultLevel(r.status), + status: r.status, + prev: r.prev, + curr: r.curr, + commit: { + short: r.commit?.short ?? "", + hash: r.commit?.hash ?? r.curr ?? "", + subject: r.commit?.subject ?? "", + author: r.commit?.author ?? "", + date: r.commit?.date ?? "", + }, + swift_changed: Boolean(r.swift_changed), + swift_files: r.swift_files ?? [], + engine_changed: Boolean(r.engine_changed), + expected_changed: Boolean(r.expected_changed), + counts: r.counts ?? {}, + })); + + return { + report: path.basename(reportPath), + repo, + strict_swift: Boolean(data.config?.strict_swift), + total_pairs: s.total_pairs ?? 0, + swift_pairs: s.swift_pairs ?? 0, + non_swift_pairs: s.non_swift_pairs ?? 0, + confusion: { + tp: s.confusion?.tp ?? 0, + tn: s.confusion?.tn ?? 0, + fp: s.confusion?.fp ?? 0, + fn: s.confusion?.fn ?? 0, + }, + pass_pairs: s.pass_pairs ?? 0, + warn_pairs: s.warn_pairs ?? 0, + failed_pairs: s.failed_pairs ?? 0, + error_pairs: s.error_pairs ?? 0, + non_swift_fp_rate: s.non_swift_fp_rate ?? 0, + swift_detection_rate: s.swift_detection_rate ?? 0, + gate: s.gate ?? "UNKNOWN", + gate_passed: Boolean(s.gate_passed), + non_passes: nonPasses, + }; + }); + + const replayTotals = replayRows.reduce( + (acc, r) => { + acc.total_pairs += r.total_pairs; + acc.swift_pairs += r.swift_pairs; + acc.non_swift_pairs += r.non_swift_pairs; + acc.tp += r.confusion.tp; + acc.tn += r.confusion.tn; + acc.fp += r.confusion.fp; + acc.fn += r.confusion.fn; + acc.pass_pairs += r.pass_pairs; + acc.warn_pairs += r.warn_pairs; + acc.failed_pairs += r.failed_pairs; + acc.error_pairs += r.error_pairs; + return acc; + }, + { + total_pairs: 0, + swift_pairs: 0, + non_swift_pairs: 0, + tp: 0, + tn: 0, + fp: 0, + fn: 0, + pass_pairs: 0, + warn_pairs: 0, + failed_pairs: 0, + error_pairs: 0, + }, + ); + + const replayMetrics = { + non_swift_fp_rate: + replayTotals.non_swift_pairs === 0 + ? 0 + : Number((replayTotals.fp / replayTotals.non_swift_pairs).toFixed(4)), + swift_detection_rate: + replayTotals.swift_pairs === 0 + ? 0 + : Number((replayTotals.tp / replayTotals.swift_pairs).toFixed(4)), + overall_match_rate: + replayTotals.total_pairs === 0 + ? 0 + : Number(((replayTotals.tp + replayTotals.tn) / replayTotals.total_pairs).toFixed(4)), + }; + + const scenarioSummary = scenario.summary ?? {}; + const scenarioMetrics = { + scenario_count: scenarioSummary.scenario_count ?? 0, + passed: scenarioSummary.passed ?? 0, + failed: scenarioSummary.failed ?? 0, + pass_rate: scenarioSummary.pass_rate ?? 0, + fp_total: scenarioSummary.fp_total ?? 0, + fn_total: scenarioSummary.fn_total ?? 0, + gate_passed: Boolean(scenarioSummary.gate_passed), + }; + + const nonPassCases = replayRows.flatMap((r) => r.non_passes); + const nonPassByStatus = {}; + for (const c of nonPassCases) { + nonPassByStatus[c.status] = (nonPassByStatus[c.status] ?? 0) + 1; + } + + const publicBetaGate = { + replay_non_swift_fp_rate_max: 0.01, + replay_swift_detection_rate_min: 0.9, + replay_overall_match_rate_min: 0.95, + replay_error_pairs_max: 0, + sample_pass_rate_min: 1, + sample_fp_total_max: 0, + sample_fn_total_max: 0, + }; + + const gateChecks = { + replay_non_swift_fp_rate_ok: + replayMetrics.non_swift_fp_rate <= publicBetaGate.replay_non_swift_fp_rate_max, + replay_swift_detection_rate_ok: + replayMetrics.swift_detection_rate >= publicBetaGate.replay_swift_detection_rate_min, + replay_overall_match_rate_ok: + replayMetrics.overall_match_rate >= publicBetaGate.replay_overall_match_rate_min, + replay_error_pairs_ok: + replayTotals.error_pairs <= publicBetaGate.replay_error_pairs_max, + sample_pass_rate_ok: scenarioMetrics.pass_rate >= publicBetaGate.sample_pass_rate_min, + sample_fp_total_ok: scenarioMetrics.fp_total <= publicBetaGate.sample_fp_total_max, + sample_fn_total_ok: scenarioMetrics.fn_total <= publicBetaGate.sample_fn_total_max, + }; + const gatePassed = Object.values(gateChecks).every(Boolean); + + const output = { + created_at: new Date().toISOString(), + mode: "public-beta-validation-detail", + inputs: { + replay_reports: replayInputs.map((x) => x.path), + scenario_report: args.scenarioReport, + }, + replay: { + totals: replayTotals, + metrics: replayMetrics, + reports: replayRows.map((r) => ({ + report: r.report, + repo: r.repo, + strict_swift: r.strict_swift, + total_pairs: r.total_pairs, + swift_pairs: r.swift_pairs, + non_swift_pairs: r.non_swift_pairs, + confusion: r.confusion, + pass_pairs: r.pass_pairs, + warn_pairs: r.warn_pairs, + failed_pairs: r.failed_pairs, + error_pairs: r.error_pairs, + non_swift_fp_rate: r.non_swift_fp_rate, + swift_detection_rate: r.swift_detection_rate, + gate: r.gate, + gate_passed: r.gate_passed, + })), + non_pass_summary: { + count: nonPassCases.length, + by_status: nonPassByStatus, + }, + non_pass_cases: nonPassCases, + }, + sample_scenarios: scenarioMetrics, + public_beta_gate: { + thresholds: publicBetaGate, + checks: gateChecks, + passed: gatePassed, + }, + }; + + const md = [ + "# FlowMap Public Beta Validation (Detailed)", + "", + `- Generated: ${output.created_at}`, + `- Replay reports: ${output.inputs.replay_reports.length}`, + `- Scenario report: ${path.basename(output.inputs.scenario_report)}`, + "", + "## Replay Aggregate", + "", + `- Total commit pairs: ${replayTotals.total_pairs}`, + `- Swift / Non-Swift pairs: ${replayTotals.swift_pairs} / ${replayTotals.non_swift_pairs}`, + `- TP / TN / FP / FN: ${replayTotals.tp} / ${replayTotals.tn} / ${replayTotals.fp} / ${replayTotals.fn}`, + `- Non-Swift FP Rate: ${pct(replayTotals.fp, replayTotals.non_swift_pairs)}%`, + `- Swift Detection Rate: ${pct(replayTotals.tp, replayTotals.swift_pairs)}%`, + `- Overall Match Rate: ${pct(replayTotals.tp + replayTotals.tn, replayTotals.total_pairs)}%`, + "", + "## Replay Per Report", + "", + "| Repo | Strict | Pairs | Swift | Non-Swift | TP | TN | FP | FN | Warn | Fail | Error |", + "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|", + ...replayRows.map( + (r) => + `| ${r.repo} | ${r.strict_swift ? "yes" : "no"} | ${r.total_pairs} | ${r.swift_pairs} | ${r.non_swift_pairs} | ${r.confusion.tp} | ${r.confusion.tn} | ${r.confusion.fp} | ${r.confusion.fn} | ${r.warn_pairs} | ${r.failed_pairs} | ${r.error_pairs} |`, + ), + "", + "## Non-Pass Cases (Replay)", + "", + `- Non-pass total: ${nonPassCases.length}`, + ...Object.entries(nonPassByStatus).map(([status, count]) => `- ${status}: ${count}`), + "", + "| Repo | Status | Commit | Subject | Swift Files | Engine Changed | Expected Changed |", + "|---|---|---|---|---:|---:|---:|", + ...nonPassCases.map((c) => { + const commit = c.commit.short || c.curr.slice(0, 7); + const swiftCount = c.swift_files.length; + const subject = (c.commit.subject || "").replace(/\|/g, "\\|"); + return `| ${c.repo} | ${c.status} | ${commit} | ${subject} | ${swiftCount} | ${c.engine_changed ? "yes" : "no"} | ${c.expected_changed ? "yes" : "no"} |`; + }), + "", + "## Sample Scenario Regression", + "", + `- Scenario count: ${scenarioMetrics.scenario_count}`, + `- Passed / Failed: ${scenarioMetrics.passed} / ${scenarioMetrics.failed}`, + `- Pass rate: ${Number((scenarioMetrics.pass_rate * 100).toFixed(2))}%`, + `- FP total: ${scenarioMetrics.fp_total}`, + `- FN total: ${scenarioMetrics.fn_total}`, + `- Gate: ${scenarioMetrics.gate_passed ? "PASS" : "FAIL"}`, + "", + "## Public Beta Gate", + "", + `- replay_non_swift_fp_rate_ok: ${gateChecks.replay_non_swift_fp_rate_ok ? "PASS" : "FAIL"}`, + `- replay_swift_detection_rate_ok: ${gateChecks.replay_swift_detection_rate_ok ? "PASS" : "FAIL"}`, + `- replay_overall_match_rate_ok: ${gateChecks.replay_overall_match_rate_ok ? "PASS" : "FAIL"}`, + `- replay_error_pairs_ok: ${gateChecks.replay_error_pairs_ok ? "PASS" : "FAIL"}`, + `- sample_pass_rate_ok: ${gateChecks.sample_pass_rate_ok ? "PASS" : "FAIL"}`, + `- sample_fp_total_ok: ${gateChecks.sample_fp_total_ok ? "PASS" : "FAIL"}`, + `- sample_fn_total_ok: ${gateChecks.sample_fn_total_ok ? "PASS" : "FAIL"}`, + "", + `**Final Gate:** ${gatePassed ? "PASS" : "FAIL"}`, + "", + "## Notes", + "", + "- `warn_swift_no_graph_change` means Swift files changed but graph topology did not change.", + "- In non-strict mode, these warnings are tracked but do not fail the report.", + "", + ].join("\n"); + + ensureParent(args.jsonOut); + ensureParent(args.mdOut); + fs.writeFileSync(args.jsonOut, `${JSON.stringify(output, null, 2)}\n`, "utf8"); + fs.writeFileSync(args.mdOut, `${md}\n`, "utf8"); + + console.log(`json: ${args.jsonOut}`); + console.log(`md: ${args.mdOut}`); + console.log(`public beta gate: ${gatePassed ? "PASS" : "FAIL"}`); +} + +main(); +