feat: add Verilog/SystemVerilog language support with VerilogEval and RT-LLM Benchmark

[Tyronita]

Summary

Adds first-class Verilog/SystemVerilog language support to ShinkaEvolve, with two hardware design benchmarks:

• VerilogEval (156 HDLBits problems) — uses iverilog for fast local compilation + simulation, continuous scoring via mismatch ratio
• CVDP (302 NVIDIA problems) — uses Docker/CocoTB for pytest-based evaluation, continuous scoring via pass ratio

Both benchmarks support single-problem quick-start (run_evo.py) and full-sweep parallel execution (run_all.py with ProcessPoolExecutor). The UCB bandit dynamically allocates across a configurable model mix.

Core changes (shinka/)

• shinka/utils/languages.py — add "verilog" language with .sv extension and // EVOLVE-BLOCK markers
• shinka/edit/async_apply.py — map Verilog to the same comment-style edit as C/Rust
• shinka/llm/client.py — fix Azure endpoint construction (was doubling /openai/ path)

New examples

Directory	Problems	Evaluator	Requirements
examples/verilog_eval/	156 (HuggingFace download)	iverilog -g2012 + vvp	iverilog v11+
examples/cvdp/	302 (HuggingFace download)	Docker + CocoTB pytest	Docker

Each example includes:

• evaluate.py — ShinkaEvolve evaluator contract (compile → simulate → score)
• run_evo.py — single-problem quick-start
• run_all.py — parallel batch runner for the full benchmark
• download_dataset.py — fetch dataset from HuggingFace
• initial.sv — default seed module
• README.md — setup instructions and problem taxonomy

Architecture decisions

• Continuous scoring: VerilogEval uses (1 - mismatches/total_samples) * 100, CVDP uses (passed_tests/total_tests) * 100. Both give the LLM gradient signal even on partial solutions.
• Per-problem isolation: run_all.py spawns each problem as a subprocess with its own env vars, avoiding state leakage between evolution runs.
• Self-contained JSONL: Both benchmarks embed testbenches and references directly in the JSONL, eliminating external repo dependencies.
• Model mix: Defaults to azure-gpt-4-1-mini + azure-gpt-5-codex + azure-deepseek-v4-flash for the UCB bandit. Reasoning models (o4-mini) excluded from defaults since they don't support the temperature parameter.

Benchmark Results (in progress)

Running 30-generation sweeps on an Azure A100 VM (24 vCPU, 216GB RAM):

• VerilogEval: 156 problems × 30 gens × 3 models, 4 parallel workers
• CVDP: 302 problems × 30 gens × 3 models, 2 parallel workers

Early results (first ~10 VerilogEval problems): 100% solve rate on easy problems (zero, wire, not gate, popcount3), partial scores on medium problems.

Full results will be published to a separate results repository once the sweep completes.

Test plan

• pytest tests/test_edit_verilog.py — Verilog marker detection and edit application
• End-to-end VerilogEval: download → seed generation → evaluation → evolution (tested locally and on VM)
• End-to-end CVDP: Docker build → evaluation → evolution (tested on VM)
• Multi-model bandit selection working (gpt-4.1-mini, gpt-5-codex, deepseek-v4-flash)
• Full benchmark sweeps running on Azure A100 VM (156 + 302 problems)
• Collect final benchmark statistics after sweeps complete

🤖 Generated with Claude Code

[RobertTLange]

Thanks for the work here and for your interest in extending ShinkaEvolve to Verilog/SystemVerilog. The direction is useful, and the evaluator examples are a good starting point.

I found a few issues that should be addressed before merge:

1. CI currently fails on ruff

   • tests/test_cvdp_evaluator.py imports unused json and re.
   • tests/test_verilog_eval_ci.py imports unused sys.
   • This reproduces locally with:
     uv run python -m ruff check tests --exclude tests/file.py

2. The CVDP evaluator trusts paths from the JSONL harness too much

   • examples/cvdp/evaluate.py writes problem["harness"]["files"] paths directly under the temp workspace, but absolute paths or .. entries could escape that workspace.
   • _extract_rtl_path() also trusts VERILOG_SOURCES; if that path is absolute or traverses upward, the candidate RTL can be written outside the temp dir.
   • Since the evaluator then runs the provided docker-compose.yml, this is a meaningful trust boundary. Please normalize/resolve all harness and RTL paths and require them to stay inside the temp workspace, or clearly restrict/document this evaluator as trusted official-dataset-only.

3. There are unwanted docs artifacts in this PR

   • VERIFICATION.md looks like a temporary PR verification note and should not be committed.
   • The added planning/survey docs also look broader than the implementation change and likely should be removed unless maintainers explicitly want them:
     docs/cvdp_integration_plan.md, docs/verilog_benchmarks_survey.md, docs/verilog_eval_benchmark.md.

One question: were both examples run and validated end-to-end, including at least one full examples/verilog_eval evolution/evaluation path and one full examples/cvdp Docker/CocoTB evaluation with the required CVDP image?

I did run the targeted tests with:
uv run python -m pytest -q tests/test_edit_verilog.py tests/test_cvdp_evaluator.py tests/test_verilog_eval_ci.py

That passed locally: 9 passed, 4 skipped. Mypy also passed for the CI command. The current blocker is ruff plus the CVDP path/compose trust issue.

[RobertTLange]

Thanks again for putting this together.

We would love to support Verilog/SystemVerilog as first-class ShinkaEvolve languages. The core language support pieces look directionally useful.

That said, we do not want to overload the main repository with a large amount of benchmark-specific logic. The VerilogEval/CVDP runners, dataset downloaders, full-sweep orchestration, planning docs, and benchmark harness code make this PR much larger than the language-support change itself.

Could you clean up the PR so it focuses on the minimal reusable Verilog/SystemVerilog support in ShinkaEvolve? Ideally that would include:

• core language/marker support
• focused edit/apply tests
• a small, self-contained example or smoke test if needed
• removal of benchmark sweep infrastructure, temporary verification/planning docs, and unrelated changes

The larger VerilogEval/CVDP benchmark logic may be better kept in a separate example repository or follow-up contribution once the core language support lands.

[Tyronita]

Good Morning Tokyo! @RobertTLange

For the 45 of 50 problems listed in RT-LLM that can be synthesied with open source EDA tools we have 50 generations of results each:

(WIP) results will be published here:
https://github.com/Tyronita/RTLLM-ShinkaEvolve-results/

I noticed some reard hacking due to EVOLVE BLOCK placement, so I'm, re-running affected reference designs.

HF Dataset Published for traces:
https://huggingface.co/datasets/EvanOLeary/rtllm-shinka-evolve

(thank you for the Cuda-Engineer Traces - I post trained a small LM and came 2nd in a Nvidia Hackathon using Sakana AI's datasets - https://www.linkedin.com/feed/update/urn:li:activity:7467348291291291648/)

The rest of the PR should be good to review :)

Best,

Evan

[Tyronita]

Hey @RobertTLange — the PR's trimmed to the minimal Verilog/SystemVerilog support + one self-contained example, and it's green locally (ruff / mypy / pytest). The full PPA results live in a gist so they don't clutter the PR:

→ Results: https://gist.github.com/Tyronita/7e4c6959fc8609fa0d07f5663ebdfeb8

30-second summary

27 of 45 in-scope RTLLM designs beat the human reference (PPA = 100 · geomean(area, depth, power); 100 = the reference). Correctness is held by a Yosys SAT equivalence gate + an interface freeze + a code-review pass — which caught and rejected 4 testbench-overfit "wins". The gist has per-design code diffs, a worked example, a 45-design graph collage, and a HuggingFace dataset of every candidate.

Sidenote: the results and all their tooling are kept entirely out of this PR — it's just the language support + the one example.