autoresearch-problems

A universal, framework-agnostic library of benchmark problems for LLM-driven automated research and program evolution frameworks (FunSearch, ShinkaEvolve, OpenEvolve, BLADE, etc.).

Philosophy

A problem IS its evaluator. The evaluator is fixed. Everything else — the prompt, the candidate program, the candidate's dependencies — is fluid and owned by the research framework. This library owns the evaluator contract and a catalog of problems.

┌──────────────────────────────┐     ┌──────────────────────────────┐
│     PROGRAM SANDBOX          │     │     EVALUATOR SANDBOX        │
│                              │     │                              │
│  LLM-generated code          │     │  Fixed evaluator code        │
│  Unknown/changing deps       │     │  Known, stable deps          │
│  Owned by: research framework│     │  Owned by: this library      │
│                              │     │                              │
│  Runs the candidate program  │────▶│  Receives output             │
│  Returns raw output          │     │  Validates, scores           │
│                              │     │  Returns EvalResult          │
└──────────────────────────────┘     └──────────────────────────────┘

The library does not run the candidate program. It only evaluates the output. Optional program runners are provided as helpers.

Quick Start

pip install autoresearch-problems

from autoresearch_problems import registry

# List all available problems
registry.list_problems()
# ['combinatorics/cap_set', 'combinatorics/online_bin_packing', 'geometry/circle_packing']

# Load a problem spec
spec = registry.load("combinatorics/cap_set")
print(spec.description)
print(spec.initial_prompt)   # suggested LLM prompt
print(spec.initial_program)  # seed solution

# Evaluate a candidate output
import numpy as np
S = np.array([[0, 0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0, 0]])
result = spec.evaluate(S)
print(result.score, result.valid)

Problem Catalog

Problem ID	Category	Description
combinatorics/cap_set	Combinatorics	Largest subset of F_3^n with no three-term arithmetic progression
combinatorics/online_bin_packing	Combinatorics	Online bin-packing heuristic (minimise bins used)
geometry/circle_packing	Geometry	Pack n circles in a unit square (maximise min pairwise distance)
geometry/minimizing_max_min_dist_2d	Geometry	Place 16 points in 2D to maximise ratio of min to max pairwise distance
geometry/minimizing_max_min_dist_3d	Geometry	Place 14 points in 3D to maximise ratio of min to max pairwise distance
analysis/erdos_min_overlap	Analysis	Minimise the Erdős minimum overlap constant C₅ via step function h: [0,2]→[0,1]
analysis/first_autocorr_ineq	Analysis	Minimise the first autocorrelation inequality constant C₁
analysis/second_autocorr_ineq	Analysis	Maximise the second autocorrelation inequality lower bound C₂
analysis/third_autocorr_ineq	Analysis	Minimise the third autocorrelation inequality constant C₃

Optional: Running Untrusted Code

The library ships a SubprocessRunner that executes LLM-generated code in an isolated subprocess:

pip install "autoresearch-problems[runners]"

from autoresearch_problems.program_runners import SubprocessRunner

runner = SubprocessRunner(timeout=30.0)
output = runner.execute(code=spec.initial_program, function_name=spec.function_name)
result = spec.evaluate(output)

Adding a New Problem

1. Create a directory: src/autoresearch_problems/problems/<category>/<name>/
2. Add spec.yaml with metadata:

name: my_problem
category: my_category
description: "..."
function_name: solve
output_type: numpy_array
evaluator_entrypoint: evaluate
evaluator_dependencies:
  - "numpy>=1.24"
parameters:
  n: 10
timeout_seconds: 30.0
maximize: true

3. Add evaluator.py with an evaluate(output, **parameters) -> EvalResult function.
4. Optionally add initial_prompt.md and initial_program.py.

The registry auto-discovers problems by scanning for spec.yaml files.

Data Models

EvalResult

@dataclass
class EvalResult:
    score: float          # primary metric
    valid: bool           # did the output meet constraints?
    execution_time: float = 0.0
    error: str = ""
    metrics: dict = field(default_factory=dict)

ProblemSpec

@dataclass(frozen=True)
class ProblemSpec:
    name: str
    category: str
    description: str
    function_name: str
    output_type: str
    evaluator_code: str
    evaluator_entrypoint: str
    evaluator_dependencies: list[str]
    parameters: dict
    timeout_seconds: float
    maximize: bool
    known_best_score: float | None
    initial_prompt: str | None
    initial_program: str | None
    source: str
    tags: list[str]

    def evaluate(self, output) -> EvalResult: ...

License

MIT