solverforge-bench

solverforge-bench is the official SolverForge benchmark surface. It groups benchmarks by SolverForge solve shape and runs them through one shared Python harness so timing, watchdog handling, result rows, logging, CSV output, and optional PostgreSQL persistence stay comparable across problem families.

Problem packages are adapters. They load cases, build solver callables, validate returned solutions, compute benchmark-specific fields, and hand the result back to the shared framework.

What It Provides

• List-variable CVRP benchmarks under list-variable/cvrp/, exercising SolverForge route-list solving on vehicle-routing instances
• Scalar-variable employee scheduling benchmarks under scalar-variable/employee-scheduling/, using the bundled INRC-II nurse rostering corpus
• Scalar-variable job-shop scheduling benchmarks under scalar-variable/job-shop-scheduling/, using the bundled JSPLIB instances
• One shared harness for solver registration, run matrices, wall-clock timing, overshoot calculation, watchdog containment, result rows, CSV files, run logs, and solver stdout/stderr capture
• TOML configuration for benchmark selection, solvers, time limits, run kind, logging, watchdog settings, and PostgreSQL persistence
• PostgreSQL warehouse output for cron-driven and release-tagged runs, backed by checked-in SQL migrations and latest-run views

Prerequisites

The validation targets need only Python. The benchmark targets also build the native SolverForge adapters and every comparison adapter in the default solver set, so a full run needs:

• python3.14 (the Makefile’s HOST_PYTHON) for the repository .venv
• a Rust toolchain with cargo, and maturin on PATH, for the native SolverForge adapters
• a JDK with mvn for the JVM-based comparison adapters
• cmake and a C++ compiler for the native comparison adapters; curl and git download their sources and release archives
• taskset (util-linux): benchmark runs pin each suite to its own CPU core
• for the -db targets, a PostgreSQL server, psql, and sqlx (cargo install sqlx-cli --no-default-features --features postgres)

The *-solverforge-* targets build only the native SolverForge adapter, so they need Python, Rust with maturin, and taskset.

Setup

Create the repository virtualenv with Python 3.14:

python3.14 -m venv .venv
. .venv/bin/activate
pip install -e .

The root Makefile uses the same .venv for CVRP, employee scheduling, job-shop scheduling, normalization, and nightly runs. make install-python-deps creates or refreshes it and installs the pinned published solverforge Python package into it.

Benchmark Families

CVRP is the canonical list-variable benchmark. Employee scheduling and job-shop scheduling are the canonical scalar-variable benchmarks: the first covers nurse-to-shift assignment on INRC-II instances, the second covers operation sequencing on JSPLIB instances.

Common Commands

Validate adapters and reference data before treating a benchmark run as evidence:

make validate-cvrp
make validate-employee-scheduling
make validate-employee-model-parity
make validate-job-shop-scheduling

Run quick smoke benchmarks:

make bench-cvrp-quick
make bench-cvrp-solverforge-quick
make bench-employee-scheduling-quick
make bench-employee-scheduling-solverforge-quick
make bench-job-shop-scheduling-quick
make bench-job-shop-scheduling-solverforge-quick

Run canonical benchmark sets:

make bench-cvrp
make bench-employee-scheduling
make bench-job-shop-scheduling

The quick CVRP target runs three instances at 1 and 10 seconds. The quick employee-scheduling target runs n005w4 at 1 and 10 seconds. The quick job-shop target runs the JSPLIB quick group (ft06 and la01) at 1 and 10 seconds. The canonical employee-scheduling and job-shop targets use the canonical group from their bundled manifests (INRC-II and all 162 JSPLIB instances) and the 1, 10, and 60 second budgets. The *-solverforge-* targets run the same slices with only native SolverForge.

Unified Harness

The Make targets call the same root benchmark entrypoint:

PYTHONPATH=src:list-variable/cvrp/src:scalar-variable/employee-scheduling/src:scalar-variable/job-shop-scheduling/src \
  .venv/bin/python3 scripts/run_benchmark.py cvrp \
  --run-kind quick \
  --num-instances 3 \
  --time-limits 1 10

PYTHONPATH=src:list-variable/cvrp/src:scalar-variable/employee-scheduling/src:scalar-variable/job-shop-scheduling/src \
  .venv/bin/python3 scripts/run_benchmark.py employee-scheduling \
  --run-kind quick \
  --datasets n005w4 \
  --time-limits 1 10

PYTHONPATH=src:list-variable/cvrp/src:scalar-variable/employee-scheduling/src:scalar-variable/job-shop-scheduling/src \
  .venv/bin/python3 scripts/run_benchmark.py job-shop-scheduling \
  --run-kind quick \
  --dataset-set quick \
  --time-limits 1 10

Shared options include:

--config CONFIG
--solver SOLVER
--time-limits SECONDS...
--run-kind quick|candidate|tag
--nightly | --no-nightly
--release-tag TAG
--save-postgres | --no-save-postgres
--postgres-url URL
--log-level LEVEL
--show-solver-output | --no-show-solver-output
--capture-solver-output | --no-capture-solver-output

cvrp adds --num-instances. employee-scheduling and job-shop-scheduling add --dataset-set and --datasets.

Timing Contract

The requested time limit is passed to the solver and measured with wall-clock timing around the solver call. The watchdog is separate: it exists to contain runaway invocations, not to discard every solution that returns slightly after the nominal budget.

The harness records:

overshoot_seconds = max(0, actual_time_seconds - time_limit_seconds)
overshoot_ratio = overshoot_seconds / time_limit_seconds
wall_time_over_limit = actual_time_seconds > time_limit_seconds * 1.1

Only watchdog-killed invocations lose the returned solution, because the child process was forcibly terminated.

TOML Configuration

Use a TOML file when a run needs to be repeatable outside a Make target:

benchmark = "cvrp"
solver = ["solverforge"]
time_limits = [1]
run_kind = "quick"
nightly = false

[postgres]
save = false
url = "postgresql://postgres@localhost/solverforge_bench"

[logging]
level = "INFO"
show_solver_output = true
capture_solver_output = true

[benchmarks.cvrp]
num_instances = 3

[benchmarks.employee-scheduling]
dataset_set = "quick"
datasets = ["n005w4"]

Command-line options override TOML values. Make targets accept the same file through BENCH_CONFIG:

make bench-cvrp-quick BENCH_CONFIG=benchmark.example.toml

Use benchmark.nightly.example.toml as the cron-oriented template for the combined nightly job.

PostgreSQL Results

Normal benchmark targets write CSV evidence artifacts. The -db targets apply migrations and save the same run to PostgreSQL:

make db-check
make db-create
make db-migrate

make bench-cvrp-quick-db
make bench-employee-scheduling-quick-db
make bench-job-shop-scheduling-quick-db
make bench-nightly-db

make bench-nightly-db is the cron entrypoint. It builds all three benchmark stacks, applies migrations once, then runs full CVRP, canonical employee scheduling, and canonical job-shop scheduling, each pinned to its own core, as one PostgreSQL-saving nightly job.

PostgreSQL stores run catalog data, solver versions, and per-result rows. Warehouse diagnostics read benchmark_result_facts, latest_benchmark_runs, or latest_benchmark_result_facts instead of reconstructing the run/result join. Published results read only publishable_benchmark_runs, publishable_benchmark_result_facts, or their latest_publishable_benchmark_runs and latest_publishable_benchmark_result_facts counterparts: those views admit only completed runs from a clean, identified commit with a complete expected/observed matrix, valid fair-start witnesses, and complete runtime provenance. benchmark_run_publication_audit names the criterion that withheld a run.

Published Results

The results page contains the latest publishable full canonical nightly for each problem, with the complete downloadable evidence. The current cohort completed on October 18–19, 2026 (UTC), from benchmark commit 2f71da41a82308c5f0da2f2c1cea8969991861a1:

These runs tested native SolverForge 0.22.0 and Python 0.6.15; the results page lists every solver and version in each matrix. Each matrix covers the 1, 10, and 60 second budgets. Official references cover every selected instance in all three problems.

Official References

A gap is only meaningful against a stated reference, so every problem resolves its reference from a versioned catalog beside its instances: one mechanism across routing, scheduling, and rostering, instead of one per problem.

Each value carries the source that published it, that source’s revision, and whether the value is a proven optimum or a best known upper bound. Those are different claims, so they are never mixed silently — a solver that matches a proven optimum has no gap, while one that matches a best known bound may still be above the true optimum.

Read tested gates in ascending order: 1s, 10s, 60s. The first feasible gate counts each instance at the earliest tested budget that returned a valid solution. Within each gate, feasibility ranks first, then feasibility coverage at earlier gates in ascending order, then solution quality (reference gap, or validated mean cost without references). Runtime is diagnostic, never a ranking criterion. Separate budget runs are not a continuous search trajectory, and first-incumbent timestamps were not recorded. Late returns remain visible and are flagged; requested budgets are not strict deadline guarantees.

Regenerate a catalog from its source, or check that a committed one still matches the source it claims:

make verify-reference-catalogs   # every catalog reproduces from its source
make load-reference-catalogs     # carry the committed values into PostgreSQL

verify-reference-catalogs fails when a committed value no longer regenerates, so a hand-edited reference cannot reach the warehouse. Loading is idempotent and updates benchmark_reference_catalog; the benchmark_reference_resolved view then reports the effective reference for each result, preferring the pinned official value over whatever a run recorded.

Where no official value exists, the run reports that absence rather than substituting a bound of its own. The employee-scheduling canonical selection grades the exact scenario, history, and ordered-week combinations covered by those 37 official values. Its results report both feasibility and reference gaps. On the published results page the coverage is stated per problem, next to the numbers.

Result Columns

Generated CSV rows use one global snake_case schema. Core columns include:

run_id, benchmark_name, benchmark_category, dataset, dataset_set, instance,
instance_size, solver, solver_version, time_limit_seconds,
actual_time_seconds, overshoot_seconds, overshoot_ratio,
wall_time_over_limit, watchdog_limit_seconds, watchdog_killed, run_error,
solver_stdout_path, solver_stderr_path, hard_feasible, cost, reported_cost,
fresh_cost, reference_cost, quality_ratio, validation_error,
solution_artifact

Benchmark-specific fields such as nurses, weeks, validator_model_delta, and score_drift are preserved as optional native columns.

External References

• GitHub repository
• Benchmark methodology