Measured performance

Benchmarks

Latest full benchmark results from solverforge-bench, grouped by problem and labeled with the versions actually tested.

Latest completed full canonical nightly runs, with the versions actually tested. Quick and partial runs are excluded; these are not measurements of newer releases.

Read the gates from left to right: 1s → 10s → 60s. First, compare how many instances reach a valid solution at the earliest tested gate. Then compare quality at that same gate; later gates show improvement. A short process runtime is not a win.

Each gate ranks feasibility → earlier-gate feasibility → solution quality, never process duration. Among solvers with equal current coverage, more valid solutions at 1s wins first, then at 10s where applicable; quality breaks the remaining tie. 0% gap matches the reference. Without references, lower validated mean cost is shown instead. Quality averages cover feasible results only; different solved subsets are not a like-for-like quality comparison. Problems are compared separately.

Capacitated vehicle routing

Assign customers to vehicle routes while respecting capacity. Lower route cost is better.

CVRPLIB-X · 100 instances · 2100 recorded results · Completed

Reference values for all 100 instances, from CVRPLIB (Uchoa et al. 2017 Set X): proven optima where the source closed the instance, best known bounds otherwise.

Earliest feasible gate first; compare feasibility and quality within each gate. Runtime does not rank solutions.

1-second budget 10-second budget 60-second budget

Feasible results

Instances solved to a hard-feasible solution, out of 100.

1-second budget

  1. pyvrp 1s 100/100
  2. solverforge 1s 100/100
  3. ortools 1s 100/100
  4. pyhygese 1s 98/100
  5. solverforge-py 1s 59/100
  6. timefold 1s 13/100
  7. highs 1s 1/100

10-second budget

  1. pyvrp 10s 100/100
  2. solverforge 10s 100/100
  3. ortools 10s 100/100
  4. timefold 10s 100/100
  5. pyhygese 10s 99/100
  6. solverforge-py 10s 57/100
  7. highs 10s 10/100

60-second budget

  1. pyvrp 60s 100/100
  2. solverforge 60s 100/100
  3. ortools 60s 100/100
  4. timefold 60s 100/100
  5. pyhygese 60s 99/100
  6. solverforge-py 60s 76/100
  7. highs 60s 29/100

First feasible gate

Instances first observed feasible at this tested budget, out of 100. Each instance counts once: earlier is better. Independent budget runs, not first-solution timestamps; late returns are retained. Never feasible: pyvrp 0; solverforge 0; ortools 0; pyhygese 0; solverforge-py 24; timefold 0; highs 71.

1-second budget

  1. pyvrp 1s 100/100
  2. solverforge 1s 100/100
  3. ortools 1s 100/100
  4. pyhygese 1s 98/100
  5. solverforge-py 1s 59/100
  6. timefold 1s 13/100
  7. highs 1s 1/100

10-second budget

  1. pyvrp 10s 0/100
  2. solverforge 10s 0/100
  3. ortools 10s 0/100
  4. timefold 10s 87/100
  5. pyhygese 10s 1/100
  6. solverforge-py 10s 0/100
  7. highs 10s 9/100

60-second budget

  1. pyvrp 60s 0/100
  2. solverforge 60s 0/100
  3. ortools 60s 0/100
  4. timefold 60s 0/100
  5. pyhygese 60s 1/100
  6. solverforge-py 60s 17/100
  7. highs 60s 19/100

Mean gap to reference

Feasible results with a reference cost only. 0% matches the reference; lower is better. Rows follow feasibility-first ranking, not gap alone. Coverage and sample counts are shown on every row; means over different solved subsets are not like-for-like. Bars are scaled to this problem's largest mean gap, 679.96%. Subpixel gaps use a one-pixel mark; exact values are shown beside the bars.

1-second budget

  1. pyvrp 1s 3.09% 100/100 feasible · 100 gap samples
  2. solverforge 1s 5.76% 100/100 feasible · 100 gap samples
  3. ortools 1s 10.48% 100/100 feasible · 100 gap samples
  4. pyhygese 1s 2.60% 98/100 feasible · 98 gap samples
  5. solverforge-py 1s 5.96% 59/100 feasible · 59 gap samples
  6. timefold 1s 80.52% 13/100 feasible · 13 gap samples
  7. highs 1s 226.22% 1/100 feasible · 1 gap samples

10-second budget

  1. pyvrp 10s 1.22% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100
  2. solverforge 10s 5.10% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100
  3. ortools 10s 7.00% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100
  4. timefold 10s 57.84% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 13/100
  5. pyhygese 10s 1.61% 99/100 feasible · 99 gap samples Earlier feasibility: 1s 98/100
  6. solverforge-py 10s 5.77% 57/100 feasible · 57 gap samples Earlier feasibility: 1s 59/100
  7. highs 10s 644.32% 10/100 feasible · 10 gap samples Earlier feasibility: 1s 1/100

60-second budget

  1. pyvrp 60s 0.59% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100 · 10s 100/100
  2. solverforge 60s 4.99% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100 · 10s 100/100
  3. ortools 60s 5.62% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100 · 10s 100/100
  4. timefold 60s 19.02% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 13/100 · 10s 100/100
  5. pyhygese 60s 1.07% 99/100 feasible · 99 gap samples Earlier feasibility: 1s 98/100 · 10s 99/100
  6. solverforge-py 60s 5.56% 76/100 feasible · 76 gap samples Earlier feasibility: 1s 59/100 · 10s 57/100
  7. highs 60s 679.96% 29/100 feasible · 29 gap samples Earlier feasibility: 1s 1/100 · 10s 10/100

1-second budget

Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
pyvrp 0.14.1 100 / 100 3.09% (100 samples) 65153.04 (100 samples) 1.59 s 1.59 s 93 / 100
solverforge 0.22.0 100 / 100 5.76% (100 samples) 67114.99 (100 samples) 1.05 s 1.05 s 5 / 100
ortools 9.15.6755 100 / 100 10.48% (100 samples) 69785.72 (100 samples) 1.14 s 1.14 s 36 / 100
pyhygese 0.1.0 98 / 100 2.60% (98 samples) 63179.00 (98 samples) 1.06 s 1.06 s 5 / 100
solverforge-py 0.6.15 59 / 100 5.96% (59 samples) 42047.97 (59 samples) 2.76 s 4.14 s 100 / 100
timefold 2.7.1 13 / 100 80.52% (13 samples) 38852.23 (13 samples) 2.57 s 2.30 s 100 / 100
highs 1.15.1 1 / 100 226.22% (1 samples) 90008.00 (1 samples) 1.44 s 2.48 s 100 / 100
10-second budget
Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
pyvrp 0.14.1 100 / 100 1.22% (100 samples) 64042.87 (100 samples) 10.58 s 10.58 s 21 / 100
solverforge 0.22.0 100 / 100 5.10% (100 samples) 65969.66 (100 samples) 10.04 s 10.04 s 0 / 100
ortools 9.15.6755 100 / 100 7.00% (100 samples) 67011.47 (100 samples) 10.11 s 10.11 s 0 / 100
timefold 2.7.1 100 / 100 57.84% (100 samples) 97639.29 (100 samples) 11.25 s 11.25 s 100 / 100
pyhygese 0.1.0 99 / 100 1.61% (99 samples) 63600.33 (99 samples) 10.08 s 10.08 s 0 / 100
solverforge-py 0.6.15 57 / 100 5.77% (57 samples) 42333.82 (57 samples) 11.62 s 13.16 s 80 / 100
highs 1.15.1 10 / 100 644.32% (10 samples) 140288.40 (10 samples) 10.22 s 14.02 s 89 / 100
60-second budget
Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
pyvrp 0.14.1 100 / 100 0.59% (100 samples) 63519.30 (100 samples) 60.59 s 60.59 s 0 / 100
solverforge 0.22.0 100 / 100 4.99% (100 samples) 65892.81 (100 samples) 60.04 s 60.04 s 0 / 100
ortools 9.15.6755 100 / 100 5.62% (100 samples) 66468.01 (100 samples) 60.11 s 60.11 s 0 / 100
timefold 2.7.1 100 / 100 19.02% (100 samples) 75240.71 (100 samples) 61.25 s 61.25 s 0 / 100
pyhygese 0.1.0 99 / 100 1.07% (99 samples) 64472.25 (99 samples) 60.38 s 60.53 s 1 / 100
solverforge-py 0.6.15 76 / 100 5.56% (76 samples) 51908.25 (76 samples) 63.39 s 66.30 s 38 / 100
highs 1.15.1 29 / 100 679.96% (29 samples) 185048.86 (29 samples) 60.55 s 70.27 s 69 / 100
Run provenance

Full canonical nightly candidate · completed 2026-10-19T00:26:12.477365+00:00 (UTC). The warehouse's publication checks passed; the benchmark repository was clean.

Run 03ef371a-a0a2-42ff-af47-87d73d5963a7
Harness commit 2f71da41a823.

Employee scheduling

Assign nurses to shifts while respecting hard rules. Among feasible schedules, lower penalty cost is better.

INRC-II · 37 instances · 555 recorded results · Completed

Reference values for all 37 instances, from INRC-II official test dataset (mobiz.vives.be/inrc2): best known bounds, not proven optima.

Earliest feasible gate first; compare feasibility and quality within each gate. Runtime does not rank solutions.

1-second budget 10-second budget 60-second budget

Feasible results

Instances solved to a hard-feasible solution, out of 37.

1-second budget

  1. solverforge 1s 35/37
  2. solverforge-py 1s 30/37
  3. highs 1s 11/37
  4. timefold 1s 5/37
  5. ortools 1s 1/37

10-second budget

  1. solverforge 10s 37/37
  2. timefold 10s 36/37
  3. solverforge-py 10s 34/37
  4. highs 10s 30/37
  5. ortools 10s 3/37

60-second budget

  1. solverforge 60s 37/37
  2. solverforge-py 60s 37/37
  3. timefold 60s 37/37
  4. highs 60s 31/37
  5. ortools 60s 8/37

First feasible gate

Instances first observed feasible at this tested budget, out of 37. Each instance counts once: earlier is better. Independent budget runs, not first-solution timestamps; late returns are retained. Never feasible: solverforge 0; solverforge-py 0; highs 5; timefold 0; ortools 29.

1-second budget

  1. solverforge 1s 35/37
  2. solverforge-py 1s 30/37
  3. highs 1s 11/37
  4. timefold 1s 5/37
  5. ortools 1s 1/37

10-second budget

  1. solverforge 10s 2/37
  2. timefold 10s 31/37
  3. solverforge-py 10s 4/37
  4. highs 10s 19/37
  5. ortools 10s 2/37

60-second budget

  1. solverforge 60s 0/37
  2. solverforge-py 60s 3/37
  3. timefold 60s 1/37
  4. highs 60s 2/37
  5. ortools 60s 5/37

Mean gap to reference

Feasible results with a reference cost only. 0% matches the reference; lower is better. Rows follow feasibility-first ranking, not gap alone. Coverage and sample counts are shown on every row; means over different solved subsets are not like-for-like. Bars are scaled to this problem's largest mean gap, 3423.96%. Subpixel gaps use a one-pixel mark; exact values are shown beside the bars.

1-second budget

  1. solverforge 1s 2030.43% 35/37 feasible · 35 gap samples
  2. solverforge-py 1s 2042.26% 30/37 feasible · 30 gap samples
  3. highs 1s 2021.44% 11/37 feasible · 11 gap samples
  4. timefold 1s 129.44% 5/37 feasible · 5 gap samples
  5. ortools 1s 118.41% 1/37 feasible · 1 gap samples

10-second budget

  1. solverforge 10s 1532.04% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 35/37
  2. timefold 10s 543.08% 36/37 feasible · 36 gap samples Earlier feasibility: 1s 5/37
  3. solverforge-py 10s 1919.35% 34/37 feasible · 34 gap samples Earlier feasibility: 1s 30/37
  4. highs 10s 2979.25% 30/37 feasible · 30 gap samples Earlier feasibility: 1s 11/37
  5. ortools 10s 76.25% 3/37 feasible · 3 gap samples Earlier feasibility: 1s 1/37

60-second budget

  1. solverforge 60s 782.36% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 35/37 · 10s 37/37
  2. solverforge-py 60s 1324.56% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 30/37 · 10s 34/37
  3. timefold 60s 450.20% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 5/37 · 10s 36/37
  4. highs 60s 3423.96% 31/37 feasible · 31 gap samples Earlier feasibility: 1s 11/37 · 10s 30/37
  5. ortools 60s 374.63% 8/37 feasible · 8 gap samples Earlier feasibility: 1s 1/37 · 10s 3/37

1-second budget

Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
solverforge 0.22.0 35 / 37 2030.43% (35 samples) 61375.86 (35 samples) 1.12 s 1.12 s 8 / 37
solverforge-py 0.6.15 30 / 37 2042.26% (30 samples) 59948.50 (30 samples) 2.25 s 2.81 s 35 / 37
highs 1.15.1 11 / 37 2021.44% (11 samples) 40833.64 (11 samples) 1.82 s 1.79 s 34 / 37
timefold 2.7.1 5 / 37 129.44% (5 samples) 5052.00 (5 samples) 2.53 s 2.75 s 37 / 37
ortools 9.15.6755 1 / 37 118.41% (1 samples) 4390.00 (1 samples) 1.05 s 1.41 s 28 / 37
10-second budget
Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
solverforge 0.22.0 37 / 37 1532.04% (37 samples) 49759.19 (37 samples) 10.10 s 10.10 s 0 / 37
timefold 2.7.1 36 / 37 543.08% (36 samples) 18773.06 (36 samples) 11.71 s 11.83 s 37 / 37
solverforge-py 0.6.15 34 / 37 1919.35% (34 samples) 58656.32 (34 samples) 11.32 s 11.65 s 20 / 37
highs 1.15.1 30 / 37 2979.25% (30 samples) 78293.83 (30 samples) 10.38 s 11.11 s 6 / 37
ortools 9.15.6755 3 / 37 76.25% (3 samples) 3171.67 (3 samples) 10.06 s 10.17 s 1 / 37
60-second budget
Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
solverforge 0.22.0 37 / 37 782.36% (37 samples) 27426.62 (37 samples) 60.10 s 60.10 s 0 / 37
solverforge-py 0.6.15 37 / 37 1324.56% (37 samples) 43554.46 (37 samples) 61.67 s 61.67 s 2 / 37
timefold 2.7.1 37 / 37 450.20% (37 samples) 16417.97 (37 samples) 61.89 s 61.89 s 1 / 37
highs 1.15.1 31 / 37 3423.96% (31 samples) 95461.45 (31 samples) 60.57 s 61.04 s 1 / 37
ortools 9.15.6755 8 / 37 374.63% (8 samples) 12726.88 (8 samples) 60.10 s 60.36 s 0 / 37
Run provenance

Full canonical nightly candidate · completed 2026-10-18T13:21:26.683904+00:00 (UTC). The warehouse's publication checks passed; the benchmark repository was clean.

Run f3c914f9-9695-446c-aab3-1241b7249021
Harness commit 2f71da41a823.

Job-shop scheduling

Sequence jobs on machines while respecting operation order and machine capacity. Lower makespan is better.

JSPLIB · 162 instances · 2430 recorded results · Completed

Reference values for all 162 instances, from ScheduleOpt/benchmarks: proven optima where the source closed the instance, best known bounds otherwise.

Earliest feasible gate first; compare feasibility and quality within each gate. Runtime does not rank solutions.

1-second budget 10-second budget 60-second budget

Feasible results

Instances solved to a hard-feasible solution, out of 162.

1-second budget

  1. solverforge 1s 162/162
  2. solverforge-py 1s 162/162
  3. ortools 1s 161/162
  4. timefold 1s 100/162
  5. highs 1s 34/162

10-second budget

  1. solverforge 10s 162/162
  2. solverforge-py 10s 162/162
  3. ortools 10s 162/162
  4. timefold 10s 159/162
  5. highs 10s 90/162

60-second budget

  1. solverforge 60s 162/162
  2. solverforge-py 60s 162/162
  3. ortools 60s 162/162
  4. timefold 60s 162/162
  5. highs 60s 122/162

First feasible gate

Instances first observed feasible at this tested budget, out of 162. Each instance counts once: earlier is better. Independent budget runs, not first-solution timestamps; late returns are retained. Never feasible: solverforge 0; solverforge-py 0; ortools 0; timefold 0; highs 40.

1-second budget

  1. solverforge 1s 162/162
  2. solverforge-py 1s 162/162
  3. ortools 1s 161/162
  4. timefold 1s 100/162
  5. highs 1s 34/162

10-second budget

  1. solverforge 10s 0/162
  2. solverforge-py 10s 0/162
  3. ortools 10s 1/162
  4. timefold 10s 59/162
  5. highs 10s 56/162

60-second budget

  1. solverforge 60s 0/162
  2. solverforge-py 60s 0/162
  3. ortools 60s 0/162
  4. timefold 60s 3/162
  5. highs 60s 32/162

Mean gap to reference

Feasible results with a reference cost only. 0% matches the reference; lower is better. Rows follow feasibility-first ranking, not gap alone. Coverage and sample counts are shown on every row; means over different solved subsets are not like-for-like. Bars are scaled to this problem's largest mean gap, 35.55%. Subpixel gaps use a one-pixel mark; exact values are shown beside the bars.

1-second budget

  1. solverforge 1s 14.12% 162/162 feasible · 162 gap samples
  2. solverforge-py 1s 14.25% 162/162 feasible · 162 gap samples
  3. ortools 1s 12.16% 161/162 feasible · 161 gap samples
  4. timefold 1s 30.30% 100/162 feasible · 100 gap samples
  5. highs 1s 21.31% 34/162 feasible · 34 gap samples

10-second budget

  1. solverforge 10s 12.26% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162
  2. solverforge-py 10s 12.65% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162
  3. ortools 10s 8.17% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 161/162
  4. timefold 10s 26.96% 159/162 feasible · 159 gap samples Earlier feasibility: 1s 100/162
  5. highs 10s 32.76% 90/162 feasible · 90 gap samples Earlier feasibility: 1s 34/162

60-second budget

  1. solverforge 60s 10.75% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162 · 10s 162/162
  2. solverforge-py 60s 10.88% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162 · 10s 162/162
  3. ortools 60s 5.83% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 161/162 · 10s 162/162
  4. timefold 60s 13.85% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 100/162 · 10s 159/162
  5. highs 60s 35.55% 122/162 feasible · 122 gap samples Earlier feasibility: 1s 34/162 · 10s 90/162

1-second budget

Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
solverforge 0.22.0 162 / 162 14.12% (162 samples) 2108.77 (162 samples) 1.02 s 1.02 s 2 / 162
solverforge-py 0.6.15 162 / 162 14.25% (162 samples) 2110.47 (162 samples) 1.05 s 1.05 s 8 / 162
ortools 9.15.6755 161 / 162 12.16% (161 samples) 2054.78 (161 samples) 0.93 s 0.93 s 1 / 162
timefold 2.7.1 100 / 162 30.30% (100 samples) 1592.74 (100 samples) 1.94 s 1.94 s 162 / 162
highs 1.15.1 34 / 162 21.31% (34 samples) 1097.62 (34 samples) 1.00 s 1.22 s 49 / 162
10-second budget
Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
solverforge 0.22.0 162 / 162 12.26% (162 samples) 2079.57 (162 samples) 10.01 s 10.01 s 0 / 162
solverforge-py 0.6.15 162 / 162 12.65% (162 samples) 2085.12 (162 samples) 10.05 s 10.05 s 0 / 162
ortools 9.15.6755 162 / 162 8.17% (162 samples) 2017.97 (162 samples) 8.23 s 8.23 s 0 / 162
timefold 2.7.1 159 / 162 26.96% (159 samples) 2435.41 (159 samples) 10.94 s 10.94 s 24 / 162
highs 1.15.1 90 / 162 32.76% (90 samples) 1582.02 (90 samples) 9.95 s 10.09 s 8 / 162
60-second budget
Solver / tested versionFeasibleMean gapMean costRuntime (feasible)Mean runtimeOver budget
solverforge 0.22.0 162 / 162 10.75% (162 samples) 2055.91 (162 samples) 60.02 s 60.02 s 0 / 162
solverforge-py 0.6.15 162 / 162 10.88% (162 samples) 2057.80 (162 samples) 60.05 s 60.05 s 0 / 162
ortools 9.15.6755 162 / 162 5.83% (162 samples) 1976.46 (162 samples) 44.48 s 44.48 s 0 / 162
timefold 2.7.1 162 / 162 13.85% (162 samples) 2260.70 (162 samples) 60.94 s 60.94 s 0 / 162
highs 1.15.1 122 / 162 35.55% (122 samples) 1850.55 (122 samples) 57.56 s 58.22 s 0 / 162
Run provenance

Full canonical nightly candidate · completed 2026-10-19T00:50:44.168746+00:00 (UTC). The warehouse's publication checks passed; the benchmark repository was clean.

Run e9e294de-0d5c-4db2-a1eb-c77f23171734
Harness commit 2f71da41a823.

Evidence and methodology

Gates are independent runs with requested budgets, not samples of one continuous search. First feasible gate means the earliest tested budget that returned a hard-feasible result for that instance; it is not the instant feasibility was discovered. No first-incumbent timestamps were recorded.

Runtime is measured wall-clock duration, shown only as a diagnostic. “Over budget” counts results flagged by the harness’s wall-time tolerance; late returned solutions are retained, so these gates are not strict deadline guarantees. The tables start at 1s; expand the later gates to inspect improvement.

Download the recorded results and run metadata (JSON) · How we benchmark SolverForge · Benchmark source