Measured performance
Benchmarks
Latest full benchmark results from solverforge-bench, grouped by problem and labeled with the versions actually tested.
Latest completed full canonical nightly runs, with the versions actually tested. Quick and partial runs are excluded; these are not measurements of newer releases.
Read the gates from left to right: 1s → 10s → 60s. First, compare how many instances reach a valid solution at the earliest tested gate. Then compare quality at that same gate; later gates show improvement. A short process runtime is not a win.
Each gate ranks feasibility → earlier-gate feasibility → solution quality, never process duration. Among solvers with equal current coverage, more valid solutions at 1s wins first, then at 10s where applicable; quality breaks the remaining tie. 0% gap matches the reference. Without references, lower validated mean cost is shown instead. Quality averages cover feasible results only; different solved subsets are not a like-for-like quality comparison. Problems are compared separately.
Capacitated vehicle routing
Assign customers to vehicle routes while respecting capacity. Lower route cost is better.
CVRPLIB-X · 100 instances · 2100 recorded results · Completed
Reference values for all 100 instances, from CVRPLIB (Uchoa et al. 2017 Set X): proven optima where the source closed the instance, best known bounds otherwise.
Earliest feasible gate first; compare feasibility and quality within each gate. Runtime does not rank solutions.
Feasible results
Instances solved to a hard-feasible solution, out of 100.
1-second budget
- pyvrp 1s 100/100
- solverforge 1s 100/100
- ortools 1s 100/100
- pyhygese 1s 98/100
- solverforge-py 1s 59/100
- timefold 1s 13/100
- highs 1s 1/100
10-second budget
- pyvrp 10s 100/100
- solverforge 10s 100/100
- ortools 10s 100/100
- timefold 10s 100/100
- pyhygese 10s 99/100
- solverforge-py 10s 57/100
- highs 10s 10/100
60-second budget
- pyvrp 60s 100/100
- solverforge 60s 100/100
- ortools 60s 100/100
- timefold 60s 100/100
- pyhygese 60s 99/100
- solverforge-py 60s 76/100
- highs 60s 29/100
First feasible gate
Instances first observed feasible at this tested budget, out of 100. Each instance counts once: earlier is better. Independent budget runs, not first-solution timestamps; late returns are retained. Never feasible: pyvrp 0; solverforge 0; ortools 0; pyhygese 0; solverforge-py 24; timefold 0; highs 71.
1-second budget
- pyvrp 1s 100/100
- solverforge 1s 100/100
- ortools 1s 100/100
- pyhygese 1s 98/100
- solverforge-py 1s 59/100
- timefold 1s 13/100
- highs 1s 1/100
10-second budget
- pyvrp 10s 0/100
- solverforge 10s 0/100
- ortools 10s 0/100
- timefold 10s 87/100
- pyhygese 10s 1/100
- solverforge-py 10s 0/100
- highs 10s 9/100
60-second budget
- pyvrp 60s 0/100
- solverforge 60s 0/100
- ortools 60s 0/100
- timefold 60s 0/100
- pyhygese 60s 1/100
- solverforge-py 60s 17/100
- highs 60s 19/100
Mean gap to reference
Feasible results with a reference cost only. 0% matches the reference; lower is better. Rows follow feasibility-first ranking, not gap alone. Coverage and sample counts are shown on every row; means over different solved subsets are not like-for-like. Bars are scaled to this problem's largest mean gap, 679.96%. Subpixel gaps use a one-pixel mark; exact values are shown beside the bars.
1-second budget
- pyvrp 1s 3.09% 100/100 feasible · 100 gap samples
- solverforge 1s 5.76% 100/100 feasible · 100 gap samples
- ortools 1s 10.48% 100/100 feasible · 100 gap samples
- pyhygese 1s 2.60% 98/100 feasible · 98 gap samples
- solverforge-py 1s 5.96% 59/100 feasible · 59 gap samples
- timefold 1s 80.52% 13/100 feasible · 13 gap samples
- highs 1s 226.22% 1/100 feasible · 1 gap samples
10-second budget
- pyvrp 10s 1.22% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100
- solverforge 10s 5.10% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100
- ortools 10s 7.00% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100
- timefold 10s 57.84% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 13/100
- pyhygese 10s 1.61% 99/100 feasible · 99 gap samples Earlier feasibility: 1s 98/100
- solverforge-py 10s 5.77% 57/100 feasible · 57 gap samples Earlier feasibility: 1s 59/100
- highs 10s 644.32% 10/100 feasible · 10 gap samples Earlier feasibility: 1s 1/100
60-second budget
- pyvrp 60s 0.59% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100 · 10s 100/100
- solverforge 60s 4.99% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100 · 10s 100/100
- ortools 60s 5.62% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 100/100 · 10s 100/100
- timefold 60s 19.02% 100/100 feasible · 100 gap samples Earlier feasibility: 1s 13/100 · 10s 100/100
- pyhygese 60s 1.07% 99/100 feasible · 99 gap samples Earlier feasibility: 1s 98/100 · 10s 99/100
- solverforge-py 60s 5.56% 76/100 feasible · 76 gap samples Earlier feasibility: 1s 59/100 · 10s 57/100
- highs 60s 679.96% 29/100 feasible · 29 gap samples Earlier feasibility: 1s 1/100 · 10s 10/100
1-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| pyvrp 0.14.1 | 100 / 100 | 3.09% (100 samples) | 65153.04 (100 samples) | 1.59 s | 1.59 s | 93 / 100 |
| solverforge 0.22.0 | 100 / 100 | 5.76% (100 samples) | 67114.99 (100 samples) | 1.05 s | 1.05 s | 5 / 100 |
| ortools 9.15.6755 | 100 / 100 | 10.48% (100 samples) | 69785.72 (100 samples) | 1.14 s | 1.14 s | 36 / 100 |
| pyhygese 0.1.0 | 98 / 100 | 2.60% (98 samples) | 63179.00 (98 samples) | 1.06 s | 1.06 s | 5 / 100 |
| solverforge-py 0.6.15 | 59 / 100 | 5.96% (59 samples) | 42047.97 (59 samples) | 2.76 s | 4.14 s | 100 / 100 |
| timefold 2.7.1 | 13 / 100 | 80.52% (13 samples) | 38852.23 (13 samples) | 2.57 s | 2.30 s | 100 / 100 |
| highs 1.15.1 | 1 / 100 | 226.22% (1 samples) | 90008.00 (1 samples) | 1.44 s | 2.48 s | 100 / 100 |
10-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| pyvrp 0.14.1 | 100 / 100 | 1.22% (100 samples) | 64042.87 (100 samples) | 10.58 s | 10.58 s | 21 / 100 |
| solverforge 0.22.0 | 100 / 100 | 5.10% (100 samples) | 65969.66 (100 samples) | 10.04 s | 10.04 s | 0 / 100 |
| ortools 9.15.6755 | 100 / 100 | 7.00% (100 samples) | 67011.47 (100 samples) | 10.11 s | 10.11 s | 0 / 100 |
| timefold 2.7.1 | 100 / 100 | 57.84% (100 samples) | 97639.29 (100 samples) | 11.25 s | 11.25 s | 100 / 100 |
| pyhygese 0.1.0 | 99 / 100 | 1.61% (99 samples) | 63600.33 (99 samples) | 10.08 s | 10.08 s | 0 / 100 |
| solverforge-py 0.6.15 | 57 / 100 | 5.77% (57 samples) | 42333.82 (57 samples) | 11.62 s | 13.16 s | 80 / 100 |
| highs 1.15.1 | 10 / 100 | 644.32% (10 samples) | 140288.40 (10 samples) | 10.22 s | 14.02 s | 89 / 100 |
60-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| pyvrp 0.14.1 | 100 / 100 | 0.59% (100 samples) | 63519.30 (100 samples) | 60.59 s | 60.59 s | 0 / 100 |
| solverforge 0.22.0 | 100 / 100 | 4.99% (100 samples) | 65892.81 (100 samples) | 60.04 s | 60.04 s | 0 / 100 |
| ortools 9.15.6755 | 100 / 100 | 5.62% (100 samples) | 66468.01 (100 samples) | 60.11 s | 60.11 s | 0 / 100 |
| timefold 2.7.1 | 100 / 100 | 19.02% (100 samples) | 75240.71 (100 samples) | 61.25 s | 61.25 s | 0 / 100 |
| pyhygese 0.1.0 | 99 / 100 | 1.07% (99 samples) | 64472.25 (99 samples) | 60.38 s | 60.53 s | 1 / 100 |
| solverforge-py 0.6.15 | 76 / 100 | 5.56% (76 samples) | 51908.25 (76 samples) | 63.39 s | 66.30 s | 38 / 100 |
| highs 1.15.1 | 29 / 100 | 679.96% (29 samples) | 185048.86 (29 samples) | 60.55 s | 70.27 s | 69 / 100 |
Run provenance
Full canonical nightly candidate · completed 2026-10-19T00:26:12.477365+00:00 (UTC). The warehouse's publication checks passed; the benchmark repository was clean.
Run 03ef371a-a0a2-42ff-af47-87d73d5963a7
Harness commit 2f71da41a823.
Employee scheduling
Assign nurses to shifts while respecting hard rules. Among feasible schedules, lower penalty cost is better.
INRC-II · 37 instances · 555 recorded results · Completed
Reference values for all 37 instances, from INRC-II official test dataset (mobiz.vives.be/inrc2): best known bounds, not proven optima.
Earliest feasible gate first; compare feasibility and quality within each gate. Runtime does not rank solutions.
Feasible results
Instances solved to a hard-feasible solution, out of 37.
1-second budget
- solverforge 1s 35/37
- solverforge-py 1s 30/37
- highs 1s 11/37
- timefold 1s 5/37
- ortools 1s 1/37
10-second budget
- solverforge 10s 37/37
- timefold 10s 36/37
- solverforge-py 10s 34/37
- highs 10s 30/37
- ortools 10s 3/37
60-second budget
- solverforge 60s 37/37
- solverforge-py 60s 37/37
- timefold 60s 37/37
- highs 60s 31/37
- ortools 60s 8/37
First feasible gate
Instances first observed feasible at this tested budget, out of 37. Each instance counts once: earlier is better. Independent budget runs, not first-solution timestamps; late returns are retained. Never feasible: solverforge 0; solverforge-py 0; highs 5; timefold 0; ortools 29.
1-second budget
- solverforge 1s 35/37
- solverforge-py 1s 30/37
- highs 1s 11/37
- timefold 1s 5/37
- ortools 1s 1/37
10-second budget
- solverforge 10s 2/37
- timefold 10s 31/37
- solverforge-py 10s 4/37
- highs 10s 19/37
- ortools 10s 2/37
60-second budget
- solverforge 60s 0/37
- solverforge-py 60s 3/37
- timefold 60s 1/37
- highs 60s 2/37
- ortools 60s 5/37
Mean gap to reference
Feasible results with a reference cost only. 0% matches the reference; lower is better. Rows follow feasibility-first ranking, not gap alone. Coverage and sample counts are shown on every row; means over different solved subsets are not like-for-like. Bars are scaled to this problem's largest mean gap, 3423.96%. Subpixel gaps use a one-pixel mark; exact values are shown beside the bars.
1-second budget
- solverforge 1s 2030.43% 35/37 feasible · 35 gap samples
- solverforge-py 1s 2042.26% 30/37 feasible · 30 gap samples
- highs 1s 2021.44% 11/37 feasible · 11 gap samples
- timefold 1s 129.44% 5/37 feasible · 5 gap samples
- ortools 1s 118.41% 1/37 feasible · 1 gap samples
10-second budget
- solverforge 10s 1532.04% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 35/37
- timefold 10s 543.08% 36/37 feasible · 36 gap samples Earlier feasibility: 1s 5/37
- solverforge-py 10s 1919.35% 34/37 feasible · 34 gap samples Earlier feasibility: 1s 30/37
- highs 10s 2979.25% 30/37 feasible · 30 gap samples Earlier feasibility: 1s 11/37
- ortools 10s 76.25% 3/37 feasible · 3 gap samples Earlier feasibility: 1s 1/37
60-second budget
- solverforge 60s 782.36% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 35/37 · 10s 37/37
- solverforge-py 60s 1324.56% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 30/37 · 10s 34/37
- timefold 60s 450.20% 37/37 feasible · 37 gap samples Earlier feasibility: 1s 5/37 · 10s 36/37
- highs 60s 3423.96% 31/37 feasible · 31 gap samples Earlier feasibility: 1s 11/37 · 10s 30/37
- ortools 60s 374.63% 8/37 feasible · 8 gap samples Earlier feasibility: 1s 1/37 · 10s 3/37
1-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| solverforge 0.22.0 | 35 / 37 | 2030.43% (35 samples) | 61375.86 (35 samples) | 1.12 s | 1.12 s | 8 / 37 |
| solverforge-py 0.6.15 | 30 / 37 | 2042.26% (30 samples) | 59948.50 (30 samples) | 2.25 s | 2.81 s | 35 / 37 |
| highs 1.15.1 | 11 / 37 | 2021.44% (11 samples) | 40833.64 (11 samples) | 1.82 s | 1.79 s | 34 / 37 |
| timefold 2.7.1 | 5 / 37 | 129.44% (5 samples) | 5052.00 (5 samples) | 2.53 s | 2.75 s | 37 / 37 |
| ortools 9.15.6755 | 1 / 37 | 118.41% (1 samples) | 4390.00 (1 samples) | 1.05 s | 1.41 s | 28 / 37 |
10-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| solverforge 0.22.0 | 37 / 37 | 1532.04% (37 samples) | 49759.19 (37 samples) | 10.10 s | 10.10 s | 0 / 37 |
| timefold 2.7.1 | 36 / 37 | 543.08% (36 samples) | 18773.06 (36 samples) | 11.71 s | 11.83 s | 37 / 37 |
| solverforge-py 0.6.15 | 34 / 37 | 1919.35% (34 samples) | 58656.32 (34 samples) | 11.32 s | 11.65 s | 20 / 37 |
| highs 1.15.1 | 30 / 37 | 2979.25% (30 samples) | 78293.83 (30 samples) | 10.38 s | 11.11 s | 6 / 37 |
| ortools 9.15.6755 | 3 / 37 | 76.25% (3 samples) | 3171.67 (3 samples) | 10.06 s | 10.17 s | 1 / 37 |
60-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| solverforge 0.22.0 | 37 / 37 | 782.36% (37 samples) | 27426.62 (37 samples) | 60.10 s | 60.10 s | 0 / 37 |
| solverforge-py 0.6.15 | 37 / 37 | 1324.56% (37 samples) | 43554.46 (37 samples) | 61.67 s | 61.67 s | 2 / 37 |
| timefold 2.7.1 | 37 / 37 | 450.20% (37 samples) | 16417.97 (37 samples) | 61.89 s | 61.89 s | 1 / 37 |
| highs 1.15.1 | 31 / 37 | 3423.96% (31 samples) | 95461.45 (31 samples) | 60.57 s | 61.04 s | 1 / 37 |
| ortools 9.15.6755 | 8 / 37 | 374.63% (8 samples) | 12726.88 (8 samples) | 60.10 s | 60.36 s | 0 / 37 |
Run provenance
Full canonical nightly candidate · completed 2026-10-18T13:21:26.683904+00:00 (UTC). The warehouse's publication checks passed; the benchmark repository was clean.
Run f3c914f9-9695-446c-aab3-1241b7249021
Harness commit 2f71da41a823.
Job-shop scheduling
Sequence jobs on machines while respecting operation order and machine capacity. Lower makespan is better.
JSPLIB · 162 instances · 2430 recorded results · Completed
Reference values for all 162 instances, from ScheduleOpt/benchmarks: proven optima where the source closed the instance, best known bounds otherwise.
Earliest feasible gate first; compare feasibility and quality within each gate. Runtime does not rank solutions.
Feasible results
Instances solved to a hard-feasible solution, out of 162.
1-second budget
- solverforge 1s 162/162
- solverforge-py 1s 162/162
- ortools 1s 161/162
- timefold 1s 100/162
- highs 1s 34/162
10-second budget
- solverforge 10s 162/162
- solverforge-py 10s 162/162
- ortools 10s 162/162
- timefold 10s 159/162
- highs 10s 90/162
60-second budget
- solverforge 60s 162/162
- solverforge-py 60s 162/162
- ortools 60s 162/162
- timefold 60s 162/162
- highs 60s 122/162
First feasible gate
Instances first observed feasible at this tested budget, out of 162. Each instance counts once: earlier is better. Independent budget runs, not first-solution timestamps; late returns are retained. Never feasible: solverforge 0; solverforge-py 0; ortools 0; timefold 0; highs 40.
1-second budget
- solverforge 1s 162/162
- solverforge-py 1s 162/162
- ortools 1s 161/162
- timefold 1s 100/162
- highs 1s 34/162
10-second budget
- solverforge 10s 0/162
- solverforge-py 10s 0/162
- ortools 10s 1/162
- timefold 10s 59/162
- highs 10s 56/162
60-second budget
- solverforge 60s 0/162
- solverforge-py 60s 0/162
- ortools 60s 0/162
- timefold 60s 3/162
- highs 60s 32/162
Mean gap to reference
Feasible results with a reference cost only. 0% matches the reference; lower is better. Rows follow feasibility-first ranking, not gap alone. Coverage and sample counts are shown on every row; means over different solved subsets are not like-for-like. Bars are scaled to this problem's largest mean gap, 35.55%. Subpixel gaps use a one-pixel mark; exact values are shown beside the bars.
1-second budget
- solverforge 1s 14.12% 162/162 feasible · 162 gap samples
- solverforge-py 1s 14.25% 162/162 feasible · 162 gap samples
- ortools 1s 12.16% 161/162 feasible · 161 gap samples
- timefold 1s 30.30% 100/162 feasible · 100 gap samples
- highs 1s 21.31% 34/162 feasible · 34 gap samples
10-second budget
- solverforge 10s 12.26% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162
- solverforge-py 10s 12.65% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162
- ortools 10s 8.17% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 161/162
- timefold 10s 26.96% 159/162 feasible · 159 gap samples Earlier feasibility: 1s 100/162
- highs 10s 32.76% 90/162 feasible · 90 gap samples Earlier feasibility: 1s 34/162
60-second budget
- solverforge 60s 10.75% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162 · 10s 162/162
- solverforge-py 60s 10.88% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 162/162 · 10s 162/162
- ortools 60s 5.83% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 161/162 · 10s 162/162
- timefold 60s 13.85% 162/162 feasible · 162 gap samples Earlier feasibility: 1s 100/162 · 10s 159/162
- highs 60s 35.55% 122/162 feasible · 122 gap samples Earlier feasibility: 1s 34/162 · 10s 90/162
1-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| solverforge 0.22.0 | 162 / 162 | 14.12% (162 samples) | 2108.77 (162 samples) | 1.02 s | 1.02 s | 2 / 162 |
| solverforge-py 0.6.15 | 162 / 162 | 14.25% (162 samples) | 2110.47 (162 samples) | 1.05 s | 1.05 s | 8 / 162 |
| ortools 9.15.6755 | 161 / 162 | 12.16% (161 samples) | 2054.78 (161 samples) | 0.93 s | 0.93 s | 1 / 162 |
| timefold 2.7.1 | 100 / 162 | 30.30% (100 samples) | 1592.74 (100 samples) | 1.94 s | 1.94 s | 162 / 162 |
| highs 1.15.1 | 34 / 162 | 21.31% (34 samples) | 1097.62 (34 samples) | 1.00 s | 1.22 s | 49 / 162 |
10-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| solverforge 0.22.0 | 162 / 162 | 12.26% (162 samples) | 2079.57 (162 samples) | 10.01 s | 10.01 s | 0 / 162 |
| solverforge-py 0.6.15 | 162 / 162 | 12.65% (162 samples) | 2085.12 (162 samples) | 10.05 s | 10.05 s | 0 / 162 |
| ortools 9.15.6755 | 162 / 162 | 8.17% (162 samples) | 2017.97 (162 samples) | 8.23 s | 8.23 s | 0 / 162 |
| timefold 2.7.1 | 159 / 162 | 26.96% (159 samples) | 2435.41 (159 samples) | 10.94 s | 10.94 s | 24 / 162 |
| highs 1.15.1 | 90 / 162 | 32.76% (90 samples) | 1582.02 (90 samples) | 9.95 s | 10.09 s | 8 / 162 |
60-second budget
| Solver / tested version | Feasible | Mean gap | Mean cost | Runtime (feasible) | Mean runtime | Over budget |
|---|---|---|---|---|---|---|
| solverforge 0.22.0 | 162 / 162 | 10.75% (162 samples) | 2055.91 (162 samples) | 60.02 s | 60.02 s | 0 / 162 |
| solverforge-py 0.6.15 | 162 / 162 | 10.88% (162 samples) | 2057.80 (162 samples) | 60.05 s | 60.05 s | 0 / 162 |
| ortools 9.15.6755 | 162 / 162 | 5.83% (162 samples) | 1976.46 (162 samples) | 44.48 s | 44.48 s | 0 / 162 |
| timefold 2.7.1 | 162 / 162 | 13.85% (162 samples) | 2260.70 (162 samples) | 60.94 s | 60.94 s | 0 / 162 |
| highs 1.15.1 | 122 / 162 | 35.55% (122 samples) | 1850.55 (122 samples) | 57.56 s | 58.22 s | 0 / 162 |
Run provenance
Full canonical nightly candidate · completed 2026-10-19T00:50:44.168746+00:00 (UTC). The warehouse's publication checks passed; the benchmark repository was clean.
Run e9e294de-0d5c-4db2-a1eb-c77f23171734
Harness commit 2f71da41a823.
Evidence and methodology
Gates are independent runs with requested budgets, not samples of one continuous search. First feasible gate means the earliest tested budget that returned a hard-feasible result for that instance; it is not the instant feasibility was discovered. No first-incumbent timestamps were recorded.
Runtime is measured wall-clock duration, shown only as a diagnostic. “Over budget” counts results flagged by the harness’s wall-time tolerance; late returned solutions are retained, so these gates are not strict deadline guarantees. The tables start at 1s; expand the later gates to inspect improvement.
Download the recorded results and run metadata (JSON) · How we benchmark SolverForge · Benchmark source