How to Participate

QOBLIB thrives on community contributions. Whether you have a better bound, a new quantum run, or a useful negative result — every submission enriches the benchmark for the whole community.

Build a submission Full guide on GitHub ↗ CSV template ↗
Submission Process

To contribute a solution to any problem instance, submit your results via Pull Request to the QOBLIB repository. Every PR must be approved by at least two committee members before it can be merged.

  1. 1
    Build your submission file

    Use the interactive submission builder or fill in the CSV template directly. Every row covers one problem instance; shared fields (submitter, affiliation, hardware, etc.) can be filled once and applied to all rows.

  2. 2
    Validate locally

    Run the automated checker misc/ci/check_submission.py to verify your directory structure, CSV format, and solution feasibility before submitting. See misc/README.md for full usage instructions.

  3. 3
    Open a Pull Request

    Fork the repository, commit your _summary.csv and solution files, then open a PR against main. The CI checker runs automatically and posts its result as a comment; a failing run blocks the merge until fixed. Every PR requires approval from at least two committee members.

    If CI fails: read the checker comment on your PR — it lists each failing check with the exact row and column. Fix the issue locally, re-run misc/ci/check_submission.py to confirm it passes, then push a new commit to the same branch. The CI job re-runs automatically on every push.
  4. 4
    Best-known values update automatically

    You do not need to touch any solutions/ directory. When your PR is merged, a GitHub Action recomputes best-known values from all feasible submissions and credits the first source to reach each new best.

What is Required of a Solution
Validity — always required

Your solution file must parse and match the instance's dimensions and format. A malformed file always fails, regardless of other declarations.

Feasibility — always checked automatically

Solution feasibility is always verified by the automated checker on every PR. If a run did not produce a feasible solution, set # Feasible Runs to 0 — the checker will then accept the valid file even if constraints are violated, allowing honest reporting of exploratory or unsuccessful runs.

Optimality — never required

A valid, feasible heuristic solution that is not optimal is accepted. Optimality is only cross-checked when you assert a proven optimum by setting Optimality Bound equal to your Best Objective Value; otherwise leave it as N/A.

Required Information

Each benchmark submission should include the following fields (see the full CSV template for details).

Category Field Notes
Identification & Attribution Problem Identifier of the considered problem instance
Submitter Name(s) of the author(s); comma-separated list in double quotes for multiple authors
Affiliation Institution(s) in the same order as the authors; repeat a shared affiliation so the count matches
Date Date of submission
Reference Link to paper or repository with full details (hardware specs, software versions, hyperparameters)
Objective & Model Best Objective Value Best value found across all repetitions
Optimality Bound Lower/upper bound if proven optimal; otherwise N/A
Modeling Approach How the problem was formulated
# Decision Variables Total number of decision variables
# Binary Variables Number of binary decision variables
# Integer Variables Number of integer decision variables
# Continuous Variables Number of continuous decision variables
# Non-Zero Coefficients Number of non-zero coefficients in the objective and constraints
Coefficients Type Type of coefficients (integer, binary, continuous)
Coefficients Range Min/max values of non-zero coefficients
Algorithm Details Workflow Pre-processing, pre-solvers, main algorithm, post-processing
Algorithm Type Deterministic or stochastic
Paradigm Classical, Quantum Simulator, or Quantum Hardware
# Runs Number of times the experiment was repeated
# Feasible Runs Number of runs that found a feasible solution
# Successful Runs Runs with objective within ε of the best found value
Success Threshold The threshold ε defining a successful run
Hardware & Runtime Hardware Specifications Complete specifications of all hardware used
Total Runtime End-to-end execution time for the complete workflow
Time to Solution Time to find the best solution
CPU Runtime CPU runtime to run the workflow
GPU Runtime GPU runtime to run the workflow
QPU Runtime QPU runtime to run the workflow
Other HW Runtime Runtime on any other hardware used
Miscellaneous Remarks Additional notes or information
Objective Time Series (Optional)

To enable convergence analysis and Time-to-Solution (TTS) verification, you may include an objective time series file alongside your solution files. This is optional but strongly encouraged for stochastic runs.

File naming & placement

Place the file in each instance subdirectory, named <instance>_objective_time_series.json (plain JSON) or <instance>_objective_time_series.json.gz (gzip-compressed for large files).

Format

The file must be a JSON array of runs (one element per independent run). Each run is itself an array of incumbent update objects recorded whenever the best-found objective value improves: 

[
  [
    {"Time": 0.001, "Incumbent": 120.0},
    {"Time": 0.218, "Incumbent":   5.0},
    {"Time": 0.435, "Incumbent":   0.0}
  ],
  [
    {"Time": 0.002, "Incumbent":  80.0},
    {"Time": 0.651, "Incumbent":   0.0}
  ]
]
Key Type Description
Time number Wall-clock time in seconds from the start of the run at which this incumbent was first reached
Incumbent number or null The best objective value found up to this point in the run. Use null when no feasible solution has been found yet, so a solver log (e.g. Gurobi's root-relaxation rows) can be exported verbatim
Rules
  • The Incumbent sequence within each run must be non-increasing for minimization (non-decreasing for maximization).
  • A null Incumbent is allowed only before the run's first numeric incumbent; such entries are skipped by the monotonicity check.
  • The first entry should record the initial incumbent (the starting solution's objective value).
  • The last entry's Time should correspond to the end of the run so that TTS can be computed from it.
  • There is no required correspondence between the number of runs in the time series and # Runs in the CSV, but they should ideally match.
CI checker behavior

The automated checker validates JSON structure (correct types, required keys, non-empty runs) and enforces monotonicity: the Incumbent sequence within each run must be non-increasing for minimization problems and non-decreasing for maximization problems. A violation is reported as an error with the exact run and entry index. Entries with a null Incumbent (no feasible solution yet) are accepted and skipped, as long as they precede the run's first numeric incumbent. Providing a time series is entirely optional; its absence is reported as informational only.

Contributing a New Problem Instance

The instance sets for each problem class are curated by the committee, but well-motivated proposals to extend them are welcome — for example when you have identified a challenging real-world graph, a larger size tier, or a structurally distinct instance not represented in the existing set.

When to propose a new instance
Good candidates
  • Fills a gap in difficulty or size range (larger tier, denser family, distinct structure)
  • Originates from a real application and is verifiably hard for classical solvers
  • Discussed with a committee member in advance
Not suitable
  • Trivially solved by standard classical solvers
  • Duplicate of an existing instance
  • No clear source or generation procedure
What to include in the PR
  1. 1
    The instance file

    Place it in NN-problem/instances/ and name it according to the existing naming convention for that problem class (see the instances/README.md of the target problem).

  2. 2
    A reference solution or known bound

    Even a heuristic or partial solution placed in NN-problem/solutions/ helps the committee assess difficulty and is used to seed the best-known-value table.

  3. 3
    Generation provenance

    Include in the PR description a code snippet, random seed, or external citation sufficient to reproduce or credit the instance. Instances from external sources must have a verifiable URL or publication reference.

  4. 4
    Update the instances README

    Add an entry to instances/README.md (or the Instance Sources list) documenting the file format, size parameters, and origin of the new instance.

Review & acceptance

Instance PRs are reviewed by at least two committee members who assess difficulty, structural coverage, and provenance. Accepted instances are merged into main and automatically picked up by the site builder at the next deployment. Rejected proposals receive a written explanation with guidance on how to revise.

To propose an entirely new problem class, please open a GitHub Issue first to align on scope and checker requirements before investing in a full PR.

Best Practices
Solution Files
  • Follow the format specified in each problem class directory
  • Include validation information where applicable
  • Name files according to the instance naming convention
Documentation
  • Be as detailed as possible in your reference material
  • Include reproducible instructions
  • Document any deviations from standard approaches
  • Report negative results — they are valuable to the community
Stochastic Algorithms
  • Run at least 5 independent trials (10+ recommended)
  • Report statistical measures (mean, median, std dev) when possible
  • Document random seeds for reproducibility
Runtime Measurements
  • Measure wall-clock time for total runtime
  • Separate classical and quantum processing times
  • Exclude compilation and queue times
  • Report hardware specifications completely
Questions?

Get in touch

Open an issue in the repository or reach out to any committee member directly.

Open an issue ↗ Meet the committee