Test resource benchmarks

Simtools records the resource requirements of its test suites in a separate benchmark dashboard. The dashboard is updated by the scheduled and post-merge CI-test-benchmarks workflow. It contains the unit and integration suite totals and the slower integration-test nodes, separated by model version.

Each benchmark suite runs once and serially on ubuntu-24.04. A pytest plugin records wall time, CPU time, peak RSS of pytest and its application descendants, on hosted runners. These measurements are noisy; use repeated runs to identify trends.

For local resource measurements, install the test and benchmark extras with pip install -e '.[tests,benchmark]'.

Individual integration-test charts are published only when the test wall time meets the configurable threshold, which defaults to five seconds. Fast tests still run, contribute to the suite totals, and remain available in the raw workflow artifact. Change integration_min_wall_time when manually dispatching the workflow to investigate a different range.

To investigate a change manually, dispatch CI-test-benchmarks from the Actions page and provide the model-version list. Pull requests do not publish benchmark data. Existing correctness workflows remain the source of test status.

Excluding an integration test

An application workflow can exclude one test from resource benchmarking while leaving normal integration CI unchanged. Set a non-empty reason in the YAML configuration:

applications:
- application: simtools-example
  exclude_from_resource_benchmark: "Requires an unstable external service."
  configuration:
    output_path: simtools-output

The reason and excluded test ID are included in the raw benchmark artifact. Use this only for workloads that are not representative or cannot produce a stable measurement.