Integration Tests#

Integration tests run installed simtools-* applications from version-controlled workflow files in tests/integration_tests/config/ and validate their outputs.

Scope#

Integration tests should cover:

representative command-line use cases for each application
external interfaces such as the model repository and downloaded input files
internal interfaces where one simtools output becomes another input
selected compatibility checks for generated products

The tests follow the following levels of assertion, from weakest to strongest:

execution only;
file presence;
parse or schema validation;
semantic invariants;
deterministic reference comparison.

Workflow Files#

Each workflow file should define:

application
configuration
docs.title and docs.summary
test_name
integration_tests
optional test_use_case and test_requirement

One focused workflow per file is preferred unless an existing application pattern already groups related cases.

Commands#

Run all integration tests:

pytest --no-cov tests/integration_tests/test_applications_from_config.py

Run one application:

pytest -v -k "simtools-<app-name>" \
  tests/integration_tests/test_applications_from_config.py

Run one workflow:

pytest -v -k "simtools-<app-name>_<test_name>" \
  tests/integration_tests/test_applications_from_config.py

Run with a selected model version:

pytest -v --model_version 6.0.2 -k "<test_name>" \
  tests/integration_tests/test_applications_from_config.py

Run with simulation models from Git#

To run the integration tests with a local Git repository and revision:

pytest --no-cov -v \
  --model_version 7.0.0 \
  --simulation_models_git_path ../simulation-models.git \
  --simulation_models_git_revision v0.17.2 \
  tests/integration_tests/test_applications_from_config.py

The integration harness passes the path to each application subprocess. To run one workflow:

pytest --no-cov -vv \
  --model_version 7.0.0 \
  --simulation_models_git_path ../simulation-models.git \
  --simulation_models_git_revision v0.17.2 \
  'tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-docs-produce-reports_run0]'

Filesystem and Git sources cannot be configured together. All model-dependent workflows use one of these repository sources.

Resources#

Tests resolve resources from SIMTOOLS_TESTS_PATH and the resource-version in dependency_versions.yml. The environment variable identifies the local simtools-tests checkout; the catalog selects the versioned resource directory below it and records the Git ref used to obtain the repository. The corresponding command-line option is --simtools_tests_path.

CI checks out the configurable simtools_tests_branch and simulation_model_branch, both defaulting to main. These branches are independent of the release refs recorded in the dependency catalog; see CI repository branches.

<simtools-tests>/simtools-tests/<resource-version>/integration_tests/
  static/
  generated/
  downloaded/

Use ${static:path/to/file} for maintained inputs and ${generated:path/to/file} for generated reference products. Use ${downloaded:path/to/file} for externally downloaded resources. To select a resource set:

pytest --simtools_tests_path /full/path/to/simtools-tests/simtools-tests \
  tests/integration_tests/test_applications_from_config.py

To select another resource directory, update resource-version in dependency_versions.yml.

Versioned resource bundles are archived in gammasim/simtools-tests. Use the resource applications to create and synchronize these bundles:

simtools-resources-test-generate generates versioned test resources and validates configured static files. The resource generation and release workflow is documented in Test resources.

Use the current versioned resource set from simtools-tests for development, PR CI, and compatibility checks.

Prepared model parameters#

An integration test can prepare model-parameter inputs in its temporary directory instead of storing them in a versioned resource bundle. Use workflow schema version 0.6.0 for preparation steps, which run first and may only use simtools-get-model-parameter. Refer to their outputs with ${prepared:path/to/file} in the tested application’s configuration or a validation reference.

preparation:
- application: simtools-get-model-parameter
  configuration:
    parameter: array_layouts
    parameter_version: 2.0.2
    site: North
    output_file: array_layouts-2.0.2.json
configuration:
  array_layout_parameter_file: ${prepared:array_layouts-2.0.2.json}

Preparation outputs are isolated per test, are not added to simtools-tests, and cannot use absolute paths or .. path segments.

Validation#

Declare generated artifacts in test_outputs. Every declared artifact must exist. Add an ordered validations list when content also needs validation. Validator selection is explicit and does not depend on the filename suffix.

Available validator types are format, reference, data_schema, table, metadata, hdf5_datasets, hdf5_product, log, simtel, simtel_config, and model_parameter.

The validation interface has no legacy aliases: output checks must use test_outputs and explicit validator types.

Keep generated outputs deterministic by fixing seeds, labels, event counts, worker counts, and version-specific expectations.

Composable output validation#

Each output owns its location and validation rules. Product schemas validate stable structure such as columns, types, and units. Table and metadata rules describe expectations specific to the tested workflow. For data_schema, use a local schema path, URL, or a name relative to the schemas bundled with simtools. Schema resolution uses the same loader as applications and does not require a source checkout.

test_outputs:
- file: job_grid.ecsv
  path_descriptor: output_path
  validations:
  - type: data_schema
    schema: job_grid_density.schema.yml
  - type: table
    minimum_rows: 1
    unique_columns: [run_number]
    columns:
      primary:
        allowed_values: [gamma, proton]
      energy_min:
        range:
          minimum: 30.0
          maximum: 300.0
          unit: GeV
  - type: metadata
    required_keys: [job_grid_summary]
    relations:
    - left: job_grid_summary.simulation_rows
      equals: table.row_count
    - left: job_grid_summary.total_showers
      equals: table.column_sum
      column: showers_per_run

minimum_rows rejects empty or unexpectedly short tables. unique_columns checks complete columns for duplicate values. Column rules support allowed_values and inclusive or exclusive numerical range bounds, with an optional unit. Metadata paths use dotted mapping notation.

Metadata relations compare a dotted metadata path with either the table row count or a named column sum.

Table and metadata validators use Astropy table format auto-detection, so they can validate any table format supported by the installed Astropy I/O registry.

The simtel validator defaults to shower events and accepts explicit event ranges without exposing the underlying reader API:

- type: simtel
  event_type: shower
  event:
    pe_sum: {range: [20, 1000]}
    trigger_time: {range: [0, 50]}

The reference validator compares JSON, YAML, or ECSV files. ECSV comparisons check row order, column values and units by default. Use columns to select a subset, key_columns to compare rows in deterministic key order, metadata to include table metadata, and typed filters with operators such as equal, less, greater_equal, in, or not_in. Reference filtering never executes configuration text as code.

Differential output checks#

The pull-request differential job runs configurations marked with differential_test: true in their application entry against the merge base and the pull request, using the same model and test-resource revisions. The pull request’s selection is used for both checkouts; a selected case missing from the merge base is an error. Its diff_report.md lists changed comparable files and up to 20 differences per file. ECSV, FITS, HDF5, JSON, and YAML outputs are compared by their contents. Generated activity and product identifiers, timestamps, temporary test paths in provenance, and the simtools software version are normalized. Other metadata and scientific values remain significant. Temporary test configuration files and logs are ignored. Binary outputs are reported by presence and size for context and do not fail the job. Comparison errors cannot be approved with the expected-output-change label.

Review every reported difference to determine whether it changes an expected scientific product. Add the expected-output-change pull-request label when the output change is intentional; the report remains available in the job summary and artifact. The job is informational and is not a required check.