# Integration Tests

Integration tests run installed `simtools-*` applications from version-controlled
workflow files in `tests/integration_tests/config/` and validate their outputs.

## Scope

Integration tests should cover:

- representative command-line use cases for each application
- external interfaces such as the model repository and downloaded input files
- internal interfaces where one `simtools` output becomes another input
- selected compatibility checks for generated products

The tests follow the following levels of assertion, from weakest to strongest:

- execution only;
- file presence;
- parse or schema validation;
- semantic invariants;
- deterministic reference comparison.

## Workflow Files

Each workflow file should define:

- `application`
- `configuration`
- `docs.title` and `docs.summary`
- `test_name`
- `integration_tests`
- optional `test_use_case` and `test_requirement`

One focused workflow per file is preferred unless an existing application
pattern already groups related cases.

## Commands

Run all integration tests:

```bash
pytest --no-cov tests/integration_tests/test_applications_from_config.py
```

Run one application:

```bash
pytest -v -k "simtools-<app-name>" \
  tests/integration_tests/test_applications_from_config.py
```

Run one workflow:

```bash
pytest -v -k "simtools-<app-name>_<test_name>" \
  tests/integration_tests/test_applications_from_config.py
```

Run with a selected model version:

```bash
pytest -v --model_version 6.0.2 -k "<test_name>" \
  tests/integration_tests/test_applications_from_config.py
```

### Run with simulation models from Git

To run the integration tests with a local Git repository and revision:

```bash
pytest --no-cov -v \
  --model_version 7.0.0 \
  --simulation_models_git_path ../simulation-models.git \
  --simulation_models_git_revision v0.17.2 \
  tests/integration_tests/test_applications_from_config.py
```

The integration harness passes the path to each application subprocess. To run one workflow:

```bash
pytest --no-cov -vv \
  --model_version 7.0.0 \
  --simulation_models_git_path ../simulation-models.git \
  --simulation_models_git_revision v0.17.2 \
  'tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-docs-produce-reports_run0]'
```

Filesystem and Git sources cannot be configured together. All model-dependent workflows use
one of these repository sources.

## Resources

Tests resolve resources from `SIMTOOLS_TESTS_PATH` and the `resource-version`
in `dependency_versions.yml`. The environment variable identifies the local
`simtools-tests` checkout; the catalog selects the versioned resource directory
below it and records the Git ref used to obtain the repository. The corresponding
command-line option is `--simtools_tests_path`.

CI checks out the configurable `simtools_tests_branch` and `simulation_model_branch`, both
defaulting to `main`. These branches are independent of the release refs recorded in the
dependency catalog; see [CI repository branches](dependency_versions.md#ci-repository-branches).

```text
<simtools-tests>/simtools-tests/<resource-version>/integration_tests/
  static/
  generated/
  downloaded/
```

Use `${static:path/to/file}` for maintained inputs and
`${generated:path/to/file}` for generated reference products.
Use `${downloaded:path/to/file}` for externally downloaded resources.
To select a resource set:

```bash
pytest --simtools_tests_path /full/path/to/simtools-tests/simtools-tests \
  tests/integration_tests/test_applications_from_config.py
```

To select another resource directory, update `resource-version` in
`dependency_versions.yml`.

Versioned resource bundles are archived in
[`gammasim/simtools-tests`](https://github.com/gammasim/simtools-tests). Use
the resource applications to create and synchronize these bundles:

- [`simtools-resources-test-generate`](../user-guide/applications/simtools-resources-test-generate.md)
  generates versioned test resources and validates configured static files.
The resource generation and release workflow is documented in
[Test resources](testing_resources.md).

Use the current versioned resource set from `simtools-tests` for development,
PR CI, and compatibility checks.

### Prepared model parameters

An integration test can prepare model-parameter inputs in its temporary directory instead of
storing them in a versioned resource bundle. Use workflow schema version `0.6.0` for
`preparation` steps, which run first and may only use
`simtools-get-model-parameter`. Refer to their outputs with `${prepared:path/to/file}` in the
tested application's `configuration` or a validation reference.

```yaml
preparation:
- application: simtools-get-model-parameter
  configuration:
    parameter: array_layouts
    parameter_version: 2.0.2
    site: North
    output_file: array_layouts-2.0.2.json
configuration:
  array_layout_parameter_file: ${prepared:array_layouts-2.0.2.json}
```

Preparation outputs are isolated per test, are not added to `simtools-tests`, and cannot use
absolute paths or `..` path segments.

## Validation

Declare generated artifacts in `test_outputs`. Every declared artifact must
exist. Add an ordered `validations` list when content also needs validation.
Validator selection is explicit and does not depend on the filename suffix.

Available validator types are `format`, `reference`, `data_schema`, `table`,
`metadata`, `hdf5_datasets`, `hdf5_product`, `log`, `simtel`,
`simtel_config`, and `model_parameter`.

The validation interface has no legacy aliases: output checks must use
`test_outputs` and explicit validator types.

Keep generated outputs deterministic by fixing seeds, labels, event counts,
worker counts, and version-specific expectations.

## Composable output validation

Each output owns its location and validation rules. Product schemas validate
stable structure such as columns, types, and units. Table and metadata rules
describe expectations specific to the tested workflow. For `data_schema`, use a
local schema path, URL, or a name relative to the schemas bundled with simtools.
Schema resolution uses the same loader as applications and does not require a
source checkout.

```yaml
test_outputs:
- file: job_grid.ecsv
  path_descriptor: output_path
  validations:
  - type: data_schema
    schema: job_grid_density.schema.yml
  - type: table
    minimum_rows: 1
    unique_columns: [run_number]
    columns:
      primary:
        allowed_values: [gamma, proton]
      energy_min:
        range:
          minimum: 30.0
          maximum: 300.0
          unit: GeV
  - type: metadata
    required_keys: [job_grid_summary]
    relations:
    - left: job_grid_summary.simulation_rows
      equals: table.row_count
    - left: job_grid_summary.total_showers
      equals: table.column_sum
      column: showers_per_run
```

`minimum_rows` rejects empty or unexpectedly short tables. `unique_columns`
checks complete columns for duplicate values. Column rules support
`allowed_values` and inclusive or exclusive numerical `range` bounds, with an
optional unit. Metadata paths use dotted mapping notation.

Metadata relations compare a dotted metadata path with either the table row
count or a named column sum.

Table and metadata validators use Astropy table format auto-detection, so they
can validate any table format supported by the installed Astropy I/O registry.

The `simtel` validator defaults to shower events and accepts explicit event
ranges without exposing the underlying reader API:

```yaml
- type: simtel
  event_type: shower
  event:
    pe_sum: {range: [20, 1000]}
    trigger_time: {range: [0, 50]}
```

The `reference` validator compares JSON, YAML, or ECSV files. ECSV comparisons
check row order, column values and units by default. Use `columns` to select a
subset, `key_columns` to compare rows in deterministic key order, `metadata`
to include table metadata, and typed `filters` with operators such as `equal`,
`less`, `greater_equal`, `in`, or `not_in`. Reference filtering never executes
configuration text as code.

## Differential output checks

The pull-request differential job runs configurations marked with
`differential_test: true` in their application entry against the merge base
and the pull request, using the same model and test-resource revisions. The
pull request's selection is used for both checkouts; a selected case missing
from the merge base is an error. Its `diff_report.md` lists changed comparable
files and up to 20 differences per file. ECSV, FITS, HDF5, JSON, and YAML
outputs are compared by their contents. Generated activity and product identifiers,
timestamps, temporary test paths in provenance, and the simtools software version
are normalized. Other metadata and scientific values remain significant. Temporary
test configuration files and logs are ignored. Binary outputs are reported by presence and size
for context and do not fail the job. Comparison errors cannot be approved with
the `expected-output-change` label.

Review every reported difference to determine whether it changes an expected
scientific product. Add the `expected-output-change` pull-request label when
the output change is intentional; the report remains available in the job
summary and artifact. The job is informational and is not a required check.