Test Resources#

Test resources are maintained input and reference files for integration and resource-management tests. Integration-test resources are maintained in the versioned simtools-tests repository. Ordinary unit tests must not depend on repository-local, downloaded, or external resource files; generate minimal inputs in tmp_test_directory instead.

Resource files require updates, e.g., when file formats change or new features are added to the software.

Types of resource files#

The examples in the description of the simtools-resources-test-generate application provides guidance on how to test and generate the different types of resource files.

Static test files#

Static test files are manually maintained by the developers. The number of static files should be kept to a minimum.

Static test files are located in the static directory of the versioned bundle. All static files are listed in static/static_manifest.yml in the simtools-tests repository.

Downloaded test files#

Downloaded test files are downloaded from external sources (e.g., from the simulation models repository) into the downloaded directory of the versioned bundle. The list of files to download is defined in the file config_files/download_files.yml in the simtools-tests repository.

Generated test files#

Most test files are generated by simtools applications. Config files for generating test files are located in config_files/*yml and can be generated using the simtools-resources-test-generate application.

Resource files and simtools releases#

For a new simtools release, the workflow is:

  1. Define a release candidate in simtools (e.g., v0.34.0-rc). The simtools CI workflow builds and publishes release-candidate containers for this release candidate.

  2. Define a new test resource version in simtools-tests (e.g., v0.34.0), update the workflow definitions with the above container version, and generate the new resource set with the release-candidate container.

  3. Run the relevant integration and unit tests locally against the new resources.

  4. On failure, fix the resource generation workflow or the simtools code and repeat steps 1-3.

  5. Release both simtools and simtools-tests.

This keeps the archived resource set aligned with the released software and avoids generating release resources from an unpinned development environment.

Applications#