Testing#

The Testing module provides tools for testing the code. This might be part of the integration tests or unit tests.

assertions#

Functions asserting certain conditions are met (used e.g., in integration tests).

testing.assertions.assert_file_type(file_type, file_name)[source]#

Assert that the file is of the given type.

Parameters:

file_type: str: File type (json, yaml).
file_name: str: File name.

testing.assertions.assert_hdf5_dataset_min_rows(file_name, expected_min_rows)[source]#

Assert that named HDF5 datasets contain at least the requested number of rows.

Parameters:

file_namestr or pathlib.Path: HDF5 file to inspect.
expected_min_rowsmapping: Mapping from HDF5 dataset names to their minimum required row counts.

Returns:

bool: True when every requested dataset meets its minimum row count.

Raises:

AssertionError: If a requested item is missing, is not an HDF5 dataset, or contains too few rows.
OSError: If file_name is not a readable HDF5 file.

testing.assertions.assert_hdf5_datasets(file_name, expected_datasets)[source]#

Assert that an HDF5 file contains the requested root datasets.

Parameters:

file_namestr or pathlib.Path: HDF5 file to inspect.
expected_datasetssequence of str: Names of root-level HDF5 datasets that must be present.

Returns:

bool: True when all requested datasets are present.

Raises:

AssertionError: If a requested item is missing or is not an HDF5 dataset.
OSError: If file_name is not a readable HDF5 file.

testing.assertions.check_log_files(log_file, file_test)[source]#

Check log file (plain, tar) for wanted and forbidden patterns.

Parameters:

log_filestr: Path to the log file.
file_testdict: Dictionary with the test configuration.

Returns:

bool: True if the logs are correct.

testing.assertions.check_output_from_sim_telarray(file, file_test)[source]#

Check that the sim_telarray simulation result is reasonable and matches the expected output.

Parameters:

file: Path: Path to the sim_telarray file.
file_test: dict: File test description including expected output and metadata.

Raises:

ValueError: If the file is not a zstd compressed file.

configuration#

Integration test configuration.

exception testing.configuration.VersionError[source]#: Raise if model version requested is not supported.

testing.configuration.configure(config, tmp_test_directory, request)[source]#

Prepare configuration and command for integration tests.

Parameters:

config: dict: Configuration dictionary.
tmp_test_directory: str: Temporary test directory (from pytest fixture).
request: request: Request object.

Returns:

str: command to run the application test.
str: config file model version.

testing.configuration.create_tmp_output_path(tmp_test_directory, config)[source]#

Create temporary output path.

Parameters:

tmp_test_directory: str: Temporary directory.
config: dict: Configuration dictionary.

Returns:

str: path to the temporary output directory.

testing.configuration.get_application_command(app, config_file=None, config_string=None)[source]#

Return the command to run the application with the given config file.

Parameters:

app: str: Name of the application.
config_file: str: Configuration file.
config_string: str: Configuration string (e.g., ‘–version’)

Returns:

str: command to run the application test.

testing.configuration.get_list_of_test_configurations(config_files, test_resources_path=None)[source]#

Return list of test configuration dictionaries or test names.

Read all configuration files for testing.

Parameters:

config_files: list: List of integration test configuration files.
test_resources_path: str or pathlib.Path, optional: Base directory containing the static, generated, and downloaded resource directories.

Returns:

list:: list of test names or of configuration dictionaries.

testing.configuration.resolve_prepared_resource_paths(value, prepared_resources_path)[source]#

Resolve per-test prepared-resource references recursively.

Parameters:

valueobject: Configuration value, mapping, or sequence to resolve.
prepared_resources_pathstr or pathlib.Path: Directory containing files created by integration-test preparation steps.

Returns:

object: Configuration with ${prepared:...} references replaced by absolute paths.

Raises:

ValueError: If a prepared-resource reference escapes its preparation directory.

testing.configuration.resolve_test_resource_paths(value, test_resources_path=None)[source]#: Resolve test-resource references recursively.

resource_generation#

Generate and collect versioned resources used by simtools integration tests.

testing.resource_generation.download_files(config_file, target_dir)[source]#

Download external files listed in a resource-generation configuration.

Parameters:

config_filestr or pathlib.Path: YAML file containing base_urls and files entries.
target_dirstr or pathlib.Path: Base directory for each entry’s relative target_path.

Returns:

list[pathlib.Path]: Downloaded file paths.

Raises:

FileNotFoundError: If the configuration or a remote file does not exist.
ValueError: If the configuration structure is invalid or a version placeholder is unresolved.

testing.resource_generation.generate_test_resources(args_dict, run_time=None, context=None)[source]#

Download inputs and run resource-generation workflows for a release.

Parameters:

args_dictdict: Dictionary containing command line arguments. Optional key overwrite_collection_files (bool) allows collection output files to be overwritten when different source files have the same basename. Defaults to False.
run_timelist or None: Prepared runtime command. If provided, reuse it instead of preparing the runtime environment from the workflow configuration.

testing.resource_generation.get_integration_test_directory(test_directory, simtools_version)[source]#

Return the integration-test directory for a simtools release.

Parameters:

test_directorystr or pathlib.Path: Root directory of the simtools-tests repository (or a directory containing it).
simtools_versionstr: Version directory to use, for example v0.32.0.

Returns:

pathlib.Path: Path to the release-specific integration_tests directory.

testing.resource_generation.get_resource_generation_directory(test_directory, simtools_version)[source]#

Return the configuration directory for resource generation.

Parameters:

test_directorystr or pathlib.Path: Root directory of the simtools-tests repository.
simtools_versionstr: Version directory to use, for example v0.32.0.

Returns:

pathlib.Path: Existing release-specific config_files directory.

Raises:

FileNotFoundError: If the configuration directory does not exist.

testing.resource_generation.initialize_test_resources(test_directory, simtools_version, template_version)[source]#

Create a release-specific resource directory from an existing version.

Only workflow inputs are copied. Generated, downloaded, and log files are deliberately left out so the new release gets a clean resource set.

testing.resource_generation.run_configured_applications(args_dict, config_dir, log_dir, run_time, replacements, unique_tmp_directories=False, context=None)[source]#

Run all resource-generation workflows using one prepared runtime.

Parameters:

args_dictdict: Dictionary containing command line arguments.
config_dirstr or pathlib.Path: Directory containing the *.config.yml workflow files.
log_dirstr or pathlib.Path: Destination directory for one log file per workflow.
run_timelist or None: Prepared runtime command. If provided, reuse it instead of preparing the runtime environment from the workflow configuration.
replacementsdict[str, str] or None: Placeholders replaced recursively in each workflow configuration.
unique_tmp_directoriesbool: Use a fresh temporary subtree for each workflow invocation.

testing.resource_generation.validate_static_files(manifest_file)[source]#

Validate static integration-test files against their manifest.

Parameters:

manifest_filestr or pathlib.Path: YAML manifest containing relative file_name and sha256 values.

Raises:

FileNotFoundError: If the manifest does not exist.
ValueError: If entries are invalid, files are missing or unlisted, or checksums differ.

options#

Helpers for pytest options mirrored by environment variables.

testing.options.add_mirrored_options(parser)[source]#: Add pytest options whose values can also come from environment variables.

testing.options.get_mirrored_option(config, option_name)[source]#: Return a command-line option, falling back to its environment variable.

output_validation#

output_validation.artifacts#

Output-artifact location and existence checks.

class testing.output_validation.artifacts.OutputArtifact(path, descriptor)[source]#

A configured integration-test output and its resolved path.

assert_exists()[source]#: Raise an informative assertion error when the artifact is missing.

classmethod from_descriptor(configuration, descriptor)[source]#: Create an artifact from a workflow output descriptor.

output_validation.hdf5#

HDF5 integration-output validators.

testing.output_validation.hdf5.validate_datasets(path, required=None, minimum_rows=None)[source]#: Validate configured HDF5 datasets and their minimum row counts.

testing.output_validation.hdf5.validate_product(path, product)[source]#: Validate one structured HDF5 product using its registered validator.

output_validation.model_parameters#

Source-neutral model-parameter output validation.

testing.output_validation.model_parameters.validate(path, rule, configuration, model_reader=None)[source]#

Compare a generated parameter value with its model reference.

Parameters:

model_readerobject, optional: Reader selected for the current validation run. A reader is created from configuration only when this is omitted.

output_validation.reference#

Reference-file comparison validators.

testing.output_validation.reference.compare_ecsv_files(first_file, second_file, tolerance=1e-05, columns=None, metadata=False, filters=None, key_columns=None)[source]#: Compare ECSV rows, selected columns, units, and optional metadata.

testing.output_validation.reference.compare_files(first_file, second_file, tolerance=1e-05, columns=None, metadata=False, filters=None, key_columns=None)[source]#: Compare supported structured files.

testing.output_validation.reference.compare_json_or_yaml_files(first_file, second_file, tolerance=0.01)[source]#: Compare JSON or YAML documents, tolerating numerical model values.

testing.output_validation.reference.compare_values(first, second, tolerance, value_field=False)[source]#: Compare values recursively, including numerically tolerant model values.

testing.output_validation.reference.difference_report(reference_file, output_file, tolerance=1e-05, columns=None, metadata=False, filters=None, key_columns=None)[source]#: Return a readable difference report for a reference and generated file.

testing.output_validation.reference.resolve_path(path)[source]#: Resolve a repository-relative reference path.

output_validation.registry#

Validator registry and handlers for integration-test outputs.

testing.output_validation.registry.run_validator(artifact, rule, context)[source]#: Run one explicitly configured output validator.

testing.output_validation.registry.validate_data_schema(artifact, rule, _context)[source]#: Validate a tabular output against a simtools data schema.

testing.output_validation.registry.validate_format(artifact, rule, _context)[source]#: Validate that an output has the configured format.

testing.output_validation.registry.validate_hdf5_datasets(artifact, rule, _context)[source]#: Validate required HDF5 datasets and minimum row counts.

testing.output_validation.registry.validate_hdf5_product(artifact, rule, _context)[source]#: Validate a registered structured HDF5 product.

testing.output_validation.registry.validate_log(artifact, rule, _context)[source]#: Validate wanted and forbidden log patterns.

testing.output_validation.registry.validate_metadata(artifact, rule, _context)[source]#: Validate ECSV metadata keys and content relations.

testing.output_validation.registry.validate_model_parameter(artifact, rule, context)[source]#: Compare a generated model parameter with its selected model source.

testing.output_validation.registry.validate_reference(artifact, rule, _context)[source]#: Compare an artifact with a reference file.

testing.output_validation.registry.validate_simtel(artifact, rule, _context)[source]#: Validate sim_telarray event output and metadata.

testing.output_validation.registry.validate_simtel_config(artifact, rule, _context)[source]#: Compare a generated sim_telarray configuration with a reference.

testing.output_validation.registry.validate_table(artifact, rule, _context)[source]#: Validate table rows, uniqueness, allowed values, and ranges.

output_validation.runner#

Orchestration of integration-test output validation.

testing.output_validation.runner.validate_application_output(config, from_command_line=None, from_config_file=None, model_reader=None)[source]#

Validate all explicitly declared output artifacts.

Parameters:

model_readerobject, optional: Reader to reuse for model-parameter validations.

testing.output_validation.runner.versions_match(from_command_line, from_config_file)[source]#: Return whether configured output validations apply to model versions.

output_validation.simtel#

sim_telarray output and configuration validators.

testing.output_validation.simtel.compare_config_files(reference_file, output_file)[source]#: Compare sim_telarray parameters and control lines.

testing.output_validation.simtel.validate_output(path, rule)[source]#: Validate sim_telarray event data and metadata.

output_validation.table#

Generic table and metadata validators.

testing.output_validation.table.get_path(value, dotted_path)[source]#: Read a dotted path from nested mappings.

testing.output_validation.table.has_path(value, dotted_path)[source]#: Return whether a dotted path exists in nested mappings.

testing.output_validation.table.read_table(path)[source]#: Read an Astropy-supported output table using format auto-detection.

testing.output_validation.table.validate_data_schema(path, schema_file)[source]#: Validate a table against a simtools data-product schema.

testing.output_validation.table.validate_metadata(path, rule)[source]#: Validate table metadata keys and metadata-to-content relations.

testing.output_validation.table.validate_table(path, rule)[source]#: Validate table row and column expectations.

helpers#

Helper functions for integration testing.

testing.helpers.skip_multiple_version_test(config, model_version)[source]#: Skip a test which is not meant for multiple versions if multiple versions are given.