Science-test workflow modules#

Modules for loading release definitions, coordinating selected test runs, and publishing their reports.

campaign#

Set up and run the science tests of one release.

science_tests.campaign.run_release(release_dir, *, context_file=None, template_dir=None, sites=None, tests=None, dry_run=False, allow_production=False, preflight=False, overwrite=False, application_args=None)[source]#

Validate and run the selected site/test pairs of one release.

Parameters:

release_dirstr or pathlib.Path: Directory containing release.yml and optional site overrides.
context_filestr or pathlib.Path or None: Context with candidate, baseline, and production-configuration paths; <release_dir>/context.yml by default.
template_dirstr or pathlib.Path or None: Shared template; discovered above the release directory by default.
sites, testsiterable of str or None: Selected sites and tests; the required tests of each site by default. Dependencies selected together run first; otherwise their products must already exist.
dry_runbool: Validate the complete selection without writes or execution.
allow_productionbool: Permit explicitly selected production tests after grid review.
preflightbool: Check host paths and run one worker-side canary per site before production.
overwritebool: Rebuild derived products and archive failed, finished productions for a retry.
application_argsdict or None: Common runner arguments such as model-source and environment overrides.

Returns:

dict: Release summary with the result of every required site/test pair.

Raises:

ValueError: Invalid definitions, context, selection, or production permission.
RuntimeError: A selected test failed or is incomplete.

science_tests.campaign.setup_release(release_dir, template_dir=None, context_file=None, *, baseline_version=None, simulation_models_git_revision=None)[source]#

Create a release directory from the shared templates.

Copies release.yml, all site templates, and the context template, replacing the release-label placeholder by the name of the release directory’s parent.

Parameters:

release_dirstr or pathlib.Path: Destination directory.
template_dirstr or pathlib.Path or None: Shared template; discovered above the release directory by default.
context_filestr or pathlib.Path or None: Context template; context.example.yml of the template by default.
baseline_versionstr or None: Baseline release label; also the last component of the baseline context path.
simulation_models_git_revisionstr or None: Model Git revision recorded in the release definition.

Returns:

pathlib.Path: Created release directory.

acceptance#

Evaluate science-test products against their acceptance contracts.

Expected products are file patterns that must match non-empty files. Acceptance rules apply numerical limits to values read from JSON documents or from selected rows of CSV result tables. A valid value outside a limit is a warning; missing or invalid evidence is an error.

exception science_tests.acceptance.AcceptanceError[source]#: A valid measurement exceeds a configured acceptance limit.

science_tests.acceptance.evaluate_products(definition, replacements)[source]#

Check expected products, the result-status file, and numerical acceptance rules.

Parameters:

definitiondict: Test definition with products, rule and optional result_status_file.
replacementsdict: Placeholder values.

Returns:

dict: status (pass or warn) with reason and warnings for a warning.

science_tests.acceptance.matched_files(pattern)[source]#

Return files matching a glob pattern, requiring at least one and no empty file.

Parameters:

patternstr: Absolute or relative glob pattern.

Returns:

list of pathlib.Path: Matching files.

science_tests.acceptance.nonempty(path)[source]#

Return whether a path is a non-empty file or a directory containing one.

Parameters:

pathpathlib.Path: File or directory.

Returns:

bool: True for a non-empty file or a directory with a non-empty file.

definitions#

Load and validate science-test release definitions.

A release directory contains release.yml, context.yml and optional site overrides. Test definitions, workflows, site defaults, and acceptance contracts are read from the shared science-test template. Workflows reference release values through the placeholders defined in this module.

class science_tests.definitions.ReleaseDefinition(release_dir, template_dir, release, catalogue, context, products, rules, suites=<factory>)[source]#: Validated definitions of one release, as returned by load_release().

science_tests.definitions.campaign_signature(bundle, model_source)[source]#

Return a SHA-256 digest of all definitions and the external context of a campaign.

Parameters:

bundleReleaseDefinition: Release definitions.
model_sourcedict: Effective model-source settings.

Returns:

str: Hex digest.

science_tests.definitions.dependency_order(test_ids, catalogue)[source]#

Return the tests and all their dependencies once each, dependencies first.

Parameters:

test_idsiterable of str: Test names.
cataloguedict: Test definitions by name.

Returns:

list of str: Ordered test names.

science_tests.definitions.load_context(context_file)[source]#

Load context paths, resolving the shared storage root environment variable.

Parameters:

context_filestr or pathlib.Path: Context YAML file.

Returns:

dict: Context values with ${SIMTOOLS_SCIENCE_TESTS_ROOT} resolved.

science_tests.definitions.load_release(release_dir, template_dir=None, context_file=None)[source]#

Load and validate all definitions of one release.

Parameters:

release_dirstr or pathlib.Path: Release science-test directory.
template_dirstr or pathlib.Path or None: Shared template; discovered above the release directory by default.
context_filestr or pathlib.Path or None: Context file; <release_dir>/context.yml by default.

Returns:

ReleaseDefinition: Release, catalogue, context, site suites, and acceptance contracts.

science_tests.definitions.load_yaml(path)[source]#

Load a YAML mapping.

Parameters:

pathpathlib.Path: YAML file.

Returns:

dict: File contents.

science_tests.definitions.model_source_arguments(application_args, bundle)[source]#

Apply context and release model-source settings over runner arguments.

The first of release, context, and runner arguments defining a model path selects the source type; settings of the other source type are cleared.

Parameters:

application_argsdict or None: Runner arguments.
bundleReleaseDefinition: Release definitions.

Returns:

dict: Updated copy of the runner arguments.

science_tests.definitions.resolve_template_dir(release_dir, template_dir=None)[source]#

Return the given template directory or the nearest one above the release.

Parameters:

release_dirpathlib.Path: Release directory whose ancestors are searched.
template_dirstr or pathlib.Path or None: Explicit template directory.

Returns:

pathlib.Path: Shared science-test template directory.

science_tests.definitions.resolve_workflow(bundle, workflow)[source]#

Return a workflow file, preferring a release override over the template.

Parameters:

bundleReleaseDefinition: Release definitions.
workflowstr: Workflow path relative to the template directory.

Returns:

pathlib.Path: Workflow file.

science_tests.definitions.select(requested, available, kind)[source]#

Return the requested names (all available ones by default), matching case-insensitively.

Parameters:

requestediterable of str or None: Requested names.
availableiterable of str: Valid names.
kindstr: Name kind used in error messages.

Returns:

list of str: Unique selected names.

science_tests.definitions.shared_runtime_environment(template_dir, replacements, runtime_role=None)[source]#

Return the shared runtime environment, or the one selected by runtime_role.

Parameters:

template_dirpathlib.Path: Template directory containing the optional run_time.yml.
replacementsdict: Placeholder values.
runtime_rolestr or None: Key in runtime_environments; runtime_environment is used when omitted.

Returns:

dict or None: Runtime environment; None if no runtime file exists and no role is requested.

science_tests.definitions.site_replacements(bundle, site, report_path)[source]#

Return placeholder values for one site.

Parameters:

bundleReleaseDefinition: Release definitions.
sitestr: Site key.
report_pathpathlib.Path: Work directory of the test.

Returns:

dict: Placeholder strings mapped to their values.

science_tests.definitions.test_definition(bundle, test_id)[source]#

Return one catalogue entry with its acceptance contracts and resolved workflow path.

Parameters:

bundleReleaseDefinition: Release definitions.
test_idstr: Test name.

Returns:

dict: Catalogue entry extended by workflow_path, products, rule, uses_baseline, and release configuration_overrides.

science_tests.definitions.validate_config(configuration, schema_file)[source]#

Validate one science-test configuration against its packaged schema.

Parameters:

configurationdict: Configuration to validate.
schema_filestr: Schema file name in the science-test schema directory.

science_tests.definitions.validate_context(bundle, uses_baseline)[source]#

Require absolute context paths and separate candidate and baseline directories.

Parameters:

bundleReleaseDefinition: Release definitions.
uses_baselinebool: Whether any selected test reads baseline products.

execution#

Prepare and execute science-test workflows for one site.

A workflow is resolved from its template: applications referencing the telescope placeholder are repeated for each telescope of the site, placeholders are replaced, release configuration overrides are applied, and the shared runtime environment is added. The effective workflow is written to the test’s work directory and executed from there.

class science_tests.execution.PreparedTest(site, test_id, definition, replacements, args, workflow, work, report_root, result)[source]#: Resolved workflow and planned result of one site/test pair.

science_tests.execution.execute_tests(tests, catalogue, release_dir, run_id, overwrite=False)[source]#

Execute prepared tests in order, blocking tests whose selected dependencies failed.

Parameters:

testslist of PreparedTest: Prepared tests in dependency order.
cataloguedict: Test definitions by name.
release_dirpathlib.Path: Release directory for published reports.
run_idstr: Identifier used for archives.
overwritebool: Rebuild derived products and allow retrying failed productions.

Returns:

list of dict: Test results.

science_tests.execution.log_planned(test)[source]#: Log the runtime and output paths of a planned test.

science_tests.execution.prepare_test(bundle, site, test_id, application_args, run_id)[source]#

Resolve one site/test workflow without executing it.

Parameters:

bundleReleaseDefinition: Release definitions.
site, test_idstr: Site and test names.
application_argsdict: Runner arguments including the model source.
run_idstr: Identifier shared by all tests of one runner call.

Returns:

PreparedTest: Prepared test with its planned result.

science_tests.execution.run_test(test, release_dir, run_id, overwrite=False)[source]#

Execute one prepared test, evaluate its products, and publish its report.

Parameters:

testPreparedTest: Prepared test.
release_dirpathlib.Path: Release directory for published reports.
run_idstr: Identifier used for archives.
overwritebool: Rebuild derived products and allow retrying failed productions.

preflight#

Pre-submission path checks for science-test production runs.

Checks run on the submit host and in one high-priority worker job using the production backend configuration. Checked are all absolute context paths and the absolute *_PATH entries of the job environment file.

science_tests.preflight.context_paths(context)[source]#

Return the absolute paths configured in a science-test context.

Parameters:

contextdict: Science-test context.

Returns:

dict: Absolute paths by context key.

science_tests.preflight.run_worker_canary(test, context, work_dir)[source]#

Run one worker job checking the context and environment paths in the container.

Parameters:

testsimtools.science_tests.execution.PreparedTest: Prepared production test whose first application defines backend_config.
contextdict: Science-test context.
work_dirpathlib.Path: Work directory for the canary job.

science_tests.preflight.validate_context_paths(context)[source]#

Require every absolute context path to be a non-empty file or directory.

Parameters:

contextdict: Science-test context.

production#

Check, collect, and retry science-test productions.

Production tests submit jobs and record a submission manifest below the candidate site directory. Collection waits for those jobs and reconciles their outputs with the reviewed job grid before downstream tests use them.

science_tests.production.archive_failed_production(definition, replacements, run_id, paths)[source]#

Archive a failed, finished production and its reports before a retry.

Parameters:

definitiondict: Production test definition.
replacementsdict: Placeholder values.
run_idstr: Identifier used as archive directory name.
pathsiterable of pathlib.Path: Work and report directories archived together with the site production.

science_tests.production.check_inputs(definition, replacements)[source]#

Require declared inputs and, where configured, complete reviewed productions.

Parameters:

definitiondict: Test definition.
replacementsdict: Placeholder values.

science_tests.production.check_reviewed_production(definition, replacements)[source]#

Reconcile completed jobs and their outputs with the reviewed job grid.

Parameters:

definitiondict: Test definition with reviewed_grid.
replacementsdict: Placeholder values.

Returns:

dict: Expected and observed job counts.

science_tests.production.collect_production(definition, replacements)[source]#

Collect finished jobs of all submissions and validate completed productions.

Parameters:

definitiondict: Collection test definition.
replacementsdict: Placeholder values.

Returns:

dict: pending result while jobs are queued, otherwise the production coverage.

reports#

Publish per-test reports and the release summary of science tests.

science_tests.reports.release_summary(bundle, results, dry_run, signature)[source]#

Merge new results into the previous summary of the same campaign definition.

Parameters:

bundleReleaseDefinition: Release definitions.
resultslist of dict: Results of this runner call.
dry_runbool: Whether the results are planned only.
signaturestr: Campaign digest; previous results are kept only for the same digest.

Returns:

dict: Release summary with results for all required site/test pairs.

science_tests.reports.write_summary(report_dir, summary)[source]#

Write the release summary as JSON and Markdown.

Parameters:

report_dirpathlib.Path: Release report directory.
summarydict: Release summary returned by release_summary().

science_tests.reports.write_test_report(work, report_root, release_dir, result, publish=True)[source]#

Publish the products of a successful test to the release report directory.

Parameters:

workpathlib.Path: Work directory with the test products.
report_rootpathlib.Path: Report directory of this site/test.
release_dirpathlib.Path: Release directory; report paths are recorded relative to it.
resultdict: Test result, updated with report paths and publication size.
publishbool: Publish products; False removes any previous report.

telescope_throughput#

Read and compare validation results for overall telescope throughput, including optical components, camera response, and night-sky-background rates.

Read and compare telescope-throughput validation results.

A result is the directory written by simtools-validate-telescope-throughput for one telescope: a combined YAML summary with the case metadata (meta) and scalar metrics, plus one ECSV wavelength table per efficiency channel (for example shower, muon, nsb).

science_tests.telescope_throughput.compare_results(baseline, candidate, tolerances=None, compare_by=None)[source]#

Compare metrics of two results with identical case settings.

Parameters:

baseline, candidatedict: Results returned by read_result().
tolerancesdict or None: Advisory relative tolerances by metric name.
compare_byiterable of str or None: Case metadata keys allowed to differ.

Returns:

dict: Status, both cases, per-metric changes, and diagnostics.

science_tests.telescope_throughput.load_results(path)[source]#

Load telescope results from a summary file, a result directory, or its parent.

Parameters:

pathstr or pathlib.Path: Combined summary file, one telescope result directory, or a directory with one result subdirectory per telescope.

Returns:

dict: Results keyed by telescope name.

Raises:

ValueError: No or ambiguous summaries were found.

science_tests.telescope_throughput.parse_tolerances(values)[source]#

Parse METRIC=VALUE advisory relative tolerances.

Parameters:

valuesiterable of str or list of str: Tolerance strings, possibly nested by repeated command-line options.

Returns:

dict: Non-negative relative tolerances by metric name.

science_tests.telescope_throughput.read_result(path)[source]#

Read one combined summary and the wavelength tables written beside it.

Parameters:

pathstr or pathlib.Path: Combined summary file or the directory containing exactly one.

Returns:

dict: Telescope name, case metadata, metrics as quantities, and tables by channel.

science_tests.telescope_throughput.write_json(path, value)[source]#: Write a JSON document, creating its parent directory.