simtools-compare-science-results

Compare telescope-throughput results of two releases.

Reads the combined summaries and wavelength tables written by simtools-validate-telescope-throughput and compares them telescope by telescope without rerunning any calculation. Writes science-comparison.json and report.pdf to the output path.

Compare telescope-throughput results of two releases without running a calculation. Each input is a combined summary written by simtools-validate-telescope-throughput, the directory containing it, or a directory with one such result directory per telescope. Use --telescopes to restrict the comparison to selected telescopes.

science-comparison.json records, per telescope, the summary metadata of both results and, for every summary metric, the values in the baseline unit, the absolute and relative change, and whether a configured advisory tolerance (--tolerance METRIC=VALUE) is exceeded. A zero baseline has no relative change. Differing summary metadata, missing metrics, or tolerances for unknown metrics make the comparison incomplete; allow metadata keys to differ with repeated --compare_by options.

report.pdf contains, per telescope, the scalar changes, the detected spectra of all channels with wavelength tables in both results (for example shower, muon, and NSB), and the optical efficiency components of each channel (mirror reflectivity and shadowing, camera filter and pixel throughput, quantum efficiency, atmospheric transmission, and total efficiency) as absolute values and relative changes. Unequal wavelength grids are diagnosed without resampling. Table problems are reported as table_diagnostics; with --require_wavelength_tables they make the comparison incomplete.

simtools-compare-science-results \
    --baseline_path /data/baseline/telescope_throughput \
    --candidate_path /data/candidate/telescope_throughput \
    --telescopes LSTN-01 LSTN-02 \
    --compare_by model_version \
    --tolerance cam_eff=0.05 \
    --output_path comparison

Command line arguments

options

–help

show this help message and exit

–baseline_path

Combined summary, telescope result directory, or directory of telescope results.

Required.

–candidate_path

Combined summary, telescope result directory, or directory of telescope results.

Required.

–baseline_label

Default: baseline.

–candidate_label

Default: candidate.

–telescopes

Restrict the comparison to these telescopes.

–compare_by

Summary metadata key allowed to differ; repeat for each key.

–tolerance

Advisory relative tolerances as METRIC=VALUE.

–require_wavelength_tables

Mark the comparison incomplete if wavelength tables cannot be compared.

application

–show_options

Print available values for a supported option and exit.

Integration example

Compare telescope-throughput science results

Compare exported telescope-throughput evidence without rerunning calculations.

Application: simtools-compare-science-results

Integration test: compare_science_results_telescope_throughput.yml

Command

simtools-compare-science-results --config tests/integration_tests/config/compare_science_results_telescope_throughput.yml

Configuration

baseline_path: ${generated:telescope_throughput/telescope_throughput_summary_North_LSTN-02_za0.0deg_azm000deg.yml}
candidate_path: ${generated:telescope_throughput/telescope_throughput_summary_North_LSTN-02_za0.0deg_azm000deg.yml}
baseline_label: reference
candidate_label: candidate
output_path: science-comparison