simtools-compare-productions#

Compare simulation productions or summarize their resource requirements.

Overview#

This application compares simulation products from two or more productions at the event or signal level, or summarizes CORSIKA and sim_telarray resource requirements for a production. For event and signal comparisons, each production is identified by a label and one or more comma-separated input file patterns. Multiple files belonging to one label are aggregated.

The first production is the baseline. Every following production is compared with that baseline, so at least two production descriptors are required and production order matters. Production labels must be unique.

The application supports event-level and signal-level comparisons. Trigger-histogram files should normally be produced with simtools-write-trigger-histograms.

Metadata-based event comparisons write pair coverage to comparison_inventory.json. Compact reports keep only aggregate event counts and labels in comparison_statistics.json; plot metrics are annotated in the PDF. Check pairing errors before interpreting the results. Use --report_mode diagnostic to retain histogram arrays and detailed figures. Compact reports omit energy distributions for fixed-energy pairs and angular-distance distributions for on-axis pairs when both productions’ metadata confirms those conditions. This applies to telescope-type plots too. Missing metadata keeps the distributions visible.

For computing summaries, use --comparison_level computing --baseline_path <production-root>. Add --candidate_path to compare a second production and repeated --select options to filter both. Outputs include resource_requirements.md, resource_requirements.json, and resource plots for CORSIKA and sim_telarray. Time and storage are normalized per event; peak memory is per process. Diagnostic mode also retains detailed ECSV records. Reports list the CPU model, CPU counts, container image, and host count behind each production’s records; compare timings only between matching environments. The comparison fails when the two productions do not cover the same configurations and process roles, and the error lists the uncovered configurations and skipped resource records.

simtools-compare-productions \
    --comparison_level computing \
    --baseline_path /data/production \
    --candidate_path /data/optimized \
    --baseline_label reference \
    --candidate_label optimized \
    --select configuration.primary=gamma \
    --output_path resource-requirements

For signal-level comparisons, use --comparison_level signal with sim_telarray files. By default, all telescopes shared by the input files are processed. Use --array_layout_name with one or more telescope names to restrict the comparison, for example:

simtools-compare-productions \
    --comparison_level signal \
    --array_layout_name LSTN-01 MSTN-01 \
    --production flat /data/flat/run.simtel.zst \
    --production curved /data/curved/run.simtel.zst \
    --output_path signal-comparisons

--telescope_name is accepted as an alias of --array_layout_name.

For event-level comparisons, use array_layout_name to restrict the comparison to selected layouts. Without this option, all array-layout references found in the input files are aggregated into one comparison. With multiple selected layouts, each layout is compared independently and written to its own directory below output_path/<layout-name>/.

For production metadata manifests, pass the baseline and candidate metadata directories instead of legacy file descriptors. Repeated select expressions choose configurations, while compare_by lists configuration fields that are allowed to differ. Each matched configuration is written below output_path/<input-stem>/. If the baseline and candidate stems differ, the directory name joins them with -vs-; if multiple pairs would use the same name, a configuration hash is appended.

simtools-compare-productions \
    --baseline_path /data/baseline/trigger_histograms \
    --candidate_path /data/candidate/trigger_histograms \
    --select configuration.primary=gamma \
    --compare_by configuration.atmosphere \
    --array_layout_name CTAO-North-Alpha \
    --output_path comparisons

Input and output#

Role	Argument or file	Format	Description
Input	`production`	HDF5	Repeated label and comma-separated file-pattern pairs.
Input	`array_layout_name`	Array-layout or telescope name(s)	Optional selection for event-level layouts or signal-level telescopes.
Output	`output_path`	Directory	Directory for figures and the statistics report.
Output	`comparison_statistics.json`	JSON	Machine-readable comparison statistics.

Event-level comparisons include trigger multiplicity, trigger combinations, single-telescope and mixed-type trigger distributions, telescope participation, and simulated/triggered distributions of primary energy, core distance, and angular distance. Cumulative distributions and per-telescope-type plots are also written when the corresponding input data are available. Individual figures may be skipped when their input data are absent.

Figures are written as PNG files by default. Use --figure_format pdf to write PDF files, or --figure_format png pdf to write both formats. Other formats supported by Matplotlib can also be selected.

Command line arguments#

options#

–help

show this help message and exit

–production

Production descriptor for event or signal comparison: –production <label> <comma-separated input file patterns>. Repeat for each production; the first production is the baseline.

–baseline_path

Production directory containing baseline metadata manifests.

–candidate_path

Optional candidate production directory containing metadata manifests.

–baseline_label

Display label for the baseline production in computing comparisons.

–candidate_label

Display label for the candidate production in computing comparisons.

–compare_by

Configuration field allowed to differ between baseline and candidate.

–comparison_level

Comparison level to execute.

Default: events. Choices: events, signal, computing.

–report_mode

Render compact PDF panels, full diagnostics, or no figures while retaining metrics.

Default: diagnostic. Choices: compact, diagnostic, none.

application#

–select: Selection expression as dotted.path=value. Can be repeated.
–show_options: Print available values for a supported option and exit.

simulation model#

–array_layout_name: Restrict event-level comparison to selected array layout name(s), or signal-level comparison to selected telescope name(s).

Integration example#

Compare event-level production products#

Compare trigger multiplicity and telescope participation across production histograms.

Application: simtools-compare-productions

Integration test: compare_productions_events.yml

Command

simtools-compare-productions --config tests/integration_tests/config/compare_productions_events.yml

Configuration

array_layout_name:
- CTAO-North-Alpha
production:
- baseline
- ${generated:gamma_diffuse_run000010_za20deg_azm000deg_North_CTAO-North-Alpha_7.0.0_test.trigger_histograms.hdf5}
- candidate
- ${generated:gamma_diffuse_run000010_za20deg_azm000deg_North_CTAO-North-Alpha_7.0.0_test.trigger_histograms.hdf5}
output_path: simtools-output