Simulation Productions#

The production of large sets of simulated events is the core functionality of the simtools framework. Simtools is designed to use the CTAO workload management system (WMS) and (especially for testing) the HT Condor job submission system.

Configure and Produce Simulations#

The following steps are executed for each new simulation production in the order Configure, Produce, Verify and Report.

Configure#

Warning

Incomplete documentation:

  • description of job configuration.

  • concept of Production Configuration repository.

  • …

New productions are configured using templates from the simulation production configuration repository. The configuration defines primary particles, observation parameters (zenith angle ranges, night-sky background (NSB) levels, array layouts), and production parameters including:

  • simtools and Simulation Models versions

  • Simulation production version

  • Observational parameter grid definition (e.g., zenith angle binning)

  • Required statistics or metric values for production sizing

Defining a new configuration triggers the following steps:

  1. Derive CORSIKA configuration limits (e.g., energy ranges or maximum core scatter radius) for the production grid using the application simtools-production-generate-grid to determine configuration parameters from CORSIKA configuration lookup tables

  2. Calculate required event statistics for each grid point (if not user-specified) using the application simtools-production-generate-grid and the following inputs:

    1. metric parameters for statistical requirements (e.g., effective collection area uncertainty after gamma/hadron separation)

    2. DL2 event data (with or without gamma/hadron cuts applied) todo clear definition needed

  3. Estimate required computing resources (optional) using:

    1. resource lookup tables

    2. (a missing simtools)

  4. Choose or set the correct production model for the period of interest from the production tables defined in the Simulation Models database.

  5. Write configuration files to SimPipe Production Configuration GitLab (energy ranges, versions, etc.) todo: to be defined

    1. includes configuration parameter versioning

    2. writing of CWL files here?

Produce#

Warning

Incomplete documentation:

  • description of interface to WMS and how to submit jobs.

  • description of interface to BDMS and how output files are stored.

Uses simtools-simulate-prod to execute for each production run on the compute nodes:

  • queries the Simulation Models database to generate CORSIKA and sim_telarray configuration and model files

  • generates run scripts for CORSIKA and sim_telarray (includes “multi-pipe” setup, meaning several sim_telarray for a single CORSIKA run)

  • executes the air shower, telescopes, and array simulations

  • collects final results including reduced event data and log files

The Produce step generates the following output files:

  • log files from CORSIKA, sim_telarray, and simtools

  • sim_telarray output files in eventio format (includes event data and histogram files)

  • reduced event data files in HDF5 format generated by simtools using simtools-generate-simtel-event-data

Local productions#

Configure and submit a local production using the simtools-simulate-prod command. The submit engine is configured with the --submit_engine local command line option, see simulate_prod_gamma_20_deg_North.yml for an example configuration.

Running a local production is a simple way to test the production system, but in general too slow to simulate a large number of events.

Running simtools on HTCondor using Apptainers#

Simtools can be run using HTCondor and Apptainers. Jobs are configured with the simtools-simulate-prod-htcondor-generator command. Configuration is similar to the simtools-simulate-prod command, with the following additions:

  • APPTAINER_IMAGE is the path to the Apptainer image to be used for the simulation.

  • PRIORITY is the priority of the job in HTCondor.

  • RUN_NUMBER and NUMBER_OF_RUNS are the first run number and the number of runs to be simulated.

The simtools-simulate-prod-htcondor-generator tool generates two files:

  • a condor submission file to specify the apptainer, the number of jobs (equals the number of runs), the priority, and the output files.

  • a condor submission script with the simtools-simulate-prod command to be run in the apptainer.

An example for the configuration simtools-simulate-prod-htcondor-generator can be found in simulate_prod_htcondor_generator_gamma_20_deg_North.yml.

Container images are available from the GitHub and CTAO container registries and can be converted to Apptainer images using the apptainer build command. Example:

apptainer build simtools.sif docker://ghcr.io/gammasim/simtools-prod-250304-corsika-78000-bernlohr-1.69-prod6-baseline-qgs3-avx2:20250507-154410

Running Grid Productions#

Warning

Incomplete Documentation.

Verify and Report#

  • Verification of sim_telarray meta-parameters

  • Verification for each sim_telarray output file (includes log files)

  • Comparison with reference productions

Production Preparatory Steps#

CORSIKA configuration lookup tables#

Generate lookup tables for CORSIKA configuration limits as a function of zenith angle, night-sky background (NSB) level, and array layout. Limits on energy, core scatter radius, and viewcone radius are derived from triggered simulation events generated from simulation runs with generous settings. Executed for major releases or significant configuration changes (e.g., changes in telescope array):

Lookup tables are stored in the SimPipe Production Configuration gitlab repository.

Warning

Incomplete documentation and unclear:

  • where are merged tables stored? Or temporarily generated and not stored?