# Simulation Productions

The production of large sets of simulated events is the core functionality of the simtools framework.
Simtools is designed to use the CTAO workload management system (WMS) and (especially for testing) the HT Condor job submission system.

## Configure and Produce Simulations

The following steps are executed for each new simulation production in the order [Configure](#configure), [Produce](#produce), [Verify and Report](#verify-and-report).

### Configure

```{warning}
Incomplete documentation:

- description of job configuration.
- concept of Production Configuration repository.
- ...
```

New productions are configured using templates from the simulation production configuration repository.
The configuration defines primary particles, observation parameters (zenith angle ranges, night-sky background (NSB) levels, array layouts), and production parameters including:

- simtools and Simulation Models versions
- Simulation production version
- Observational parameter grid definition (e.g., zenith angle binning)
- Required statistics or metric values for production sizing

Defining a new configuration triggers the following steps:

1. Derive CORSIKA configuration limits (e.g., energy ranges or maximum core scatter radius) for the production grid using the application [simtools-production-generate-grid](production_generate_grid) to determine configuration parameters from [CORSIKA configuration lookup tables](#corsika-configuration-lookup-tables)
2. Calculate required event statistics for each grid point (if not user-specified) using the application [simtools-production-generate-grid](production_generate_grid) and the following inputs:
    1. metric parameters for statistical requirements (e.g., effective collection area uncertainty after gamma/hadron separation)
    2. DL2 event data (with or without gamma/hadron cuts applied) **todo clear definition needed**
3. Estimate required computing resources (optional) using:
    1. resource lookup tables
    2. (a missing simtools)
4. Choose or set the correct production model for the period of interest from the production tables defined in the [Simulation Models database](https://gitlab.cta-observatory.org/cta-science/simulations/simulation-model/simulation-models/-/tree/main/simulation-models/productions?ref_type=heads).
5. Write configuration files to SimPipe Production Configuration GitLab (energy ranges, versions, etc.) **todo: to be defined**
    1. includes configuration parameter versioning
    2. writing of CWL files here?

### Produce

```{warning}
Incomplete documentation:

- description of interface to WMS and how to submit jobs.
- description of interface to BDMS and how output files are stored.

```

Uses [simtools-simulate-prod](https://gammasim.github.io/simtools/user-guide/applications/simtools-simulate-prod.html)
to execute for each production run on the compute nodes:

- queries the Simulation Models database to generate CORSIKA and sim_telarray configuration and model files
- generates run scripts for CORSIKA and sim_telarray (includes "multi-pipe" setup, meaning several sim_telarray for a single CORSIKA run)
- executes the air shower, telescopes, and array simulations
- collects final results including reduced event data and log files

The *Produce* step generates the following output files:

- log files from CORSIKA, sim_telarray, and simtools
- sim_telarray output files in eventio format (includes event data and histogram files)
- reduced event data files in HDF5 format generated by simtools using [simtools-generate-simtel-event-data](https://gammasim.github.io/simtools/user-guide/applications/simtools-generate-event-data.html)

#### Local productions

Configure and submit a local production using the [simtools-simulate-prod](simulate_prod) command.
The submit engine is configured with the `--submit_engine local` command line option, see
[simulate_prod_gamma_20_deg_North.yml](https://github.com/gammasim/simtools/blob/main/tests/integration_tests/config/simulate_prod_gamma_20_deg_North.yml) for an example configuration.

Running a local production is a simple way to test the production system, but in general too slow to simulate a large number of events.

#### Running simtools on HTCondor using Apptainers

Simtools can be run using HTCondor and Apptainers. Jobs are configured with the [simtools-simulate-prod-htcondor-generator](simulate_prod_htcondor_generator) command.
Configuration is similar to the [simtools-simulate-prod](simulate_prod) command, with the following additions:

- `APPTAINER_IMAGE` is the path to the Apptainer image to be used for the simulation.
- `PRIORITY` is the priority of the job in HTCondor.
- `RUN_NUMBER` and `NUMBER_OF_RUNS` are the first run number and the number of runs to be simulated.

The [simtools-simulate-prod-htcondor-generator](simulate_prod_htcondor_generator) tool generates two files:

- a condor submission file to specify the apptainer, the number of jobs (equals the number of runs), the priority, and the output files.
- a condor submission script with the [simtools-simulate-prod](simulate_prod) command to be run in the apptainer.

An example for the configuration [simtools-simulate-prod-htcondor-generator](simulate_prod_htcondor_generator) can be found in [simulate_prod_htcondor_generator_gamma_20_deg_North.yml]([tests/integration_tests/config/simulate_prod_htcondor_generator_gamma_20_deg_North.yml](https://github.com/gammasim/simtools/blob/main/tests/integration_tests/config/simulate_prod_htcondor_generator_gamma_20_deg_North.yml)).

Container images are available from the GitHub and CTAO container registries and can be converted to Apptainer images using the `apptainer build` command.
Example:

```bash
apptainer build simtools.sif docker://ghcr.io/gammasim/simtools-prod-250304-corsika-78000-bernlohr-1.69-prod6-baseline-qgs3-avx2:20250507-154410
```

#### Running Grid Productions

```{warning}
Incomplete Documentation.
```

### Verify and Report

- Verification of sim_telarray meta-parameters
- Verification for each sim_telarray output file (includes log files)
- Comparison with reference productions

## Production Preparatory Steps

### CORSIKA configuration lookup tables

Generate lookup tables for CORSIKA configuration limits as a function of zenith angle, night-sky background (NSB) level, and array layout.
Limits on energy, core scatter radius, and viewcone radius are derived from triggered simulation events generated from
simulation runs with generous settings.
Executed for major releases or significant configuration changes (e.g., changes in telescope array):

- uses reduced event data files generated by [simtools-generate-simtel-event-data](generate_simtel_event_data) as input for each simulation run
- [simtools-merge-tables](merge_tables) merges sufficient reduced event data files into larger files (HDF5 format)
- [simtools-production-derive-corsika-limits](production_derive_corsika_limits) derives CORSIKA configuration limits for each zenith angle, NSB level, and array layout and generates lookup tables

Lookup tables are stored in the SimPipe Production Configuration gitlab repository.

```{warning}
Incomplete documentation and unclear:

- where are merged tables stored? Or temporarily generated and not stored?

```