Simulation Event I/O and Processing Utilities#

Modules for handling event-level data generated by the CORSIKA and sim_telarray simulation packages. Provide tools for reading, writing, and processing simulated shower and telescope events.

Reader#

Read reduced datasets in form of astropy tables from file.

class simtools.sim_events.reader.EventDataReader(event_data_file, telescope_list=None)[source]#

Read reduced MC data set stored in astropy tables.

filter_by_telescopes(triggered_data, triggered_shower, telescope_list=None, minimum_triggered_telescopes=1)[source]#

Filter trigger and shower data by an explicit or reader telescope list.

Parameters:

triggered_dataTriggeredEventData: Triggered event data to filter.
triggered_showerShowerEventData: Shower data corresponding to the triggered events.
telescope_listlist, optional: List of telescopes to filter by. If None, uses the reader’s telescope list.
minimum_triggered_telescopesint, optional: Minimum number of telescopes from the selected list required in an event.

Returns:

tuple: A tuple containing the filtered triggered event data and the corresponding triggered shower data.

get_reduced_simulation_file_info(simulation_file_info)[source]#

Return reduced simulation file info assuming single-valued parameters.

Applies rounding and uniqueness functions extract representative values for zenith, azimuth, and NSB level. Assumes all files share identical simulation parameters except for file names. Returns particle name instead of ID.

Logs a warning if multiple unique values are found.

Parameters:

simulation_file_infoastropy.table.Table: Dictionary containing simulation file info.

Returns:

dict: Dictionary containing the reduced simulation file info.

read_event_data(event_data_file, table_name_map=None)[source]#

Read event data and file info tables from file and apply transformations.

Allows to map tables names to their actual names in the file (e.g., “SHOWER” to “SHOWER_1”).

Parameters:

event_data_filestr: Path to the event data file.
table_name_mapdict, optional: Mapping of table names to their actual names in the file. Defaults to using the standard names “SHOWERS”, “TRIGGERS”, and “FILE_INFO”.

Returns:

tuple: A tuple with file info table, shower, triggered shower, and triggered event data.

read_table_list(event_data_file)[source]#

Read available tables from the event data file.

Rearrange dictionary with tables names into a list of dictionaries under the assumption that the file contains the tables “SHOWERS”, “TRIGGERS”, and “FILE_INFO”. Note that not all tables need to be present.

Parameters:

event_data_filestr: Path to the event data file.

Returns:

list: List of dictionaries containing the data from the tables.

scatter_area(core_scatter_min, core_scatter_max)[source]#

Calculate the scatter area of the core.

Parameters:

core_scatter_minastropy.units.Quantity: Minimum core scatter radius.
core_scatter_maxastropy.units.Quantity: Maximum core scatter radius.

Returns:

astropy.units.Quantity: Scatter area.

class simtools.sim_events.reader.ShowerEventData(shower_id=<factory>, event_id=<factory>, file_id=<factory>, simulated_energy=<factory>, x_core=<factory>, y_core=<factory>, shower_azimuth=<factory>, shower_altitude=<factory>, area_weight=<factory>, x_core_shower=<factory>, y_core_shower=<factory>, core_distance_shower=<factory>, angular_distance=<factory>)[source]#: Container for shower event data.

class simtools.sim_events.reader.TriggeredEventData(shower_id=<factory>, event_id=<factory>, file_id=<factory>, array_altitude=<factory>, array_azimuth=<factory>, telescope_list=<factory>, angular_distance=<factory>)[source]#: Container for triggered event data.

File info and headers#

Read run information and event counts through simulation-file readers.

simtools.sim_events.file_info.get_combined_eventio_run_header(sim_telarray_file, file_format='eventio')[source]#

Return the CORSIKA run header information from an eventio (sim_telarray) file.

Reads both RunHeader and MCRunHeader object from file and returns a merged dictionary. Adds primary id from the first event.

Parameters:

sim_telarray_file: str: Path to the sim_telarray file.
file_formatstr: Registered simulation-file reader, by default “eventio”.

Returns:

dict, None: CORSIKA run header. Returns None if not found.

simtools.sim_events.file_info.get_corsika_run_and_event_headers(corsika_iact_file, file_format='eventio')[source]#

Return the CORSIKA run and event headers from a CORSIKA IACT eventio file.

Parameters:

corsika_iact_file: str, Path: Path to the CORSIKA IACT eventio file.
file_formatstr: Registered simulation-file reader, by default “eventio”.

Returns:

tuple: CORSIKA run header and event header as dictionaries.

simtools.sim_events.file_info.get_corsika_run_number(file, file_format='eventio')[source]#

Return the CORSIKA run number from an eventio (CORSIKA IACT or sim_telarray) file.

Parameters:

file: str: Path to the eventio file.
file_formatstr: Registered simulation-file reader, by default “eventio”.

Returns:

int, None: CORSIKA run number. Returns None if not found.

simtools.sim_events.file_info.get_run_number(file, file_format='eventio')[source]#

Read the run number using the selected simulation-file format.

Parameters:

filestr or pathlib.Path: Simulation file.
file_formatstr: Registered reader name.

Returns:

int or None: Run number, or None when absent.

simtools.sim_events.file_info.get_simulated_events(event_io_file, file_format='eventio')[source]#

Return the number of shower and MC events from a simulation (eventio) file.

Counts are supplied by the selected reader. Shower reuse is counted separately where the file format provides reused events.

Parameters:

event_io_file: str, Path: Path to the eventio file.
file_formatstr: Registered simulation-file reader, by default “eventio”.

Returns:

tuple: Number of showers and number of MC events (MC events for sim_telarray files only).

Input metadata#

Read CORSIKA event-file headers as format-neutral input-metadata fields.

simtools.sim_events.metadata_reader.read_metadata(metadata_file_name)[source]#: Return input-metadata fields of a CORSIKA file (see MetadataCollector).

Writer#

Generate a reduced dataset from simulation files (CORSIKA/sim_telarray) using astropy tables.

class simtools.sim_events.writer.EventDataWriter(input_files, max_files=None, file_format='eventio')[source]#

Process simulation events (CORSIKA/sim_telarray) and write tables to file.

Extracts essential information from simulation files, including:

Shower parameters (energy, core location, direction)
Trigger patterns
Telescope pointing

Attributes:

input_fileslist: List of input file paths to process.
max_filesint, optional: Maximum number of files to process. By default, process all input files.
file_formatstr: Registered simulation-file format, by default “eventio”.

create_tables()[source]#: Create tables from collected data.

get_simulation_input_metadata()[source]#: Return rich provenance records for processed input files.

iter_table_chunks(chunk_size=100000)[source]#: Yield bounded table chunks while processing input files.

class simtools.sim_events.writer.TableSchemas[source]#: Define schemas for output tables with units.

Embedded metadata#

Metadata helpers for reduced simulation event lists.

simtools.sim_events.metadata.build_simulation_metadata(input_files, array_models=None, simulation_software=None)[source]#

Build the versioned simulation metadata document for one output file.

Parameters:

input_fileslist of dict: Rich per-input-file provenance records from EventDataWriter.
array_modelslist of dict, optional: Plain dictionaries exported from resolved ArrayModel instances.
simulation_softwarestr, optional: Simulation software name. When omitted, infer it from input provenance; current inputs without a software field default to sim_telarray.

Returns:

dict: JSON-compatible simulation metadata document.

simtools.sim_events.metadata.build_standard_metadata(args_dict, output_file, product_data_name='reduced_event_data', context=None)[source]#

Collect standard metadata for an embedded HDF5 product.

Parameters:

args_dictdict: Application configuration used to populate metadata.
output_filestr or pathlib.Path: Output HDF5 file represented by the metadata.
product_data_namestr, optional: Product data-model name.
contextRunContext, optional: Run context used to collect the metadata.

Returns:

dict: Standard top-level metadata.

simtools.sim_events.metadata.validate_simulation_metadata(metadata)[source]#: Validate required structural fields of a simulation metadata document.

Histograms#

Histograms for shower and (if available) triggered events.

class simtools.sim_events.histograms.EventDataHistograms(event_data_file, array_name=None, telescope_list=None, energy_bins_per_decade=10, angular_distance_bin_count=100, angular_distance_bin_width=None, core_distance_bin_count=100, core_distance_bin_width=None, skip_invalid_event_data_files=False, require_triggered_data=False)[source]#

Generate and fill histograms for shower and (if available) triggered events.

Event data is read from the reduced MC event data file. Calculate cumulative and relative (efficiency) distributions.

Parameters:

event_data_filestr or list[pathlib.Path | str]: Path to the event-data file or a list of event-data files.
array_namestr, optional: Name of the telescope array configuration (default is None).
telescope_listlist, optional: List of telescope IDs to filter the events (default is None).
energy_bins_per_decadeint, optional: Number of energy bins per decade for logarithmic energy histograms.
angular_distance_bin_widthastropy.units.Quantity, optional: Fixed angular-distance bin width. If omitted, the configured bin count is used.
core_distance_bin_widthastropy.units.Quantity, optional: Fixed core-distance bin width. If omitted, the configured bin count is used.
skip_invalid_event_data_filesbool, optional: Skip invalid event-data files while reading resolved input files.
require_triggered_databool, optional: Require triggered-event tables in each input file.

accumulate(file_info_table, shower_data, event_data, triggered_data, source_file=None)[source]#: Accumulate one already-read event dataset into all configured histograms.

calculate_cumulative_data()[source]#

Calculate cumulative distributions for triggered histograms.

Returns:

dict: Dictionary containing the cumulative histograms.

calculate_efficiency_data()[source]#

Calculate efficiency histograms (triggered divided by simulated).

Assumes that for each histogram with simulated events, there is a corresponding histogram with triggered events.

Returns:

dict: Dictionary containing the efficiency histograms.

property core_distance_bin_count#: Return the configured count for count-based core-distance bins.

property core_distance_bin_width#: Return the configured fixed core-distance bin width, if any.

property core_distance_bins#

Return bins for the core distance histogram.

CORSIKA CSCAT (‘core_scatter_max’) is defined in the shower plane.

classmethod create_accumulator(array_name=None, telescope_list=None, energy_bins_per_decade=10, angular_distance_bin_count=100, angular_distance_bin_width=None, core_distance_bin_count=100, core_distance_bin_width=None)[source]#

Create an empty histogram accumulator for externally supplied event data.

This avoids opening the same input files for every telescope configuration when one caller reads the event data once and distributes it to several accumulators.

property energy_bins#

Return bins for the energy histogram.

Align bins to full decades of energy, using the configured bins per decade, and ensure that the range covers the energy range of the events.

Returns:

np.ndarray Array of energy bin edges in TeV.

property event_data_file#: Return the first resolved event-data file, or all files for an accumulator.

fill(fill_efficiency_histogram=True)[source]#

Fill histograms with event data.

Involves looping over all event data, and therefore is the slowest part of the histogram module. Adds the histograms to the histogram dictionary.

Assume that all event data files are generated with similar configurations (self.file_info contains the file info of the last file).

Parameters:

fill_efficiency_histogrambool, optional: Whether to calculate and fill the efficiency histograms.

finalize(fill_efficiency_histogram=True)[source]#: Finalize accumulated histograms after all event datasets have been added.

get_histogram_definition(event_data_column=None, event_data=None, histogram=None, bin_edges=None, title=None, axis_titles=None, suffix=None, is_1d=True, plot_scales=None)[source]#: Return a single histogram definition (histogram is None until filled).

iter_event_data()[source]#: Yield reduced event datasets sequentially, keeping input memory bounded.

print_summary()[source]#

Print a summary of the histogram statistics.

Total number of events is retrieved from the ‘energy’ histograms.

set_loaded_histograms(histograms, file_info=None, data_ranges=None, loaded_bin_edges=None, contains_triggered_data=True)[source]#: Install histogram data loaded from a serialized histogram product.

property view_cone_bins#: Return bins for the viewcone histogram.

Validators#

Validation of run outputs: sim_telarray, CORSIKA, and reduced event list files.

simtools.sim_events.validate_run_outputs.assert_events_of_type(file, event_type='shower')[source]#

Assert that events of the expected type are present in the sim_telarray file.

Parameters:

file: Path: sim_telarray file.
event_type: str: Expected event type (e.g., “shower”, “flasher”).

simtools.sim_events.validate_run_outputs.assert_expected_sim_telarray_metadata(file, expected_sim_telarray_metadata)[source]#

Assert that expected metadata is present in the sim_telarray file.

Parameters:

file: Path: sim_telarray file.
expected_sim_telarray_metadata: dict: Expected metadata values.

simtools.sim_events.validate_run_outputs.assert_expected_sim_telarray_output_and_event_type(file, expected_sim_telarray_output, event_type='shower')[source]#

Validate expected event values and type during one file traversal.

Parameters:

file: Path: Sim_telarray output file.
expected_sim_telarray_output: dict or None: Expected event values and optional event type.
event_type: str: Expected event type (default: “shower”).

Returns:

bool: True when all requested event checks pass.

simtools.sim_events.validate_run_outputs.assert_n_showers_and_energy_range(file)[source]#

Assert the number of showers (1% tolerance) and the energy range of a sim_telarray file.

Parameters:

file: Path: sim_telarray file.

simtools.sim_events.validate_run_outputs.assert_sim_telarray_metadata(file, array_model, allow_for_changes=None)[source]#

Assert consistency of sim_telarray metadata with given array model.

Parameters:

file: Path: sim_telarray file.
array_model: ArrayModel: Array model to compare with.
allow_for_changes: list: Model parameters changed with ‘-C’ (metadata may differ for these).

simtools.sim_events.validate_run_outputs.is_equal(value1, value2, value_type)[source]#

Check if two values are equal based on their type.

Parameters:

value1, value2: any: Values to compare.
value_type: str: Type of the values (‘string’, ‘dict’, etc.).

Returns:

bool: True if the values are equal, False otherwise.

simtools.sim_events.validate_run_outputs.validate_corsika_event_numbers(data_files, expected_shower_events=None, tolerance=0.001)[source]#

Validate the number of simulated events in CORSIKA output files.

A small relative mismatch (tolerance) is only logged as a warning.

Parameters:

data_files: list of Path: CORSIKA data files.
expected_shower_events: int: Expected number of simulated shower events.
tolerance: float: Relative tolerance for a small mismatch.

simtools.sim_events.validate_run_outputs.validate_corsika_logs(log_files, expected_shower_events=None, curved_atmo=False)[source]#

Validate CORSIKA log files (raises ValueError if the expected patterns are missing).

Parameters:

log_files: list of Path: CORSIKA log files.
expected_shower_events: int, optional: Expected number of shower events.
curved_atmo: bool, optional: Whether CORSIKA was run with the curved atmosphere option.

simtools.sim_events.validate_run_outputs.validate_corsika_output(data_files, log_files, expected_shower_events=None, curved_atmo=False)[source]#

Validate CORSIKA output files.

Parameters:

data_files, log_files: list of Path: CORSIKA data and log files.
expected_shower_events: int, optional: Expected number of shower events.
curved_atmo: bool, optional: Whether CORSIKA was run with the curved atmosphere option.

simtools.sim_events.validate_run_outputs.validate_reduced_event_data_file(data_file)[source]#

Validate structure, tables, metadata, and references of one reduced-event file.

Parameters:

data_file: str or pathlib.Path: HDF5 reduced-event file to validate.

Returns:

bool: True when the file passes all checks.

simtools.sim_events.validate_run_outputs.validate_sim_events(data_files, expected_mc_events)[source]#

Validate that the number of simulated events in reduced event lists matches the expectation.

Parameters:

data_files: str, Path, list: Reduced event list files (HDF5).
expected_mc_events: int: Expected number of simulated MC events.

simtools.sim_events.validate_run_outputs.validate_sim_telarray(data_files, log_files, array_models=None, expected_mc_events=None, expected_shower_events=None, curved_atmo=False, allow_for_changes=None)[source]#

Validate sim_telarray output files and metadata (raises ValueError if inconsistent).

Parameters:

data_files, log_files: list of Path: sim_telarray output and log files.
array_models: list of ArrayModel: Array models to compare with.
expected_mc_events, expected_shower_events: int: Expected numbers of MC and shower events.
curved_atmo: bool: CORSIKA executable compiled with curved atmosphere option.
allow_for_changes: list: Model parameters changed with ‘-C’ (metadata may differ for these).

simtools.sim_events.validate_run_outputs.validate_sim_telarray_event_numbers(data_files, expected_mc_events, expected_shower_events)[source]#

Verify the number of simulated MC and shower events in sim_telarray output files.

Parameters:

data_files: list of Path: sim_telarray output files.
expected_mc_events, expected_shower_events: int: Expected numbers of simulated MC and shower events.

simtools.sim_events.validate_run_outputs.validate_sim_telarray_logs(log_files, expected_mc_events=None, expected_shower_events=None, curved_atmo=False)[source]#: Validate sim_telarray log files.

simtools.sim_events.validate_run_outputs.validate_sim_telarray_metadata(files, array_models, allow_for_changes=None)[source]#: Validate metadata in the sim_telarray output files.

simtools.sim_events.validate_run_outputs.validate_trigger_histogram_file(data_file)[source]#

Validate structure, tables, metadata, and references of one trigger-histogram file.

Parameters:

data_file: str or pathlib.Path: HDF5 trigger-histogram file to validate.

Returns:

bool: True when the file passes all checks.

simtools.sim_events.validate_run_outputs.validate_trigger_histograms_file(data_file)[source]#: Validate a trigger-histogram product using its registered product name.

Native configuration and file formats#

Select simulation-file readers independently of simulation software.

class simtools.sim_events.formats.readers.EventReader(*args, **kwargs)[source]#

Read run information, metadata, parameters, and reduced event records.

count_events()[source]#: Return shower and event counts.

iter_records(file_id=0)[source]#: Yield reduced event table names and row dictionaries.

read_metadata()[source]#: Return metadata collected while iterating reduced records.

read_run_number()[source]#: Return the run number, or None when absent.

read_simulation_parameters()[source]#: Return physical shower settings.

simtools.sim_events.formats.readers.available_formats()[source]#: Return the registered simulation-file format names.

simtools.sim_events.formats.readers.get_reader(file_name, file_format='eventio')[source]#

Create a reader for a simulation file.

Parameters:

file_namestr or pathlib.Path: Simulation input file.
file_formatstr: Registered format name. The default supports current IACT and simtel files.

Returns:

EventReader: Reader supplied by the selected factory.

Raises:

ValueError: If the requested format is not registered.

Read CORSIKA IACT and sim_telarray eventio files.

class simtools.sim_events.formats.eventio_reader.EventioReader(file_name)[source]#

Read simulation information and reduced event rows from eventio files.

Parameters:

file_namestr or pathlib.Path: Simulation input file.

count_events()[source]#: Return shower and reused-event counts.

get_nsb_level_from_sim_telarray_metadata(file)[source]#

Return NSB level from sim_telarray metadata.

Falls back to preliminary NSB level if not found.

Parameters:

filePath: Path to the sim_telarray file.

Returns:

float: NSB level.

iter_records(file_id=0)[source]#

Yield (table name, row) pairs in reduced-table units.

Shower rows are retained until all reuse positions have been read. Memory use is bounded by one shower’s reuse count.

Parameters:

file_idint: Identifier assigned by the reduced-table writer.

Yields:

tuple: Table name and dictionary matching the reduced-table schema.

read_combined_run_header()[source]#: Return merged sim_telarray run information, or None.

read_metadata()[source]#: Return metadata collected while iterating reduced records.

read_run_headers()[source]#: Return the CORSIKA IACT run and first event headers.

read_run_number()[source]#: Return the shower run number, or None if absent.

read_simulation_parameters()[source]#

Return physical shower settings in degrees and metres.

Returns:

dict: Settings accepted by SimulationParameters, plus observation levels in metres. CORSIKA header angles are converted to geographic pointing.

simtools.sim_events.formats.eventio_reader.get_combined_eventio_run_header(sim_telarray_file)[source]#

Return the CORSIKA run header information from an eventio (sim_telarray) file.

Reads both RunHeader and MCRunHeader object from file and returns a merged dictionary. Adds primary id from the first event.

Parameters:

sim_telarray_file: str: Path to the sim_telarray file.

Returns:

dict, None: CORSIKA run header. Returns None if not found.

simtools.sim_events.formats.eventio_reader.get_corsika_run_and_event_headers(corsika_iact_file)[source]#

Return the CORSIKA run and event headers from a CORSIKA IACT eventio file.

Parameters:

corsika_iact_file: str, Path: Path to the CORSIKA IACT eventio file.

Returns:

tuple: CORSIKA run header and event header as dictionaries.

simtools.sim_events.formats.eventio_reader.get_corsika_run_number(file)[source]#

Return the CORSIKA run number from an eventio (CORSIKA IACT or sim_telarray) file.

Parameters:

file: str: Path to the eventio file.

Returns:

int, None: CORSIKA run number. Returns None if not found.

simtools.sim_events.formats.eventio_reader.get_simulated_events(event_io_file)[source]#

Return the number of shower and MC events from a simulation (eventio) file.

For a sim_telarray file, the number of simulated showers and MC events is determined by counting the number of MCShower (type id 2020) and MCEvent objects (type id 2021). For a CORSIKA IACT file, the number of simulated showers is determined by counting the number of IACTShower (type id 1202).

Parameters:

event_io_file: str, Path: Path to the eventio file.

Returns:

tuple: Number of showers and number of MC events (MC events for sim_telarray files only).

production_comparison_statistics#

Compute statistics for comparisons of simulation productions.

simtools.sim_events.production_comparison_statistics.aggregate_mixed_trigger_combinations(trigger_combinations)[source]#: Aggregate mixed trigger combinations by display label in one pass.

simtools.sim_events.production_comparison_statistics.build_aligned_count_statistics(metrics_per_production, counts_per_label, metadata, metric='jensen_shannon', bin_edges=None)[source]#: Build comparison statistics for plots represented by aligned count arrays.

simtools.sim_events.production_comparison_statistics.build_quantity_distribution_statistics(metrics_per_production, quantity_name, counts_by_label, cumulative)[source]#: Build comparison statistics for quantity distribution plots.

simtools.sim_events.production_comparison_statistics.compact_plot_statistics(plot_statistics)[source]#: Keep JSON scalar-only so diagnostic histogram arrays do not create huge files.

simtools.sim_events.production_comparison_statistics.comparison_statistics_annotation(statistics)[source]#: Return comparison metric text for baseline-vs-candidate comparisons.

simtools.sim_events.production_comparison_statistics.fraction_with_poisson_errors(counts, _metrics=None)[source]#: Return normalized bin fractions and Poisson errors for counts.

simtools.sim_events.production_comparison_statistics.fractions_per_triggered_events(counts, metrics)[source]#: Normalize counts by triggered-event count and return Poisson errors.

simtools.sim_events.production_comparison_statistics.get_trigger_multiplicity_counts(metrics, bin_edges)[source]#: Return trigger multiplicity counts for the requested bin edges.

simtools.sim_events.production_comparison_statistics.initialize_comparison_statistics(metrics_per_production)[source]#: Initialize top-level comparison statistics structure.

simtools.sim_events.production_comparison_statistics.record_plot_statistics(comparison_statistics, plot_name, statistics)[source]#: Store plot-level statistics if available.

simtools.sim_events.production_comparison_statistics.record_telescope_type_participation(comparison_statistics, metrics_per_production)[source]#: Record participation-fraction comparisons for each telescope type.

simtools.sim_events.production_comparison_statistics.write_comparison_statistics_json(comparison_statistics, output_path)[source]#: Write comparison statistics JSON file in the output directory.