I/O

This module include I/O related functionality.

io_handler

Handle input and output directories and file paths.

simtools.io.io_handler.resolve_test_resource_paths(value, test_resources_path=None)

Resolve test-resource macros and canonical test-resource paths recursively.

Parameters:

valueobject

Configuration value, mapping, or sequence to resolve.

test_resources_pathstr or pathlib.Path, optional

Base directory containing the static, generated, and downloaded resource directories. The integration-test harness passes this argument explicitly.

Returns:

object

Configuration with absolute test-resource paths.

output_paths

Explicit output and model configuration paths.

class simtools.io.output_paths.OutputPaths(output_path=None, model_path=None)

Hold output and model paths for one run.

get_model_configuration_directory(model_version, sub_dir=None)

Return the output directory for model configuration files.

get_output_directory(sub_dir=None, output_path_label='default')

Create and return an output directory.

get_output_file(file_name, sub_dir=None, output_path_label='default')

Return a path to an output file, creating its parent directory.

ascii_handler

Helper module for ASCII file operations.

class simtools.io.ascii_handler.JsonNumpyEncoder(*args, compact_numeric_lists=False, **kwargs)

Convert numpy to python types as accepted by json.dump.

Lists whose elements are all numbers (int or float) can be serialized on a single line when compact_numeric_lists is enabled.

default(o)

Return default encoder.

encode(o)

Encode with compact inner numeric lists.

simtools.io.ascii_handler.collect_data_from_bytes(data, file_name, yaml_document=None)

Collect JSON or YAML data from an in-memory byte string.

Parameters:

databytes

UTF-8 encoded file contents.

file_namestr or pathlib.Path

Logical file name used to select the parser and report errors.

yaml_documentint, optional

YAML document index for multi-document YAML files.

Returns:

dict or list

Parsed file contents.

simtools.io.ascii_handler.collect_data_from_file(file_name, yaml_document=None, test_resources_path=None)

Collect data from file based on its extension.

Parameters:

file_name: str

Name of the yaml/json/ascii file.

yaml_document: None, int

Return list of yaml documents or a single document (for yaml files with several documents).

test_resources_pathstr or pathlib.Path, optional

Explicit base directory for resolving test-resource paths in structured files.

Returns:

data: dict or list

Data as dict or list.

simtools.io.ascii_handler.collect_data_from_git(file_name, git_repository, git_branch='main')

Download yaml/json/list data from a git repository via its raw URL interface.

Supports both repository URLs (with or without .git suffix) and raw URL bases containing /-/raw/.

Parameters:

file_name: str or Path

File path inside repository (e.g. path/to/file.json).

git_repository: str

Git repository URL or raw URL base.

git_branch: str

Branch or tag to use when reading from raw endpoint.

Returns:

dict or list

Parsed file content.

simtools.io.ascii_handler.collect_data_from_http(url)

Download yaml or json file from url and return it contents as dict.

File is downloaded as a temporary file and deleted afterwards.

Parameters:

url: str

URL of the yaml/json file.

Returns:

dict

Dictionary containing the file content.

Raises:

TypeError

If url is not a valid URL.

FileNotFoundError

If downloading the yaml file fails.

simtools.io.ascii_handler.read_file_encoded_in_utf_or_latin(file_name)

Read a file encoded in UTF-8 or Latin-1.

Parameters:

file_name: str

Name of the file to be read.

Returns:

list

List of lines read from the file.

Raises:

UnicodeDecodeError

If the file cannot be decoded using UTF-8 or Latin-1.

simtools.io.ascii_handler.to_builtin(data)

Convert common values to plain Python types for serialization.

simtools.io.ascii_handler.write_data_to_file(data, output_file, sort_keys=False, numpy_types=False, unique_lines=False, compact_numeric_lists=False)

Write structured data to JSON, YAML, or text file.

The file type is determined by the file extension.

Parameters:

data: dict or list

Data to be written to the file.

output_file: str or Path

Name of the file to be written.

sort_keys: bool, optional

If True, sort the keys.

numpy_types: bool, optional

If True, convert numpy types to native Python types.

unique_lines: bool, optional

If True, write only unique lines (applicable for text files).

compact_numeric_lists: bool, optional

If True, serialize numeric lists on a single line for JSON output.

table_handler

IO operations on astropy tables.

simtools.io.table_handler.group_table_rows(table, column_names)

Group astropy table rows by one or more columns.

Parameters:

tableastropy.table.Table

Table to group.

column_namesstr or sequence[str]

Column name or names used for grouping.

Returns:

dict

Mapping from group key to grouped sub-table. Single-column grouping returns scalar keys, multi-column grouping returns tuple keys.

simtools.io.table_handler.read_metadata_document(file, document_name)

Read one embedded JSON metadata document from an HDF5 file.

simtools.io.table_handler.read_table_file_type(input_files)

Determine the file type of the input files.

All input files must be of the same type (either all HDF5 or all FITS).

Parameters:

input_fileslist of str

List of input file paths.

Returns:

str

File type (‘HDF5’ or ‘FITS’).

simtools.io.table_handler.read_table_from_hdf5(file, table_name, columns=None)

Read a single astropy table from an HDF5 file.

Parameters:

filestr or Path

Path to the input HDF5 file.

table_namestr

Name of the table to read.

columnslist[str], optional

List of columns to read from a compound HDF5 dataset. If None, all columns are read.

Returns:

astropy.table.Table

The requested astropy table.

simtools.io.table_handler.read_table_list(input_file, table_names, include_indexed_tables=False)

Read available tables found in the input file.

If table_counter is True, search for tables with the same name but with different suffixes (e.g., “_0”, “_1”, etc.).

Parameters:

input_filestr or Path

Path to the input file (HDF5 or FITS).

table_nameslist of str

List of table names to search for in the input file.

include_indexed_tablesbool, optional

If True, include tables with indexed suffixes (e.g., “_0”, “_1”, etc.) in the search.

Returns:

dict

Dictionary with table names as keys and lists of found table names as values.

simtools.io.table_handler.read_tables(file, table_names, file_type=None, table_columns=None)

Read tables from a file.

Parameters:

filestr

Path to the input file.

table_nameslist of str

List of table names to read.

file_typestr

Type of the input file (‘HDF5’ or ‘FITS’).

table_columnsdict, optional

Mapping of table name to list of columns to read. Used for HDF5 only.

Returns:

dict

Dictionary with table names as keys and astropy tables as values.

simtools.io.table_handler.write_table_chunks(table_chunks, output_file, overwrite_existing=True, metadata_documents=None, activity_id=None)

Write table chunks and optional JSON documents to atomic HDF5 output.

Parameters:

table_chunksiterable

Tables to write, grouped into chunks.

output_filestr or Path

Path of the published HDF5 output file.

overwrite_existingbool, optional

Whether an existing output file may be replaced.

metadata_documentsdict or callable, optional

Named JSON documents to embed in the HDF5 output.

activity_idstr, optional

Application activity identifier. If supplied, it is included in the incomplete-file name to correlate failed writes with the application log.

simtools.io.table_handler.write_tables(tables, output_file, overwrite_existing=True, file_type=None, metadata_documents=None)

Write tables to file (overwriting if exists).

HDF5 files are written to a sibling .incomplete-<uuid> file, validated, and atomically moved to the requested output path. If writing fails, the incomplete file is retained for diagnosis and any existing output remains unchanged.

Parameters:

tableslist or dict

List or Dictionary with astropy tables as values.

output_filestr or Path

Path to the output file.

overwrite_existingbool

If True, overwrite the output file if it exists.

file_typestr

Type of the output file (‘HDF5’ or ‘FITS’).

metadata_documentsdict or callable, optional

Named JSON documents to embed in an HDF5 output. A callable is evaluated after all table chunks have been consumed.

Returns:

None

file_inspector

Inspect simtools-related files and return structured summaries.

simtools.io.file_inspector.inspect_file(file_path, max_entries=50, format_report=True, entry_name=None)

Inspect one supported file and return one or more reports.

Parameters:

file_pathstr or Path

Path to the file to inspect.

max_entriesint, optional

Maximum number of entries to include in collection-style summaries.

format_reportbool, optional

Return formatted strings instead of raw dictionaries.

entry_namestr, optional

Print one root HDF5 dataset instead of the structure report.

Returns:

list[str] or list[dict]

Inspection reports. HDF5 trigger-histogram files may yield both a generic HDF5 report and a trigger-histogram-specific consistency report.

simtools.io.file_inspector.inspect_hdf5_entry(file_path, entry_name, max_entries=50)

Return the content of one HDF5 root dataset.

JSON metadata datasets are decoded as formatted JSON. Compound datasets are read through the existing HDF5 table reader; other datasets are displayed using their scalar or array representation.

Parameters:

file_pathstr or Path

Path to the HDF5 file.

entry_namestr

Root dataset name to read.

max_entriesint, optional

Maximum number of table rows to print. Non-positive values disable truncation.

Returns:

str

Formatted dataset content.

Raises:

ValueError

If the file is not a valid HDF5 container or the root entry is not a dataset.

KeyError

If the requested root entry does not exist.

simtools.io.file_inspector.inspect_hdf5_file(file_path, max_entries=50, format_report=True)

Inspect one HDF5 file and return a report.

simtools.io.file_inspector.inspect_json_or_yaml_file(file_path, max_entries=50, format_report=True)

Inspect one JSON or YAML file and return a report.

simtools.io.file_inspector.inspect_sim_telarray_file(file_path, max_entries=50, format_report=True)

Inspect one sim_telarray file and return a report.

simtools.io.file_inspector.inspect_table_file(file_path, max_entries=50, format_report=True)

Inspect one tabular file and return a report.

simtools.io.file_inspector.inspect_text_file(file_path, max_entries=50, format_report=True)

Inspect one plain-text file and return a report.

file_type

Shared simulation file formats and product suffixes, generated default suffixes, fixed filenames, and descriptive lifecycle roles are declared in src/simtools/resources/file_types.yml. Keep product-specific placement and workflow cleanup behavior with their owning modules; the registry does not enforce deletion or content validation.

Generic helpers for matching and validating supported file types.

simtools.io.file_type.is_file_type(file_path, file_type)

Return whether the path matches one registered file type.

simtools.io.file_type.looks_like_text_file(file_path, sample_size=4096)

Return whether the file appears to be UTF-8 text.

simtools.io.file_type.matches_suffix(file_path, expected_suffixes)

Return whether the path ends with one of the expected suffix patterns.

simtools.io.file_type.product_filename(product_id)

Return a fixed filename for one registered file product.

simtools.io.file_type.product_suffixes(product_id)

Return accepted suffixes for one registered file product.

simtools.io.file_type.product_value(product_id, key)

Return one value from a registered file product.

simtools.io.file_type.validate_file_type(file_path, file_type)

Validate that a file has one of the expected suffixes.

Parameters:

file_pathstr or Path

Path to the file to validate.

file_typestr

Registered file type name, e.g., “hdf5”, “json_or_yaml”,

legacy_data_handler

Reading of legacy data files (expect that this will be obsolete in future).

simtools.io.legacy_data_handler.read_legacy_data_as_table(file_path, file_type)

Read legacy data file.

Parameters:

file_path: Path

Path to the legacy data file.

file_type: str

Type of legacy data file.

Returns:

Table

Astropy table.

Raises:

ValueError

If unsupported legacy data file type.

simtools.io.legacy_data_handler.read_legacy_lst_single_pe(file_path)

Read LST single pe file (in legacy data format).

File contains two columns: amplitude (in units of single p.e) and response.

Parameters:

file_path: Path

Path to the legacy data file.

Returns:

Table

Astropy table.