Utilities

In this section you find the reference documentation of the modules contained in the util module.

• general

• names

general

General functions useful across different parts of the code.

utils.general.change_dict_keys_case(data_dict, lower_case=True)

Change keys of a dictionary to lower or upper case recursively.

Parameters:

data_dict: dict

Dictionary to be converted.

lower_case: bool

Change keys to lower (upper) case if True (False).

Returns:

dict

Dictionary with keys converted to lower or upper case.

utils.general.clear_default_sim_telarray_cfg_directories(command)

Prefix the command to clear default sim_telarray configuration directories.

Parameters:

command: str

Command to be prefixed.

Returns:

str

Prefixed command.

utils.general.collect_final_lines(file, n_lines)

Collect final lines.

Parameters:

file: str or Path

File to collect the lines from.

n_lines: int

Number of lines to be collected.

Returns:

str

Final lines collected.

utils.general.collect_kwargs(label, in_kwargs)

Collect kwargs of the type label_* and return them as a dict.

Parameters:

label: str

Label to be collected in kwargs.

in_kwargs: dict

kwargs.

Returns:

dict

Dictionary with the collected kwargs.

utils.general.convert_keys_in_dict_to_lowercase(data)

Recursively convert all dictionary keys to lowercase.

Parameters:

data: dict

Dictionary to be converted.

Returns:

dict

Dictionary with all keys converted to lowercase.

utils.general.convert_list_to_string(data, comma_separated=False, shorten_list=False, collapse_list=False)

Convert arrays to string (if required).

Parameters:

data: object

Object of data to convert (e.g., double or list)

comma_separated: bool

If True, returns elements as a comma-separated string (default is space-separated).

shorten_list: bool

If True and all elements in the list are identical, returns a summary string like “all: value”. This is useful to make the configuration files more readable.

collapse_list: bool

If True and all elements in the list are identical, returns a single value instead of the entire list.

Returns:

object or str:

Converted data as string (if required)

utils.general.convert_string_to_list(data_string, is_float=True, force_comma_separation=False)

Convert string (as used e.g. in sim_telarray) to list.

Allow coma or space separated strings.

Parameters:

data_string: object

String to be converted

is_float: bool

If True, convert to float, otherwise to int.

force_comma_separation: bool

If True, force comma separation.

Returns:

list, str

Converted data from string (if required). Return data_string if conversion fails.

utils.general.ensure_iterable(value)

Return input value as iterable.

• Single values will return as a list with a single element.

• None values will return as empty list.

• Values of list or tuple type are not changed.

Parameters:

value: any

Input value to be converted to a iterable.

Returns:

list or tuple

Converted value as list or tuple.

utils.general.find_differences_in_json_objects(obj1, obj2, path='')

Recursively find differences between two JSON-like objects.

Parameters:

obj1: dict, list, or any

First object to compare.

obj2: dict, list, or any

Second object to compare.

path: str

Path to the current object in the JSON structure, used for reporting differences.

Returns:

list

List of differences found between the two objects, with paths indicating where the differences occur.

utils.general.find_file(name, loc)

Search for files inside of given directories, recursively, and return its full path.

Parameters:

name: str

File name to be searched for.

loc: Path or list of Path

Location of where to search for the file.

Returns:

Path

Full path of the file to be found if existing. Otherwise, None.

Raises:

FileNotFoundError

If the desired file is not found.

utils.general.get_file_age(file_path)

Get the age of a file in seconds since the last modification.

utils.general.get_list_of_files_from_command_line(file_names, suffix_list)

Get a list of files from the command line.

Files can be given as a list of file names or as a text file containing the list of files. The list of suffixes restrict the files types to be returned. Note that a file list must have a different suffix than those in the suffix list.

Parameters:

file_names: list

List of file names to be checked.

suffix_list: list

List of suffixes to be checked.

Returns:

list

List of files with the given suffixes.

utils.general.get_log_excerpt(log_file, n_last_lines=30)

Get an excerpt from a log file, namely the n_last_lines of the file.

Parameters:

log_file: str or Path

Log file to get the excerpt from.

n_last_lines: int

Number of last lines of the file to get.

Returns:

str

Excerpt from log file with header/footer

utils.general.get_log_level_from_user(log_level)

Map between logging level from the user to logging levels of the logging module.

Parameters:

log_level: str

Log level from the user.

Returns:

logging.LEVEL

The requested logging level to be used as input to logging.setLevel().

utils.general.get_structure_array_from_table(table, column_names)

Get a structured array from an astropy table for a selected list of columns.

Parameters:

table: astropy.table.Table

Table to be converted.

column_names: list

List of column names to be included in the structured array.

Returns:

numpy.ndarray

Structured array containing the table data.

utils.general.is_url(url)

Check if a string is a valid URL.

Parameters:

url: str

String to be checked.

Returns:

bool

True if url is a valid URL.

utils.general.now_date_time_in_isoformat()

Return date and time in isoformat and second accuracy.

utils.general.pack_tar_file(tar_file_name, file_list, sub_dir=None)

Pack files into a tar.gz archive.

Parameters:

tar_file_name: str

Name of the output tar.gz file.

file_list: list

List of files to include in the archive.

sub_dir: str, optional

Subdirectory within the archive to place the files.

utils.general.program_is_executable(program)

Check if program exists and is executable.

Follows https://stackoverflow.com/questions/377017/

utils.general.remove_key_from_dict(data, key_to_remove)

Remove a specific key from a dictionary recursively.

Parameters:

data: dict

Dictionary to be processed.

key_to_remove: str

Key to be removed from the dictionary.

Returns:

dict

Dictionary with the specified key removed.

utils.general.remove_substring_recursively_from_dict(data_dict, substring='\n')

Remove substrings from all strings in a dictionary.

Recursively crawls through the dictionary This e.g., allows to remove all newline characters from a dictionary.

Parameters:

data_dict: dict

Dictionary to be converted.

substring: str

Substring to be removed.

Raises:

AttributeError:

if input is not a proper dictionary.

utils.general.resolve_file_patterns(file_names)

Return a list of files names from string, list, or wildcard pattern.

Parameters:

file_names: str, list

File names to be searched for (wildcards allowed).

Returns:

list

List of file names found.

utils.general.set_default_kwargs(in_kwargs, **kwargs)

Fill in a dict with a set of default kwargs and return it.

Parameters:

in_kwargs: dict

Input dict to be filled in with the default kwargs.

**kwargs:

Default kwargs to be set.

Returns:

dict

Dictionary containing the default kwargs.

utils.general.url_exists(url)

Check if a URL exists.

Parameters:

url: str

URL to be checked.

Returns:

bool

True if URL exists.

utils.general.user_confirm()

Ask the user to enter y or n (case-insensitive) on the command line.

Returns:

bool:

True if the answer is Y/y.

utils.general.validate_data_type(reference_dtype, value=None, dtype=None, allow_subtypes=True)

Validate data type of value or type object against a reference data type.

Allow to check for exact data type or allow subtypes (e.g. uint is accepted for int). Take into account ‘file’ type as used in the model parameter database.

Parameters:

reference_dtype: str

Reference data type to be checked against.

value: any, optional

Value to be checked (if dtype is None).

dtype: type, optional

Type object to be checked (if value is None).

allow_subtypes: bool, optional

If True, allow subtypes to be accepted.

Returns:

bool:

True if the data type is valid.

geometry

A collection of functions related to geometrical transformations.

utils.geometry.calculate_circular_mean(angles)

Calculate circular mean of angles in radians.

Parameters:

angles: numpy.array

Array of angles in radians.

Returns:

float

Circular mean of the angles.

utils.geometry.convert_2d_to_radial_distr(hist_2d, xaxis, yaxis, bins=50, max_dist=1000)

Convert a 2d histogram of positions, e.g. photon positions on the ground, to a 1D distribution.

Parameters:

hist_2d: numpy.ndarray

The histogram counts.

xaxis: numpy.array

The values of the x axis (histogram bin edges) on the ground.

yaxis: numpy.array

The values of the y axis (histogram bin edges) on the ground.

bins: float

Number of bins in distance.

max_dist: float

Maximum distance to consider in the 1D histogram, usually in meters.

Returns:

np.array

The values of the 1D histogram with size = int(max_dist/bin_size).

np.array

The bin edges of the 1D histogram with size = int(max_dist/bin_size) + 1.

utils.geometry.fiducial_radius_from_shape(width, shape)

Calculate minimum radius including different geometrical shapes.

Assumes definition of shapes as in ‘camera_body_shape’ model parameter:

• circle: shape = 0, width is diameter

• hexagon: shape = 1 or 3, width is flat-to-flat distance

• square: shape = 2, width is side length

Parameters:

widthfloat

Characteristic width

shapeint

Geometrical shape parameter

Returns:

float

Minimum fiducial radius

utils.geometry.rotate(x, y, rotation_around_z_axis, rotation_around_y_axis=0)

Rotate the x and y coordinates of the telescopes.

The two rotations are:

• rotation_angle_around_z_axis gives the rotation on the observation plane (x, y)

• rotation_angle_around_y_axis allows to rotate the observation plane in space.

The function returns the rotated x and y values in the same unit given. The direction of rotation of the elements in the plane is counterclockwise, i.e., the rotation of the coordinate system is clockwise.

Parameters:

x: numpy.array or list

x positions of the entries (e.g. telescopes), usually in meters.

y: numpy.array or list

y positions of the entries (e.g. telescopes), usually in meters.

rotation_angle_around_z_axis: astropy.units.rad

Angle to rotate the array in the observation plane (around z axis) in radians.

rotation_angle_around_y_axis: astropy.units.rad

Angle to rotate the observation plane around the y axis in radians.

Returns:

2-tuple of list

x and y positions of the rotated entry (e.g. telescopes) positions.

Raises:

TypeError:

If type of x and y parameters are not valid.

RuntimeError:

If the length of x and y are different.

UnitsError:

If the unit of x and y are different.

utils.geometry.solid_angle(angle_max, angle_min=<Quantity 0. rad>)

Calculate the solid angle subtended by a given range of angles.

Parameters:

angle_max: astropy.units.Quantity

The maximum angle for which to calculate the solid angle.

angle_min: astropy.units.Quantity

The minimum angle for which to calculate the solid angle (default is 0 rad).

Returns:

astropy.units.Quantity

The solid angle subtended by the given range of angles (in steradians).

utils.geometry.transform_ground_to_shower_coordinates(x_ground, y_ground, z_ground, azimuth, altitude)

Transform ground to shower coordinates.

Assume ground to be of type ‘North-West-Up’ (NWU) coordinates.

Parameters:

x_ground: numpy.array

Ground x coordinate.

y_ground: numpy.array

Ground y coordinate.

z_ground: numpy.array

Ground z coordinate.

azimuth: numpy.array

Azimuth angle of the shower (in radians).

altitude: numpy.array

Altitude angle of the shower (in radians).

Returns:

tuple

Transformed shower coordinates (x’, y’, z’).

names

Name utilities for array elements, sites, and model parameters.

Naming in simtools:

• ‘site’: South or North

• ‘array element’: e.g., LSTN-01, MSTN-01, …

• ‘array element type’: e.g., LSTN, MSTN, …

• ‘array element ID’: e.g., 01, 02, …

• ‘array element design type’: e.g., design, test

• ‘instrument class key’: e.g., telescope, camera, structure

• ‘db collection’: e.g., telescopes, sites, calibration_devices

utils.names.array_element_common_identifiers()

Get array element IDs from CTAO common identifier.

Returns:

dict, dict

Dictionary mapping array element names to their IDs and vice versa.

utils.names.array_element_design_types(array_element_type)

Get array element design type (e.g., ‘design’ or ‘flashcam’).

Default values are [‘design’, ‘test’].

Parameters:

array_element_type

Array element type

Returns:

list

Array element design types.

utils.names.array_elements()

Get array elements and their properties.

Returns:

dict

Array elements.

utils.names.db_collection_to_instrument_class_key(collection_name='telescopes')

Return list of instrument classes for a given collection.

utils.names.file_name_with_version(file_name, suffix)

Return a file name including a semantic version with the correct suffix.

Replaces ‘Path.suffix()’, which removes trailing numbers (and therefore version numbers).

Parameters:

file_name: str

File name.

suffix: str

File suffix.

Returns:

Path

File name with version number.

utils.names.generate_array_element_name_from_type_site_id(array_element_type, site, array_element_id)

Generate a new array element name from array element type, site, and array element ID.

Parameters:

array_element_type: str

Array element type.

site: str

Site name.

array_element_id: str

Array element ID.

Returns:

str

Array element name.

utils.names.generate_file_name(file_type, suffix, site, telescope_model_name, zenith_angle, azimuth_angle=None, off_axis_angle=None, source_distance=None, mirror_number=None, label=None, extra_label=None)

Generate a file name for output, config, or plotting.

Used e.g., to generate camera_efficiency and ray_tracing output files.

Parameters:

file_type: str

Type of file (e.g., config, output, plot)

suffix: str

File suffix

site: str

South or North.

telescope_model_name: str

LSTN-01, MSTS-01, …

zenith_angle: float

Zenith angle (deg).

azimuth_angle: float

Azimuth angle (deg).

off_axis_angle: float

Off-axis angle (deg).

source_distance: float

Source distance (km).

mirror_number: int

Mirror number.

label: str

Instance label.

extra_label: str

Extra label.

Returns:

str

File name.

utils.names.get_array_element_id_from_name(array_element_name)

Get array element ID from array element name, (e.g. “01” from “MSTN-01”).

Parameters:

array_element_name: str

Array element name

Returns:

str

Array element ID.

utils.names.get_array_element_name_from_common_identifier(common_identifier)

Get array element name from common identifier as used by CTAO.

Common identifiers are numerical IDs used by the CTAO ACADA and DPPS systems.

Parameters:

common_identifier: int

Common identifier.

Returns:

str

Array element name.

utils.names.get_array_element_type_from_name(array_element_name)

Get array element type from array element name (e.g “MSTN” from “MSTN-01”).

For sites, return site name.

Parameters:

array_element_name: str

Array element name

Returns:

str

Array element type.

utils.names.get_collection_name_from_array_element_name(array_element_name, array_elements_only=True)

Get collection name (e.g., telescopes, calibration_devices) of an array element from its name.

Parameters:

array_element_name: str

Array element name (e.g. LSTN-01)

array_elements_only: bool

If True, only array elements are considered (e.g. “OBS-North” will raise a ValueError).

Returns:

str

Collection name .

Raises:

ValueError

If name is not a valid array element name.

utils.names.get_collection_name_from_parameter_name(parameter_name)

Get the db collection name for a given parameter.

Parameters:

parameter_name: str

Name of the parameter.

Returns:

str

Collection name.

Raises:

KeyError

If the parameter name is not found in the list of model parameters

utils.names.get_common_identifier_from_array_element_name(array_element_name, default_return=None)

Get numerical common identifier from array element name as used by CTAO.

Common identifiers are numerical IDs used by the CTAO ACADA and DPPS systems.

Parameters:

array_element_name: str

Array element name (e.g. LSTN-01)

Returns:

int

Common identifier.

utils.names.get_list_of_array_element_types(array_element_class='telescopes', site=None, observatory='CTAO')

Get list of array element types (e.g., [“LSTN”, “MSTN”] for the Northern site).

Parameters:

array_element_class: str

Array element class

site: str

Site name (e.g., South or North).

Returns:

list

List of array element types.

utils.names.get_simulation_software_name_from_parameter_name(parameter_name, software_name='sim_telarray', set_meta_parameter=False)

Get the name used in the given simulation software from the model parameter name.

Name convention is expected to be defined in the model parameter schema. Returns the parameter name if no simulation software name is found.

Parameters:

parameter_name: str

Model parameter name.

simulation_software: str

Simulation software name.

set_meta_parameter: bool

If True, return values with ‘set_meta_parameter’ field set to True.

Returns:

str

Simtel parameter name.

utils.names.get_site_from_array_element_name(array_element_name)

Get site name from array element name (e.g., “South” from “MSTS-01”).

Parameters:

array_element_name: str

Array element name.

Returns:

str, list

Site name(s).

utils.names.instrument_class_key_to_db_collection(class_name)

Convert instrument class key to collection name.

utils.names.is_design_type(array_element_name)

Check if array element is a design type (e.g., “MSTS-FlashCam” or “LSTN-design”).

Parameters:

array_element_name: str

Array element name.

Returns:

bool

True if array element is a design type.

utils.names.model_parameters(class_key_list=None)

Get model parameters and their properties for a given instrument class key.

Returns all model parameters if class_key is None.

Parameters:

class_key: str, None

Class key (e.g., “telescope”, “camera”, structure”).

Returns:

dict

Model parameters definitions.

utils.names.sanitize_name(name)

Sanitize name to be a valid Python identifier.

• Replaces spaces with underscores

• Converts to lowercase

• Removes characters that are not alphanumerics or underscores

• If the name starts with a number, prepend an underscore

Parameters:

name: str

name to be sanitized.

Returns:

str:

Sanitized name.

Raises:

ValueError:

if the string name can not be sanitized.

utils.names.simtel_config_file_name(site, model_version, array_name=None, telescope_model_name=None, label=None, extra_label=None)

sim_telarray config file name for a telescope.

Parameters:

site: str

South or North.

telescope_model_name: str

LST-1, MST-FlashCam, …

model_version: str

Version of the model.

label: str

Instance label.

extra_label: str

Extra label in case of multiple telescope config files.

Returns:

str

File name.

utils.names.simtel_single_mirror_list_file_name(site, telescope_model_name, model_version, mirror_number, label)

sim_telarray mirror list file with a single mirror.

Parameters:

site: str

South or North.

telescope_model_name: str

North-LST-1, South-MST-FlashCam, …

model_version: str

Version of the model.

mirror_number: int

Mirror number.

label: str

Instance label.

Returns:

str

File name.

utils.names.simulation_software()

Get simulation software names from the meta schema definition.

Returns:

list

List of simulation software names.

utils.names.site_names()

Get site names.

The list of sites is derived from the sites listed in array element definition file. Return a dictionary for compatibility with the validation ‘_validate_name’ routine.

Returns:

dict

Site names.

utils.names.site_parameters()

Return site model parameters.

utils.names.telescope_parameters()

Return telescope model parameters.

utils.names.validate_array_element_id_name(array_element_id, array_element_type=None)

Validate array element ID.

Allowed IDs are - design types (for design array elements or testing) - array element ID (e.g., 1, 5, 15) - test (for testing)

Parameters:

name: str or int

Array element ID name.

array_element_type: str

Array element type (e.g., LSTN, MSTN).

Returns:

str

Validated array element ID (added leading zeros, e.g., 1 is converted to 01).

Raises:

ValueError

If name is not valid.

utils.names.validate_array_element_name(array_element_name)

Validate array element name (e.g., MSTx-NectarCam, MSTN-01).

Forgiving validation, is it allows also to give a site name (e.g., OBS-North).

Parameters:

array_element_name: str

Array element name.

Returns:

str

Validated name.

utils.names.validate_array_element_type(array_element_type)

Validate array element type (e.g., LSTN, MSTN).

Parameters:

array_element_type: str

Array element type.

Returns:

str

Validated name.

utils.names.validate_site_name(site_name)

Validate site name.

Parameters:

site_name: str

Site name.

Returns:

str

Validated name.

value_conversion

Value and quantity conversion.

utils.value_conversion.extract_type_of_value(value) → str

Extract the string representation of the the type of a value.

For example, for a string, it returns ‘str’ rather than ‘<class ‘str’>’. Take into account also the case where the value is a numpy type.

utils.value_conversion.get_value_as_quantity(value, unit)

Get a value as a Quantity with a given unit. If value is a Quantity, convert to the given unit.

Parameters:

value:

value to get a unit. It can be a float, int, or a Quantity (convertible to ‘unit’).

unit: astropy.units.Unit

Unit to apply to ‘quantity’.

Returns:

astropy.units.Quantity

Quantity of value ‘quantity’ and unit ‘unit’.

Raises:

u.UnitConversionError

If the value cannot be converted to the given unit.

utils.value_conversion.get_value_unit_type(value, unit_str=None)

Get the value, unit and type of a value.

The value is stripped of its unit and the unit is returned in its string form (i.e., to_string()). The type is returned as a string representation of the type. For example, for a string, it returns ‘str’ rather than ‘<class ‘str’>’. An additional unit string can be given and the return value is converted to this units.

Note that Quantities are always floats, even if the original value is represented as an int.

Parameters:

value: str, int, float, bool, u.Quantity

Value to be parsed.

unit_str: str

Unit to be used for the value.

Returns:

type of value, str, str

Value, unit in string representation (to_string())), and string representation of the type of the value.

utils.value_conversion.split_value_and_unit(value, is_integer=False)

Split a value into its value and unit.

Takes into account the case where the value is a Quantity, a number, or a simtools-type string encoding a list of values and units.

Parameters:

value: str, int, float, bool, u.Quantity

Value to be parsed.

is_integer: bool

Flag to indicate if the value is an integer.

Returns:

value, str

Value and units as (value, unit), or lists of values and unites

constants

Project wide constants.