Utilities#
In this section you find the reference documentation of the modules contained in the util module.
general#
General functions useful across different parts of the code.
- utils.general.change_dict_keys_case(data_dict, lower_case=True)[source]#
Change keys of a dictionary to lower or upper case recursively.
- Parameters:
- data_dict: dict
Dictionary to be converted.
- lower_case: bool
Change keys to lower (upper) case if True (False).
- Returns:
- dict
Dictionary with keys converted to lower or upper case.
- utils.general.clear_default_sim_telarray_cfg_directories(command)[source]#
Prefix the command to clear default sim_telarray configuration directories.
- Parameters:
- command: str
Command to be prefixed.
- Returns:
- str
Prefixed command.
- utils.general.collect_final_lines(file, n_lines)[source]#
Collect final lines.
- Parameters:
- file: str or Path
File to collect the lines from.
- n_lines: int
Number of lines to be collected.
- Returns:
- str
Final lines collected.
- utils.general.collect_kwargs(label, in_kwargs)[source]#
Collect kwargs of the type label_* and return them as a dict.
- Parameters:
- label: str
Label to be collected in kwargs.
- in_kwargs: dict
kwargs.
- Returns:
- dict
Dictionary with the collected kwargs.
- utils.general.convert_keys_in_dict_to_lowercase(data)[source]#
Recursively convert all dictionary keys to lowercase.
- Parameters:
- data: dict
Dictionary to be converted.
- Returns:
- dict
Dictionary with all keys converted to lowercase.
- utils.general.convert_list_to_string(data, comma_separated=False, shorten_list=False, collapse_list=False)[source]#
Convert arrays to string (if required).
- Parameters:
- data: object
Object of data to convert (e.g., double or list)
- comma_separated: bool
If True, returns elements as a comma-separated string (default is space-separated).
- shorten_list: bool
If True and all elements in the list are identical, returns a summary string like “all: value”. This is useful to make the configuration files more readable.
- collapse_list: bool
If True and all elements in the list are identical, returns a single value instead of the entire list.
- Returns:
- object or str:
Converted data as string (if required)
- utils.general.convert_string_to_list(data_string, is_float=True, force_comma_separation=False)[source]#
Convert string (as used e.g. in sim_telarray) to list.
Allow coma or space separated strings.
- Parameters:
- data_string: object
String to be converted
- is_float: bool
If True, convert to float, otherwise to int.
- force_comma_separation: bool
If True, force comma separation.
- Returns:
- list, str
Converted data from string (if required). Return data_string if conversion fails.
- utils.general.ensure_iterable(value)[source]#
Return input value as iterable.
Single values will return as a list with a single element.
None values will return as empty list.
Values of list or tuple type are not changed.
- Parameters:
- value: any
Input value to be converted to a iterable.
- Returns:
- list or tuple
Converted value as list or tuple.
- utils.general.find_differences_in_json_objects(obj1, obj2, path='')[source]#
Recursively find differences between two JSON-like objects.
- Parameters:
- obj1: dict, list, or any
First object to compare.
- obj2: dict, list, or any
Second object to compare.
- path: str
Path to the current object in the JSON structure, used for reporting differences.
- Returns:
- list
List of differences found between the two objects, with paths indicating where the differences occur.
- utils.general.find_file(name, loc)[source]#
Search for files inside of given directories, recursively, and return its full path.
- Parameters:
- name: str
File name to be searched for.
- loc: Path or list of Path
Location of where to search for the file.
- Returns:
- Path
Full path of the file to be found if existing. Otherwise, None.
- Raises:
- FileNotFoundError
If the desired file is not found.
- utils.general.get_file_age(file_path)[source]#
Get the age of a file in seconds since the last modification.
- utils.general.get_list_of_files_from_command_line(file_names, suffix_list)[source]#
Get a list of files from the command line.
Files can be given as a list of file names or as a text file containing the list of files. The list of suffixes restrict the files types to be returned. Note that a file list must have a different suffix than those in the suffix list.
- Parameters:
- file_names: list
List of file names to be checked.
- suffix_list: list
List of suffixes to be checked.
- Returns:
- list
List of files with the given suffixes.
- utils.general.get_log_excerpt(log_file, n_last_lines=30)[source]#
Get an excerpt from a log file, namely the n_last_lines of the file.
- Parameters:
- log_file: str or Path
Log file to get the excerpt from.
- n_last_lines: int
Number of last lines of the file to get.
- Returns:
- str
Excerpt from log file with header/footer
- utils.general.get_log_level_from_user(log_level)[source]#
Map between logging level from the user to logging levels of the logging module.
- Parameters:
- log_level: str
Log level from the user.
- Returns:
- logging.LEVEL
The requested logging level to be used as input to logging.setLevel().
- utils.general.get_structure_array_from_table(table, column_names)[source]#
Get a structured array from an astropy table for a selected list of columns.
- Parameters:
- table: astropy.table.Table
Table to be converted.
- column_names: list
List of column names to be included in the structured array.
- Returns:
- numpy.ndarray
Structured array containing the table data.
- utils.general.is_url(url)[source]#
Check if a string is a valid URL.
- Parameters:
- url: str
String to be checked.
- Returns:
- bool
True if url is a valid URL.
- utils.general.now_date_time_in_isoformat()[source]#
Return date and time in isoformat and second accuracy.
- utils.general.pack_tar_file(tar_file_name, file_list, sub_dir=None)[source]#
Pack files into a tar.gz archive.
- Parameters:
- tar_file_name: str
Name of the output tar.gz file.
- file_list: list
List of files to include in the archive.
- sub_dir: str, optional
Subdirectory within the archive to place the files.
- utils.general.remove_key_from_dict(data, key_to_remove)[source]#
Remove a specific key from a dictionary recursively.
- Parameters:
- data: dict
Dictionary to be processed.
- key_to_remove: str
Key to be removed from the dictionary.
- Returns:
- dict
Dictionary with the specified key removed.
- utils.general.remove_substring_recursively_from_dict(data_dict, substring='\n')[source]#
Remove substrings from all strings in a dictionary.
Recursively crawls through the dictionary This e.g., allows to remove all newline characters from a dictionary.
- Parameters:
- data_dict: dict
Dictionary to be converted.
- substring: str
Substring to be removed.
- Raises:
- AttributeError:
if input is not a proper dictionary.
- utils.general.resolve_file_patterns(file_names)[source]#
Return a list of files names from string, list, or wildcard pattern.
- Parameters:
- file_names: str, list
File names to be searched for (wildcards allowed).
- Returns:
- list
List of file names found.
- utils.general.set_default_kwargs(in_kwargs, **kwargs)[source]#
Fill in a dict with a set of default kwargs and return it.
- Parameters:
- in_kwargs: dict
Input dict to be filled in with the default kwargs.
- **kwargs:
Default kwargs to be set.
- Returns:
- dict
Dictionary containing the default kwargs.
- utils.general.url_exists(url)[source]#
Check if a URL exists.
- Parameters:
- url: str
URL to be checked.
- Returns:
- bool
True if URL exists.
- utils.general.user_confirm()[source]#
Ask the user to enter y or n (case-insensitive) on the command line.
- Returns:
- bool:
True if the answer is Y/y.
- utils.general.validate_data_type(reference_dtype, value=None, dtype=None, allow_subtypes=True)[source]#
Validate data type of value or type object against a reference data type.
Allow to check for exact data type or allow subtypes (e.g. uint is accepted for int). Take into account ‘file’ type as used in the model parameter database.
- Parameters:
- reference_dtype: str
Reference data type to be checked against.
- value: any, optional
Value to be checked (if dtype is None).
- dtype: type, optional
Type object to be checked (if value is None).
- allow_subtypes: bool, optional
If True, allow subtypes to be accepted.
- Returns:
- bool:
True if the data type is valid.
geometry#
A collection of functions related to geometrical transformations.
- utils.geometry.calculate_circular_mean(angles)[source]#
Calculate circular mean of angles in radians.
- Parameters:
- angles: numpy.array
Array of angles in radians.
- Returns:
- float
Circular mean of the angles.
- utils.geometry.convert_2d_to_radial_distr(hist_2d, xaxis, yaxis, bins=50, max_dist=1000)[source]#
Convert a 2d histogram of positions, e.g. photon positions on the ground, to a 1D distribution.
- Parameters:
- hist_2d: numpy.ndarray
The histogram counts.
- xaxis: numpy.array
The values of the x axis (histogram bin edges) on the ground.
- yaxis: numpy.array
The values of the y axis (histogram bin edges) on the ground.
- bins: float
Number of bins in distance.
- max_dist: float
Maximum distance to consider in the 1D histogram, usually in meters.
- Returns:
- np.array
The values of the 1D histogram with size = int(max_dist/bin_size).
- np.array
The bin edges of the 1D histogram with size = int(max_dist/bin_size) + 1.
- utils.geometry.fiducial_radius_from_shape(width, shape)[source]#
Calculate minimum radius including different geometrical shapes.
Assumes definition of shapes as in ‘camera_body_shape’ model parameter:
circle: shape = 0, width is diameter
hexagon: shape = 1 or 3, width is flat-to-flat distance
square: shape = 2, width is side length
- Parameters:
- widthfloat
Characteristic width
- shapeint
Geometrical shape parameter
- Returns:
- float
Minimum fiducial radius
- utils.geometry.rotate(x, y, rotation_around_z_axis, rotation_around_y_axis=0)[source]#
Rotate the x and y coordinates of the telescopes.
The two rotations are:
rotation_angle_around_z_axis gives the rotation on the observation plane (x, y)
rotation_angle_around_y_axis allows to rotate the observation plane in space.
The function returns the rotated x and y values in the same unit given. The direction of rotation of the elements in the plane is counterclockwise, i.e., the rotation of the coordinate system is clockwise.
- Parameters:
- x: numpy.array or list
x positions of the entries (e.g. telescopes), usually in meters.
- y: numpy.array or list
y positions of the entries (e.g. telescopes), usually in meters.
- rotation_angle_around_z_axis: astropy.units.rad
Angle to rotate the array in the observation plane (around z axis) in radians.
- rotation_angle_around_y_axis: astropy.units.rad
Angle to rotate the observation plane around the y axis in radians.
- Returns:
- 2-tuple of list
x and y positions of the rotated entry (e.g. telescopes) positions.
- Raises:
- TypeError:
If type of x and y parameters are not valid.
- RuntimeError:
If the length of x and y are different.
- UnitsError:
If the unit of x and y are different.
- utils.geometry.solid_angle(angle_max, angle_min=<Quantity 0. rad>)[source]#
Calculate the solid angle subtended by a given range of angles.
- Parameters:
- angle_max: astropy.units.Quantity
The maximum angle for which to calculate the solid angle.
- angle_min: astropy.units.Quantity
The minimum angle for which to calculate the solid angle (default is 0 rad).
- Returns:
- astropy.units.Quantity
The solid angle subtended by the given range of angles (in steradians).
- utils.geometry.transform_ground_to_shower_coordinates(x_ground, y_ground, z_ground, azimuth, altitude)[source]#
Transform ground to shower coordinates.
Assume ground to be of type ‘North-West-Up’ (NWU) coordinates.
- Parameters:
- x_ground: numpy.array
Ground x coordinate.
- y_ground: numpy.array
Ground y coordinate.
- z_ground: numpy.array
Ground z coordinate.
- azimuth: numpy.array
Azimuth angle of the shower (in radians).
- altitude: numpy.array
Altitude angle of the shower (in radians).
- Returns:
- tuple
Transformed shower coordinates (x’, y’, z’).
names#
Name utilities for array elements, sites, and model parameters.
Naming in simtools:
‘site’: South or North
‘array element’: e.g., LSTN-01, MSTN-01, …
‘array element type’: e.g., LSTN, MSTN, …
‘array element ID’: e.g., 01, 02, …
‘array element design type’: e.g., design, test
‘instrument class key’: e.g., telescope, camera, structure
‘db collection’: e.g., telescopes, sites, calibration_devices
- utils.names.array_element_common_identifiers()[source]#
Get array element IDs from CTAO common identifier.
- Returns:
- dict, dict
Dictionary mapping array element names to their IDs and vice versa.
- utils.names.array_element_design_types(array_element_type)[source]#
Get array element design type (e.g., ‘design’ or ‘flashcam’).
Default values are [‘design’, ‘test’].
- Parameters:
- array_element_type
Array element type
- Returns:
- list
Array element design types.
- utils.names.array_elements()[source]#
Get array elements and their properties.
- Returns:
- dict
Array elements.
- utils.names.db_collection_to_instrument_class_key(collection_name='telescopes')[source]#
Return list of instrument classes for a given collection.
- utils.names.file_name_with_version(file_name, suffix)[source]#
Return a file name including a semantic version with the correct suffix.
Replaces ‘Path.suffix()’, which removes trailing numbers (and therefore version numbers).
- Parameters:
- file_name: str
File name.
- suffix: str
File suffix.
- Returns:
- Path
File name with version number.
- utils.names.generate_array_element_name_from_type_site_id(array_element_type, site, array_element_id)[source]#
Generate a new array element name from array element type, site, and array element ID.
- Parameters:
- array_element_type: str
Array element type.
- site: str
Site name.
- array_element_id: str
Array element ID.
- Returns:
- str
Array element name.
- utils.names.generate_file_name(file_type, suffix, site, telescope_model_name, zenith_angle, azimuth_angle=None, off_axis_angle=None, source_distance=None, mirror_number=None, label=None, extra_label=None)[source]#
Generate a file name for output, config, or plotting.
Used e.g., to generate camera_efficiency and ray_tracing output files.
- Parameters:
- file_type: str
Type of file (e.g., config, output, plot)
- suffix: str
File suffix
- site: str
South or North.
- telescope_model_name: str
LSTN-01, MSTS-01, …
- zenith_angle: float
Zenith angle (deg).
- azimuth_angle: float
Azimuth angle (deg).
- off_axis_angle: float
Off-axis angle (deg).
- source_distance: float
Source distance (km).
- mirror_number: int
Mirror number.
- label: str
Instance label.
- extra_label: str
Extra label.
- Returns:
- str
File name.
- utils.names.get_array_element_id_from_name(array_element_name)[source]#
Get array element ID from array element name, (e.g. “01” from “MSTN-01”).
- Parameters:
- array_element_name: str
Array element name
- Returns:
- str
Array element ID.
- utils.names.get_array_element_name_from_common_identifier(common_identifier)[source]#
Get array element name from common identifier as used by CTAO.
Common identifiers are numerical IDs used by the CTAO ACADA and DPPS systems.
- Parameters:
- common_identifier: int
Common identifier.
- Returns:
- str
Array element name.
- utils.names.get_array_element_type_from_name(array_element_name)[source]#
Get array element type from array element name (e.g “MSTN” from “MSTN-01”).
For sites, return site name.
- Parameters:
- array_element_name: str
Array element name
- Returns:
- str
Array element type.
- utils.names.get_collection_name_from_array_element_name(array_element_name, array_elements_only=True)[source]#
Get collection name (e.g., telescopes, calibration_devices) of an array element from its name.
- Parameters:
- array_element_name: str
Array element name (e.g. LSTN-01)
- array_elements_only: bool
If True, only array elements are considered (e.g. “OBS-North” will raise a ValueError).
- Returns:
- str
Collection name .
- Raises:
- ValueError
If name is not a valid array element name.
- utils.names.get_collection_name_from_parameter_name(parameter_name)[source]#
Get the db collection name for a given parameter.
- Parameters:
- parameter_name: str
Name of the parameter.
- Returns:
- str
Collection name.
- Raises:
- KeyError
If the parameter name is not found in the list of model parameters
- utils.names.get_common_identifier_from_array_element_name(array_element_name, default_return=None)[source]#
Get numerical common identifier from array element name as used by CTAO.
Common identifiers are numerical IDs used by the CTAO ACADA and DPPS systems.
- Parameters:
- array_element_name: str
Array element name (e.g. LSTN-01)
- Returns:
- int
Common identifier.
- utils.names.get_list_of_array_element_types(array_element_class='telescopes', site=None, observatory='CTAO')[source]#
Get list of array element types (e.g., [“LSTN”, “MSTN”] for the Northern site).
- Parameters:
- array_element_class: str
Array element class
- site: str
Site name (e.g., South or North).
- Returns:
- list
List of array element types.
- utils.names.get_simulation_software_name_from_parameter_name(parameter_name, software_name='sim_telarray', set_meta_parameter=False)[source]#
Get the name used in the given simulation software from the model parameter name.
Name convention is expected to be defined in the model parameter schema. Returns the parameter name if no simulation software name is found.
- Parameters:
- parameter_name: str
Model parameter name.
- simulation_software: str
Simulation software name.
- set_meta_parameter: bool
If True, return values with ‘set_meta_parameter’ field set to True.
- Returns:
- str
Simtel parameter name.
- utils.names.get_site_from_array_element_name(array_element_name)[source]#
Get site name from array element name (e.g., “South” from “MSTS-01”).
- Parameters:
- array_element_name: str
Array element name.
- Returns:
- str, list
Site name(s).
- utils.names.instrument_class_key_to_db_collection(class_name)[source]#
Convert instrument class key to collection name.
- utils.names.is_design_type(array_element_name)[source]#
Check if array element is a design type (e.g., “MSTS-FlashCam” or “LSTN-design”).
- Parameters:
- array_element_name: str
Array element name.
- Returns:
- bool
True if array element is a design type.
- utils.names.model_parameters(class_key_list=None)[source]#
Get model parameters and their properties for a given instrument class key.
Returns all model parameters if class_key is None.
- Parameters:
- class_key: str, None
Class key (e.g., “telescope”, “camera”, structure”).
- Returns:
- dict
Model parameters definitions.
- utils.names.sanitize_name(name)[source]#
Sanitize name to be a valid Python identifier.
Replaces spaces with underscores
Converts to lowercase
Removes characters that are not alphanumerics or underscores
If the name starts with a number, prepend an underscore
- Parameters:
- name: str
name to be sanitized.
- Returns:
- str:
Sanitized name.
- Raises:
- ValueError:
if the string name can not be sanitized.
- utils.names.simtel_config_file_name(site, model_version, array_name=None, telescope_model_name=None, label=None, extra_label=None)[source]#
sim_telarray config file name for a telescope.
- Parameters:
- site: str
South or North.
- telescope_model_name: str
LST-1, MST-FlashCam, …
- model_version: str
Version of the model.
- label: str
Instance label.
- extra_label: str
Extra label in case of multiple telescope config files.
- Returns:
- str
File name.
- utils.names.simtel_single_mirror_list_file_name(site, telescope_model_name, model_version, mirror_number, label)[source]#
sim_telarray mirror list file with a single mirror.
- Parameters:
- site: str
South or North.
- telescope_model_name: str
North-LST-1, South-MST-FlashCam, …
- model_version: str
Version of the model.
- mirror_number: int
Mirror number.
- label: str
Instance label.
- Returns:
- str
File name.
- utils.names.simulation_software()[source]#
Get simulation software names from the meta schema definition.
- Returns:
- list
List of simulation software names.
- utils.names.site_names()[source]#
Get site names.
The list of sites is derived from the sites listed in array element definition file. Return a dictionary for compatibility with the validation ‘_validate_name’ routine.
- Returns:
- dict
Site names.
- utils.names.validate_array_element_id_name(array_element_id, array_element_type=None)[source]#
Validate array element ID.
Allowed IDs are - design types (for design array elements or testing) - array element ID (e.g., 1, 5, 15) - test (for testing)
- Parameters:
- name: str or int
Array element ID name.
- array_element_type: str
Array element type (e.g., LSTN, MSTN).
- Returns:
- str
Validated array element ID (added leading zeros, e.g., 1 is converted to 01).
- Raises:
- ValueError
If name is not valid.
- utils.names.validate_array_element_name(array_element_name)[source]#
Validate array element name (e.g., MSTx-NectarCam, MSTN-01).
Forgiving validation, is it allows also to give a site name (e.g., OBS-North).
- Parameters:
- array_element_name: str
Array element name.
- Returns:
- str
Validated name.
value_conversion#
Value and quantity conversion.
- utils.value_conversion.extract_type_of_value(value) str [source]#
Extract the string representation of the the type of a value.
For example, for a string, it returns ‘str’ rather than ‘<class ‘str’>’. Take into account also the case where the value is a numpy type.
- utils.value_conversion.get_value_as_quantity(value, unit)[source]#
Get a value as a Quantity with a given unit. If value is a Quantity, convert to the given unit.
- Parameters:
- value:
value to get a unit. It can be a float, int, or a Quantity (convertible to ‘unit’).
- unit: astropy.units.Unit
Unit to apply to ‘quantity’.
- Returns:
- astropy.units.Quantity
Quantity of value ‘quantity’ and unit ‘unit’.
- Raises:
- u.UnitConversionError
If the value cannot be converted to the given unit.
- utils.value_conversion.get_value_unit_type(value, unit_str=None)[source]#
Get the value, unit and type of a value.
The value is stripped of its unit and the unit is returned in its string form (i.e., to_string()). The type is returned as a string representation of the type. For example, for a string, it returns ‘str’ rather than ‘<class ‘str’>’. An additional unit string can be given and the return value is converted to this units.
Note that Quantities are always floats, even if the original value is represented as an int.
- Parameters:
- value: str, int, float, bool, u.Quantity
Value to be parsed.
- unit_str: str
Unit to be used for the value.
- Returns:
- type of value, str, str
Value, unit in string representation (to_string())), and string representation of the type of the value.
- utils.value_conversion.split_value_and_unit(value, is_integer=False)[source]#
Split a value into its value and unit.
Takes into account the case where the value is a Quantity, a number, or a simtools-type string encoding a list of values and units.
- Parameters:
- value: str, int, float, bool, u.Quantity
Value to be parsed.
- is_integer: bool
Flag to indicate if the value is an integer.
- Returns:
- value, str
Value and units as (value, unit), or lists of values and unites
constants#
Project wide constants.