Database support#

Modules for database access. See the databases sections for details.

db_handler#

Module to handle interaction with DB.

class db.db_handler.DatabaseHandler(mongo_db_config=None)[source]#

DatabaseHandler provides the interface to the DB.

Note the two types of version variables used in this class:

db_simulation_model_version (from mongo_db_config): version of the simulation model database
model_version (from production_tables): version of the model contained in the database

Parameters:

mongo_db_config: dict: Dictionary with the MongoDB configuration (see jsonschema_db_dict for details).

add_new_parameter(par_dict, db_name=None, collection_name='telescopes', file_prefix=None)[source]#

Add a new parameter dictionary to the DB.

A new document will be added to the DB, with all fields taken from the input parameters. Parameter dictionaries are validated before submission using the corresponding schema.

Parameters:

par_dict: dict: dictionary with parameter data
db_name: str: the name of the DB
collection_name: str: The name of the collection to add a parameter to.
file_prefix: str or Path: where to find files to upload to the DB

add_production_table(production_table, db_name=None)[source]#

Add a production table to the DB.

Parameters:

production_table: dict: The production table to add to the DB.
db_name: str: the name of the DB.

export_model_file(parameter, site, array_element_name, model_version=None, parameter_version=None, export_file_as_table=False)[source]#

Export single model file from the DB identified by the parameter name.

The parameter can be identified by model or parameter version. Files can be exported as astropy tables (ecsv format).

Parameters:

parameter: str: Name of the parameter.
site: str: Site name.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
parameter_version: str: Version of the parameter.
model_version: str: Version of the model.
export_file_as_table: bool: If True, export the file as an astropy table (ecsv format).

Returns:

astropy.table.Table or None: If export_file_as_table is True

export_model_files(parameters=None, file_names=None, dest=None, db_name=None)[source]#

Export models files from the DB to given directory.

The files to be exported can be specified by file_name or retrieved from the database using the parameters dictionary.

Parameters:

parameters: dict: Dict of model parameters
file_names: list, str: List (or string) of file names to export
dest: str or Path: Location where to write the files to.

Returns:

file_id: dict of GridOut._id: Dict of database IDs of files.

generate_compound_indexes(db_name=None)[source]#

Generate compound indexes for the MongoDB collections.

Indexes based on the typical query patterns.

generate_compound_indexes_for_databases(db_name, db_simulation_model, db_simulation_model_version)[source]#

Generate compound indexes for several databases.

Parameters:

db_name: str: Name of the database.
db_simulation_model: str: Name of the simulation model.
db_simulation_model_version: str: Version of the simulation model.

Raises:

ValueError: If the requested database is not found.

get_array_elements(model_version, collection='telescopes')[source]#

Get list array elements for a given model version and collection from the DB.

Parameters:

model_version: str: Version of the model.
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices.

Returns:

list: Sorted list of all array elements found in collection

get_array_elements_of_type(array_element_type, model_version, collection)[source]#

Get array elements of a certain type (e.g. ‘LSTN’) for a DB collection.

Does not return ‘design’ models.

Parameters:

array_element_type: str: Type of the array element (e.g. LSTN, MSTS).
model_version: str: Version of the model.
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices.

Returns:

list: Sorted list of all array element names found in collection

get_collection(collection_name, db_name=None)[source]#

Get a collection from the DB.

Parameters:

collection_name: str: Name of the collection.
db_name: str: Name of the DB.

Returns:

pymongo.collection.Collection: The collection from the DB.

get_collections(db_name=None, model_collections_only=False)[source]#

List of collections in the DB.

Parameters:

db_name: str: Database name.
model_collections_only: bool: If True, only return model collections (i.e. exclude fs.files, fs.chunks)

Returns:

list: List of collection names

get_db_name(db_name=None, db_simulation_model_version=None, model_name=None)[source]#: Build DB name from configuration.

get_design_model(model_version, array_element_name, collection='telescopes')[source]#

Get the design model used for a given array element and a given model version.

Parameters:

model_version: str: Version of the model.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices.

Returns:

str: Design model for a given array element.

get_ecsv_file_as_astropy_table(file_name, db_name=None)[source]#

Read contents of an ECSV file from the database and return it as an Astropy Table.

Files are not written to disk.

Parameters:

file_name: str: The name of the ECSV file.
db_name: str: The name of the database.

Returns:

astropy.table.Table: The contents of the ECSV file as an Astropy Table.

get_model_parameter(parameter, site, array_element_name, parameter_version=None, model_version=None)[source]#

Get a single model parameter using model or parameter version.

Note that this function should not be called in a loop for many parameters, as it each call queries the database.

Parameters:

parameter: str: Name of the parameter.
site: str: Site name.
array_element_name: str: Name of the array element model.
parameter_version: str: Version of the parameter.
model_version: str: Version of the model.

Returns:

dict containing the parameter

get_model_parameters(site, array_element_name, collection, model_version)[source]#

Get model parameters using the model version.

Queries parameters for design and for the specified array element (if necessary).

Parameters:

site: str: Site name.
array_element_name: str: Name of the array element model (e.g. LSTN-01, MSTx-FlashCam, ILLN-01).
model_version: str, list: Version(s) of the model.
collection: str: Collection of array element (e.g. telescopes, calibration_devices).

Returns:

dict containing the parameters

get_model_parameters_for_all_model_versions(site, array_element_name, collection)[source]#

Get model parameters for all model versions.

Parameters:

site: str: Site name.
array_element_name: str: Name of the array element model (e.g. LSTN-01, MSTx-FlashCam, ILLN-01).
collection: str: Collection of array element (e.g. telescopes, calibration_devices).

Returns:

dict containing the parameters with model version as first key

get_model_versions(collection_name='telescopes')[source]#

Get list of model versions from the DB with caching.

Parameters:

collection_name: str: Name of the collection.

Returns:

list: List of model versions

get_simulation_configuration_parameters(simulation_software, site, array_element_name, model_version)[source]#

Get simulation configuration parameters from the DB.

Parameters:

simulation_software: str: Name of the simulation software.
site: str: Site name.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
model_version: str: Version of the model.

Returns:

dict containing the parameters

Raises:

ValueError: if simulation_software is not valid.

insert_file_to_db(file_name, db_name=None, **kwargs)[source]#

Insert a file to the DB.

Parameters:

file_name: str or Path: The name of the file to insert (full path).
db_name: str: the name of the DB
**kwargs (optional): keyword arguments for file creation.: The full list of arguments can be found in, https://www.mongodb.com/docs/manual/core/gridfs/ mostly these are unnecessary though.

Returns:

file_iD: GridOut._id: If the file exists, return its GridOut._id, otherwise insert the file and return its” “newly created DB GridOut._id.

is_remote_database()[source]#

Check if the database is remote.

Check for domain pattern like “cta-simpipe-protodb.zeuthen.desy.de”

Returns:

bool: True if the database is remote, False otherwise.

print_connection_info()[source]#: Print the connection information.

read_production_table_from_mongo_db(collection_name, model_version)[source]#

Read production table for the given collection from MongoDB.

Parameters:

collection_name: str: Name of the collection.
model_version: str: Version of the model.

Raises:

ValueError: if query returned no results.

db_model_upload#

Upload a simulation model (parameters and production tables) to the database.

db.db_model_upload.add_complete_model(tmp_dir, db, db_simulation_model, db_simulation_model_version, repository_url, repository_branch=None)[source]#

Upload a complete model including model parameters and production tables to the database.

Generate compound indexes for the new database.

Parameters:

tmp_dirPath or str: Temporary directory to use for cloning the repository.
dbDatabaseHandler: Database handler object.
db_simulation_modelstr: Name of the simulation model in the database.
db_simulation_model_versionstr: Version of the simulation model in the database.
repository_urlstr: URL of the simulation model repository to clone.
repository_branchstr, optional: Branch of the repository to use. If None, the default branch is used.

Returns:

None: This function does not return a value.

db.db_model_upload.add_model_parameters_to_db(input_path, db)[source]#

Read model parameters from a directory and upload them to the database.

Parameters:

input_pathPath, str: Path to the directory containing the model parameters.
dbDatabaseHandler: Database handler object.

db.db_model_upload.add_production_tables_to_db(input_path, db)[source]#

Read production tables from a directory and upload them to the database.

One dictionary per collection is prepared for each model version, containing tables of all array elements, sites, and configuration parameters.

Parameters:

input_pathPath, str: Path to the directory containing the production tables.
dbDatabaseHandler: Database handler object.

db.db_model_upload.add_values_from_json_to_db(file, collection, db, file_prefix)[source]#

Upload new model parameter from json files to db.

Parameters:

filelist: JSON file to be uploaded to the DB.
collectionstr: The DB collection to which to add the file.
dbDatabaseHandler: Database handler object.
file_prefixstr: Path to location of all additional files to be uploaded.

db.db_model_upload.clone_simulation_model_repository(target_dir, repository_url, db_simulation_model_version, repository_branch)[source]#

Clone a git repository containing simulation model parameters and production tables.

Parameters:

target_dirPath or str: Target directory to clone the repository into.
repository_urlstr: URL of the git repository to clone.
db_simulation_model_versionstr: Version tag of the simulation model to checkout.
repository_branchstr, optional: Branch of the repository to use. If None, the version tag is used.

Returns:

Path: Path to the cloned repository.

Database support#

db_handler#

db_model_upload#

This Page