Database support#

Modules for database access. See the databases sections for details.

db_handler#

Module to handle interaction with DB.

class db.db_handler.DatabaseHandler(db_config=None)[source]#

DatabaseHandler provides the interface to the DB.

Note the two types of version variables used in this class:

db_simulation_model_version (from db_config): version of the simulation model database
model_version (from production_tables): version of the model contained in the database

Parameters:

db_config: dict: Dictionary with the DB configuration.

add_new_parameter(par_dict, db_name=None, collection_name='telescopes', file_prefix=None)[source]#

Add a new parameter dictionary to the DB.

A new document will be added to the DB, with all fields taken from the input parameters. Parameter dictionaries are validated before submission using the corresponding schema.

Parameters:

par_dict: dict: dictionary with parameter data
db_name: str: the name of the DB
collection_name: str: The name of the collection to add a parameter to.
file_prefix: str or Path: where to find files to upload to the DB

add_production_table(production_table, db_name=None)[source]#

Add a production table to the DB.

Parameters:

production_table: dict: The production table to add to the DB.
db_name: str: the name of the DB.

export_model_file(parameter, site, array_element_name, model_version=None, parameter_version=None, export_file_as_table=False)[source]#

Export single model file from the DB identified by the parameter name.

The parameter can be identified by model or parameter version. Files can be exported as astropy tables (ecsv format).

Parameters:

parameter: str: Name of the parameter.
site: str: Site name.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
parameter_version: str: Version of the parameter.
model_version: str: Version of the model.
export_file_as_table: bool: If True, export the file as an astropy table (ecsv format).

Returns:

astropy.table.Table or None: If export_file_as_table is True

export_model_files(parameters=None, file_names=None, dest=None, db_name=None)[source]#

Export models files from the DB to given directory.

The files to be exported can be specified by file_name or retrieved from the database using the parameters dictionary.

Parameters:

parameters: dict: Dict of model parameters
file_names: list, str: List (or string) of file names to export
dest: str or Path: Location where to write the files to.

Returns:

file_id: dict of GridOut._id: Dict of database IDs of files.

generate_compound_indexes_for_databases(db_name, db_simulation_model, db_simulation_model_version)[source]#

Generate compound indexes for several databases.

Parameters:

db_name: str: Name of the database.
db_simulation_model: str: Name of the simulation model.
db_simulation_model_version: str: Version of the simulation model.

get_array_elements(model_version, collection='telescopes')[source]#

Get list array elements for a given model version and collection from the DB.

Parameters:

model_version: str: Version of the model.
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices.

Returns:

list: Sorted list of all array elements found in collection

get_array_elements_of_type(array_element_type, model_version, collection)[source]#

Get array elements of a certain type (e.g. ‘LSTN’) for a DB collection.

Does not return ‘design’ models.

Parameters:

array_element_type: str: Type of the array element (e.g. LSTN, MSTS).
model_version: str: Version of the model.
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices.

Returns:

list: Sorted list of all array element names found in collection

get_collection(collection_name, db_name=None)[source]#

Get a collection from the DB.

Parameters:

collection_name: str: Name of the collection.
db_name: str: Name of the DB.

Returns:

pymongo.collection.Collection: The collection from the DB.

get_collections(db_name=None, model_collections_only=False)[source]#

List of collections in the DB.

Parameters:

db_name: str: Database name.
model_collections_only: bool: If True, only return model collections (i.e. exclude fs.files, fs.chunks)

Returns:

list: List of collection names

get_db_name(db_name=None, db_simulation_model_version=None, model_name=None)[source]#: Build DB name from configuration.

get_design_model(model_version, array_element_name, collection='telescopes')[source]#

Get the design model used for a given array element and a given model version.

Parameters:

model_version: str: Version of the model.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices.

Returns:

str: Design model for a given array element.

get_ecsv_file_as_astropy_table(file_name, db_name=None)[source]#

Read contents of an ECSV file from the database and return it as an Astropy Table.

Files are not written to disk.

Parameters:

file_name: str: The name of the ECSV file.
db_name: str: The name of the database.

Returns:

astropy.table.Table: The contents of the ECSV file as an Astropy Table.

get_model_parameter(parameter, site, array_element_name, parameter_version=None, model_version=None)[source]#

Get a single model parameter using model or parameter version.

Note that this function should not be called in a loop for many parameters, as it each call queries the database.

Parameters:

parameter: str: Name of the parameter.
site: str: Site name.
array_element_name: str: Name of the array element model.
parameter_version: str: Version of the parameter.
model_version: str: Version of the model.

Returns:

dict containing the parameter

get_model_parameters(site, array_element_name, collection, model_version)[source]#

Get model parameters using the model version.

Queries parameters for design and for the specified array element (if necessary).

Parameters:

site: str: Site name.
array_element_name: str: Name of the array element model (e.g. LSTN-01, MSTx-FlashCam, ILLN-01).
model_version: str, list: Version(s) of the model.
collection: str: Collection of array element (e.g. telescopes, calibration_devices).

Returns:

dict containing the parameters

get_model_parameters_for_all_model_versions(site, array_element_name, collection)[source]#

Get model parameters for all model versions.

Parameters:

site: str: Site name.
array_element_name: str: Name of the array element model (e.g. LSTN-01, MSTx-FlashCam, ILLN-01).
collection: str: Collection of array element (e.g. telescopes, calibration_devices).

Returns:

dict containing the parameters with model version as first key

get_model_versions(collection_name='telescopes')[source]#

Get list of model versions from the DB with caching.

Parameters:

collection_name: str: Name of the collection.

Returns:

list: List of model versions

get_simulation_configuration_parameters(simulation_software, site, array_element_name, model_version)[source]#

Get simulation configuration parameters from the DB.

Parameters:

simulation_software: str: Name of the simulation software.
site: str: Site name.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
model_version: str: Version of the model.

Returns:

dict containing the parameters

Raises:

ValueError: if simulation_software is not valid.

insert_file_to_db(file_name, db_name=None)[source]#

Insert a file to the DB.

Parameters:

file_name: str or Path: The name of the file to insert (full path).
db_name: str: the name of the DB

Returns:

file_id: GridOut._id: If the file exists, return its GridOut._id, otherwise insert the file and return its newly created DB GridOut._id.

is_remote_database()[source]#

Check if the database is remote.

Check for domain pattern like “cta-simpipe-protodb.zeuthen.desy.de”

Returns:

bool: True if the database is remote, False otherwise.

print_connection_info()[source]#: Print the connection information.

read_production_table_from_db(collection_name, model_version)[source]#

Read production table for the given collection from MongoDB.

Parameters:

collection_name: str: Name of the collection.
model_version: str: Version of the model.

Raises:

ValueError: if query returned no results.

mongo_db#

MongoDB database handler for direct database operations.

class db.mongo_db.IdleConnectionMonitor[source]#

A listener to track MongoDB connection pool activity.

Used to monitor idle connections and log connection events. Switched on in debug mode.

connection_check_out_failed(event)[source]#: Handle connection check out failure event.

connection_check_out_started(event)[source]#: Handle connection check out started event.

connection_checked_in(event)[source]#: Handle connection checked in event.

connection_checked_out(event)[source]#: Handle connection checked out event.

connection_closed(event)[source]#: Handle connection closure event.

connection_created(event)[source]#: Handle connection creation event.

connection_ready(event)[source]#: Handle connection ready event.

pool_cleared(event)[source]#: Handle connection pool cleared event.

pool_closed(event)[source]#: Handle connection pool closure event.

pool_created(event)[source]#: Handle connection pool creation event.

pool_ready(event)[source]#: Handle connection pool ready event.

class db.mongo_db.MongoDBHandler(db_config=None)[source]#

MongoDBHandler provides low-level interface to MongoDB operations.

This class handles direct MongoDB operations including connection management, database queries, file operations via GridFS, and index generation.

Parameters:

db_config: dict: Dictionary with the MongoDB configuration (see jsonschema_db_dict for details).

find_one(query, collection_name, db_name)[source]#

Query MongoDB and return first result.

Parameters:

query: dict: Query to execute.
collection_name: str: Collection name.
db_name: str: Database name.

Returns:

dict or None: First document matching the query or None.

generate_compound_indexes(db_name)[source]#

Generate compound indexes for the MongoDB collections.

Indexes based on the typical query patterns.

Parameters:

db_name: str: Name of the database.

generate_compound_indexes_for_databases(db_name, db_simulation_model, db_simulation_model_version)[source]#

Generate compound indexes for several databases.

Parameters:

db_name: str: Name of the database.
db_simulation_model: str: Name of the simulation model.
db_simulation_model_version: str: Version of the simulation model.

Raises:

ValueError: If the requested database is not found.

get_collection(collection_name, db_name)[source]#

Get a collection from the DB.

Parameters:

collection_name: str: Name of the collection.
db_name: str: Name of the DB.

Returns:

pymongo.collection.Collection: The collection from the DB.

get_collections(db_name, model_collections_only=False)[source]#

List of collections in the DB.

Parameters:

db_name: str: Database name.
model_collections_only: bool: If True, only return model collections (i.e. exclude fs.files, fs.chunks)

Returns:

list: List of collection names

static get_db_name(db_name=None, db_simulation_model_version=None, model_name=None)[source]#

Build DB name from configuration.

Parameters:

db_name: str: Direct database name (if provided, returns this).
db_simulation_model_version: str: Version of the simulation model.
model_name: str: Name of the simulation model.

Returns:

str or None: Database name.

get_ecsv_file_as_astropy_table(file_name, db_name)[source]#

Read contents of an ECSV file from the database and return it as an Astropy Table.

Files are not written to disk.

Parameters:

file_name: str: The name of the ECSV file.
db_name: str: The name of the database.

Returns:

astropy.table.Table: The contents of the ECSV file as an Astropy Table.

static get_entry_date_from_document(document)[source]#

Extract entry date from a MongoDB document’s ObjectId.

Parameters:

document: dict: MongoDB document with ‘_id’ field.

Returns:

datetime.datetime: The generation time of the document’s ObjectId.

get_file_from_db(db_name, file_name)[source]#

Extract a file from MongoDB and return GridFS file instance.

Parameters:

db_name: str: The name of the DB with files of tabulated data
file_name: str: The name of the file requested

Returns:

GridOut: A file instance returned by GridFS find_one

Raises:

FileNotFoundError: If the desired file is not found.

insert_file_to_db(file_name, db_name, **kwargs)[source]#

Insert a file to the DB.

Parameters:

file_name: str or Path: The name of the file to insert (full path).
db_name: str: The name of the DB
**kwargs (optional): keyword arguments for file creation.: The full list of arguments can be found in https://www.mongodb.com/docs/manual/core/gridfs/

Returns:

file_id: GridOut._id: If the file exists, return its GridOut._id, otherwise insert the file and return its newly created DB GridOut._id.

insert_one(document, collection_name, db_name)[source]#

Insert a document into a collection.

Parameters:

document: dict: Document to insert.
collection_name: str: Collection name.
db_name: str: Database name.

Returns:

InsertOneResult: Result of the insert operation.

is_remote_database()[source]#

Check if the database is remote.

Check for domain pattern like “cta-simpipe-protodb.zeuthen.desy.de”

Returns:

bool: True if the database is remote, False otherwise.

list_database_names()[source]#

Get list of database names.

Returns:

list: List of database names.

print_connection_info(db_name)[source]#

Print the connection information.

Parameters:

db_name: str: Name of the database.

query_db(query, collection_name, db_name)[source]#

Query MongoDB and return results as list.

Parameters:

query: dict: Query to execute.
collection_name: str: Collection name.
db_name: str: Database name.

Returns:

list: List of documents matching the query.

Raises:

ValueError: if query returned no results.

static validate_db_config(db_config)[source]#

Validate the MongoDB configuration.

Parameters:

db_config: dict: Dictionary with the MongoDB configuration.

Returns:

dict or None: Validated MongoDB configuration or None if no valid config provided.

Raises:

ValueError: If the MongoDB configuration is invalid.

write_file_from_db_to_disk(db_name, path, file)[source]#

Extract a file from MongoDB and write it to disk.

Parameters:

db_name: str: The name of the DB with files of tabulated data
path: str or Path: The path to write the file to
file: GridOut: A file instance returned by GridFS find_one

db_model_upload#

Upload a simulation model (parameters and production tables) to the database.

db.db_model_upload.add_complete_model(tmp_dir, db, db_simulation_model, db_simulation_model_version, repository_url, repository_branch=None)[source]#

Upload a complete model including model parameters and production tables to the database.

Generate compound indexes for the new database.

Parameters:

tmp_dirPath or str: Temporary directory to use for cloning the repository.
dbDatabaseHandler: Database handler object.
db_simulation_modelstr: Name of the simulation model in the database.
db_simulation_model_versionstr: Version of the simulation model in the database.
repository_urlstr: URL of the simulation model repository to clone.
repository_branchstr, optional: Branch of the repository to use. If None, the default branch is used.

Returns:

None: This function does not return a value.

db.db_model_upload.add_model_parameters_to_db(input_path, db)[source]#

Read model parameters from a directory and upload them to the database.

Parameters:

input_pathPath, str: Path to the directory containing the model parameters.
dbDatabaseHandler: Database handler object.

db.db_model_upload.add_production_tables_to_db(input_path, db)[source]#

Read production tables from a directory and upload them to the database.

One dictionary per collection is prepared for each model version, containing tables of all array elements, sites, and configuration parameters.

Parameters:

input_pathPath, str: Path to the directory containing the production tables.
dbDatabaseHandler: Database handler object.

db.db_model_upload.add_values_from_json_to_db(file, collection, db, file_prefix)[source]#

Upload new model parameter from json files to db.

Parameters:

filelist: JSON file to be uploaded to the DB.
collectionstr: The DB collection to which to add the file.
dbDatabaseHandler: Database handler object.
file_prefixstr: Path to location of all additional files to be uploaded.

db.db_model_upload.clone_simulation_model_repository(target_dir, repository_url, db_simulation_model_version, repository_branch)[source]#

Clone a git repository containing simulation model parameters and production tables.

Parameters:

target_dirPath or str: Target directory to clone the repository into.
repository_urlstr: URL of the git repository to clone.
db_simulation_model_versionstr: Version tag of the simulation model to checkout.
repository_branchstr, optional: Branch of the repository to use. If None, the version tag is used.

Returns:

Path: Path to the cloned repository.

Database support#

db_handler#

mongo_db#

db_model_upload#

This Page