Database support

Database support#

Modules for database access. See the databases sections for details.

db_handler#

Module to handle interaction with DB.

class db.db_handler.DatabaseHandler(mongo_db_config=None)[source]#

DatabaseHandler provides the interface to the DB.

Parameters:

mongo_db_config: dict: Dictionary with the MongoDB configuration with the following entries: “db_server” - DB server address “db_api_port” - Port to use “db_api_user” - API username “db_api_pw” - Password for the API user “db_api_authentication_database” - DB with user info (optional, default is “admin”) “db_simulation_model” - Name of simulation model database

add_new_parameter(db_name, version, parameter, value, array_element_name=None, site=None, collection_name='telescopes', file_prefix=None, **kwargs)[source]#

Add a parameter value for a specific array element.

A new document will be added to the DB, with all fields taken from the input parameters.

Parameters:

db_name: str: the name of the DB
version: str: The version of the new parameter value
parameter: str: Which parameter to add
value: can be any type, preferably given in kwargs: The value to set for the new parameter
array_element_name: str: The name of the array element to add a parameter to (only used if collection_name is not “sites”).
site: str: Site name; ignored if collection_name is “telescopes”.
collection_name: str: The name of the collection to add a parameter to.
file_prefix: str or Path: where to find files to upload to the DB
kwargs: dict: Any additional fields to add to the parameter

Raises:

ValueError: If key to collection_name is not valid.

copy_array_element(db_name, element_to_copy, version_to_copy, new_array_element_name, collection_name='telescopes', db_to_copy_to=None, collection_to_copy_to=None)[source]#

Copy a full array element configuration to a new array element name.

Only a specific version is copied. (This function should be rarely used, probably only during “construction”.)

Parameters:

db_name: str: the name of the DB to copy from
element_to_copy: str: The array element to copy
version_to_copy: str: The version of the configuration to copy
new_array_element_name: str: The name of the new array element
collection_name: str: The name of the collection to copy from.
db_to_copy_to: str: The name of the DB to copy to.
collection_to_copy_to: str: The name of the collection to copy to.

Raises:

BulkWriteError

copy_documents(db_name, collection, query, db_to_copy_to, collection_to_copy_to=None)[source]#

Copy the documents matching to “query” to the DB “db_to_copy_to”.

The documents are copied to the same collection as in “db_name”. (This function should be rarely used, probably only during “construction”.)

Parameters:

db_name: str

the name of the DB to copy from

collection: str

the name of the collection to copy from

query: dict

A dictionary with a query to search for documents to copy. For example, the query below would copy all entries of version 6.0.0 from telescope LSTN-01 to “db_to_copy_to”.

query = {
    "instrument": "LSTN-01",
    "version": "6.0.0",
}

db_to_copy_to: str

The name of the DB to copy to.

Raises:

BulkWriteError

delete_query(db_name, collection, query)[source]#

Delete all entries from the DB which correspond to the provided query.

(This function should be rarely used, if at all.)

Parameters:

db_name: str

the name of the DB

collection: str

the name of the collection to copy from

query: dict

A dictionary listing the fields/values to delete. For example, the query below would delete the entire version 6.0.0 from telescope LSTN-01.

query = {
    "instrument": "LSTN-01",
    "version": "6.0.0",
}

export_file_db(db_name, dest, file_name)[source]#

Get file from the DB and write to disk.

Parameters:

db_name: str: Name of the DB to search in.
dest: str or Path: Location where to write the file to.
file_name: str: Name of the file to get.

Returns:

file_id: GridOut._id: the database ID the file was assigned when it was inserted to the DB.

Raises:

FileNotFoundError: If the desired file is not found.

export_model_files(parameters, dest)[source]#

Export all the files in a model from the DB and write them to disk.

Parameters:

parameters: dict: Dict of model parameters
dest: str or Path: Location where to write the files to.

Raises:

FileNotFoundError: if a file in parameters.values is not found

get_all_versions(parameter=None, array_element_name=None, site=None, db_name=None, collection=None)[source]#

Get all version entries in the DB of collection and/or a specific parameter.

Parameters:

parameter: str: Which parameter to get the versions of
array_element_name: str: Which array element to get the versions of (in case “collection_name” is not “sites”)
site: str: Site name.
db_name: str: Database name.
collection_name: str: The name of the collection in which to update the parameter.

Returns:

all_versions: list: List of all versions found

Raises:

ValueError: If key to collection_name is not valid.

get_collection(db_name, collection_name)[source]#

Get a collection from the DB.

Parameters:

db_name: str: Name of the DB.
collection_name: str: Name of the collection.

Returns:

pymongo.collection.Collection: The collection from the DB.

get_collections(db_name=None, model_collections_only=False)[source]#

List of collections in the DB.

Parameters:

db_name: str: Database name.

Returns:

list: List of collection names
model_collections_only: bool: If True, only return model collections (i.e. exclude fs.files, fs.chunks, metadata)

get_corsika_configuration_parameters(model_version)[source]#

Get CORSIKA configuration parameters from the DB.

Parameters:

model_versionstr: Version of the model.

Returns:

dict: Configuration parameters for CORSIKA

get_derived_values(site, array_element_name, model_version)[source]#

Get all derived values from the DB for a specific array element.

Parameters:

site: str: Site name.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
model_version: str: Version of the model.

Returns:

dict containing the parameters

get_model_parameters(site, array_element_name, model_version, collection='telescope', only_applicable=False)[source]#

Get parameters from MongoDB or simulation model repository for an array element.

An array element can be e.g., a telescope or a calibration device. Read parameters for design and for the specified array element (if necessary). This allows to overwrite design parameters with specific parameters without having to copy all model parameters when changing only a few.

Parameters:

site: str: Site name.
array_element_name: str: Name of the array element model (e.g. LSTN-01, MSTS-design)
model_version: str: Version of the model.
collection: str: collection of array element (e.g. telescopes, calibration_devices)
only_applicable: bool: If True, only applicable parameters will be read.

Returns:

dict containing the parameters

get_simulation_configuration_parameters(simulation_software, site, array_element_name, model_version)[source]#

Get simulation configuration parameters from the DB.

Parameters:

simulation_software: str: Name of the simulation software.
site: str: Site name.
array_element_name: str: Name of the array element model (e.g. MSTN, SSTS).
model_version: str: Version of the model.

Returns:

dict containing the parameters

Raises:

ValueError: if simulation_software is not valid.

get_site_parameters(site, model_version, only_applicable=False)[source]#

Get parameters from either MongoDB or simulation model repository for a specific site.

Parameters:

site: str: Site name.
model_version: str: Version of the model.
only_applicable: bool: If True, only applicable parameters will be read.

Returns:

dict containing the parameters

insert_file_to_db(file_name, db_name=None, **kwargs)[source]#

Insert a file to the DB.

Parameters:

file_name: str or Path: The name of the file to insert (full path).
db_name: str: the name of the DB
**kwargs (optional): keyword arguments for file creation.: The full list of arguments can be found in, https://docs.mongodb.com/manual/core/gridfs/#the-files-collection mostly these are unnecessary though.

Returns:

file_iD: GridOut._id: If the file exists, return its GridOut._id, otherwise insert the file and return its” “newly created DB GridOut._id.

model_version(version, db_name=None)[source]#

Return model version and check that it is valid.

Queries the database for all available model versions and check if the requested version is valid.

Parameters:

versionstr: Model version.
db_namestr: Database name.

Returns:

str: Model version.

Raises:

ValueError: if version not valid.

read_mongo_db(db_name, array_element_name, model_version, run_location, collection_name, write_files=True, only_applicable=False)[source]#

Build and execute query to Read the MongoDB for a specific array element.

Also writes the files listed in the parameter values into the sim_telarray run location

Parameters:

db_name: str: the name of the DB
array_element_name: str: Name of the array element model (e.g. MSTN-design …)
model_version: str: Version of the model.
run_location: Path or str: The sim_telarray run location to write the tabulated data files into.
collection_name: str: The name of the collection to read from.
write_files: bool: If true, write the files to the run_location.
only_applicable: bool: If True, only applicable parameters will be read.

Returns:

dict containing the parameters

Raises:

ValueError: if query returned zero results.

update_parameter_field(db_name, model_version, parameter, field, new_value, array_element_name=None, site=None, collection_name='telescopes')[source]#

Update a parameter field value for a specific array element/version.

This function only modifies the value of one of the following DB entries: Applicable, units, Type, items, minimum, maximum. These type of changes should be very rare. However they can be done without changing the Object ID of the entry since they are generally “harmless”.

Parameters:

db_name: str: the name of the DB
model_version: str: Which model version to update
parameter: str: Which parameter to update
field: str: Field to update (only options are Applicable, units, Type, items, minimum, maximum). If the field does not exist, it will be added.
new_value: type identical to the original field type: The new value to set to the field given in “field”.
array_element_name: str: Which array element to update, if None then update a site parameter
site: str, North or South: Update a site parameter (the array_element_name argument must be None)
collection_name: str: The name of the collection in which to update the parameter.

Raises:

ValueError: if field not in allowed fields

db_array_elements#

Retrieval of array elements from the database.

db.db_array_elements.get_array_element_list_for_db_query(array_element_name, db, model_version, collection)[source]#

Get array element name and design model for querying the database.

Return a list of array element names to be used for querying the database for a given array element. This is in most cases the array element itself and its design model. In cases of no design model available, the design model of the array element is returned.

Parameters:

array_element_name: str: Name of the array element model (e.g. MSTN-01).
db: DBHandler: Instance of the database handler
model_version: str: Model version.
collection: str: DB collection to get the array elements from (e.g., telescopes, calibration_devices)

Returns:

list: List of array element model names as used in the DB.

db.db_array_elements.get_array_elements(db_collection, model_version)[source]#

Get all array element names and their design model for a given DB collection.

Uses the ‘design_model’ parameter to determine the design model of the array element. Assumes that a design model is defined for every array element.

Parameters:

db_collection:: pymongo.collection.Collection
model_version: str: Model version.

Returns:

dict: Dict with array element names found and their design model

Raises:

ValueError: If query for collection name not implemented.
KeyError: If array element entry in the database is incomplete.

db.db_array_elements.get_array_elements_of_type(array_element_type, db, model_version, collection)[source]#

Get all array elements of a certain type in the specified collection in the DB.

Return e.g. for array_element_type=’MSTS’ all MSTS array elements found in the collection.

Parameters:

array_element_typestr: Type of the array element (e.g. LSTN, MSTS)
model_versionstr: Which version to get the array elements of
collectionstr: Which collection to get the array elements from: i.e. telescopes, calibration_devices
db_namestr: Database name

Returns:

list: Sorted list of all array element names found in collection

db_from_repo_handler#

Module to mimic DB interaction for simulation model DB development.

Read simulation model values from files in simulation model repository.

db.db_from_repo_handler.update_model_parameters_from_repo(parameters, site, array_element_name, parameter_collection, model_version, db_simulation_model_url)[source]#

Update model parameters with values from a repository.

Existing entries will be updated, new entries will be added.

Parameters:

parameters: dict: Existing dictionary with parameters to be updated.
site: str: Observatory site (e.g., South or North)
array_element_name: str: Array element name (e.g., MSTN-01, MSTN-design)
parameter_collection: str: Collection of parameters to be queried (e.g., telescope or site)
model_version: str: Model version to use.
db_simulation_model_url: str: URL to the simulation model repository.

Returns:

dict: Updated dictionary with parameters.

Database support

Contents

Database support#

db_handler#

db_array_elements#

db_from_repo_handler#