Database support

Database support#

Modules for database access. See the databases sections for details.

db_handler#

Module to handle interaction with DB.

class db.db_handler.DatabaseHandler(mongo_db_config=None)[source]#

DatabaseHandler provides the interface to the DB.

Parameters:

mongo_db_config: dict: Dictionary with the MongoDB configuration with the following entries: “db_server” - DB server address “db_api_port” - Port to use “db_api_user” - API username “db_api_pw” - Password for the API user “db_api_authentication_database” - DB with user info (optional, default is “admin”) “db_simulation_model” - Name of simulation model database

add_new_parameter(db_name, version, parameter, value, telescope=None, site=None, collection_name='telescopes', file_prefix=None, **kwargs)[source]#

Add a parameter value for a specific telescope.

A new document will be added to the DB, with all fields taken from the input parameters.

Parameters:

db_name: str: the name of the DB
version: str: The version of the new parameter value
parameter: str: Which parameter to add
value: can be any type, preferably given in kwargs: The value to set for the new parameter
telescope: str: The name of the telescope to add a parameter to (only used if collection_name is “telescopes”).
site: str: Site name; ignored if collection_name is “telescopes”.
collection_name: str: The name of the collection to add a parameter to.
file_prefix: str or Path: where to find files to upload to the DB
kwargs: dict: Any additional fields to add to the parameter

Raises:

ValueError: If key to collection_name is not valid. Valid entries are: ‘telescopes’ and ‘sites’.

add_tagged_version(tags, db_name=None)[source]#

Set the tag of the “Released” or “Latest” version of the MC Model.

Parameters:

released_version: str: The version name to set as “Released”
tags: dict: The version tags consisting of tag name and version name.
db_name: str: Database name

copy_documents(db_name, collection, query, db_to_copy_to, collection_to_copy_to=None)[source]#

Copy the documents matching to “query” to the DB “db_to_copy_to”.

The documents are copied to the same collection as in “db_name”. (This function should be rarely used, probably only during “construction”.)

Parameters:

db_name: str

the name of the DB to copy from

collection: str

the name of the collection to copy from

query: dict

A dictionary with a query to search for documents to copy. For example, the query below would copy all entries of prod4 version from telescope LSTN-01 to “db_to_copy_to”.

query = {
    "instrument": "LSTN-01",
    "version": "prod6",
}

db_to_copy_to: str

The name of the DB to copy to.

Raises:

BulkWriteError

copy_telescope(db_name, tel_to_copy, version_to_copy, new_tel_name, collection_name='telescopes', db_to_copy_to=None, collection_to_copy_to=None)[source]#

Copy a full telescope configuration to a new telescope name.

Only a specific version is copied. (This function should be rarely used, probably only during “construction”.)

Parameters:

db_name: str: the name of the DB to copy from
tel_to_copy: str: The telescope to copy
version_to_copy: str: The version of the configuration to copy
new_tel_name: str: The name of the new telescope
collection_name: str: The name of the collection to copy from.
db_to_copy_to: str: The name of the DB to copy to.
collection_to_copy_to: str: The name of the collection to copy to.

Raises:

BulkWriteError

delete_query(db_name, collection, query)[source]#

Delete all entries from the DB which correspond to the provided query.

(This function should be rarely used, if at all.)

Parameters:

db_name: str

the name of the DB

collection: str

the name of the collection to copy from

query: dict

A dictionary listing the fields/values to delete. For example, the query below would delete the entire prod6 version from telescope LSTN-01.

query = {
    "instrument": "LSTN-01",
    "version": "prod6",
}

export_file_db(db_name, dest, file_name)[source]#

Get file from the DB and write to disk.

Parameters:

db_name: str: Name of the DB to search in.
dest: str or Path: Location where to write the file to.
file_name: str: Name of the file to get.

Returns:

file_id: GridOut._id: the database ID the file was assigned when it was inserted to the DB.

Raises:

FileNotFoundError: If the desired file is not found.

export_model_files(parameters, dest)[source]#

Export all the files in a model from the DB and write them to disk.

Parameters:

parameters: dict: Dict of model parameters
dest: str or Path: Location where to write the files to.

Raises:

FileNotFoundError: if a file in parameters.values is not found

get_all_available_array_elements(model_version, collection, db_name=None)[source]#

Get all available array element names in the specified collection in the DB.

Parameters:

db_name: str: the name of the DB
model_version: str: Which version to get the array elements of
collection: str: Which collection to get the array elements from: i.e. telescopes, calibration_devices
db_namestr: Database name

Returns:

_available_array_elements: list: List of all array element names found in collection

get_all_versions(parameter=None, telescope_model_name=None, site=None, collection='telescopes')[source]#

Get all version entries in the DB of collection and/or a specific parameter.

Parameters:

parameter: str: Which parameter to get the versions of
telescope_model_name: str: Which telescope to get the versions of (in case “collection_name” is “telescopes”)
site: str: Site name. In case “collection_name” is “telescopes”, the site is used to build the telescope name. In case “collection_name” is “sites”, this argument sets which site parameter get the versions of
collection_name: str: The name of the collection in which to update the parameter.

Returns:

all_versions: list: List of all versions found

Raises:

ValueError: If key to collection_name is not valid. Valid entries are: ‘telescopes’ and ‘sites’.

get_available_array_elements_of_type(array_element_type, model_version, collection, db_name=None)[source]#

Get all array elements of a certain type in the specified collection in the DB.

Parameters:

array_element_typestr: Type of the array element (e.g. LSTN, MSTS)
model_versionstr: Which version to get the array elements of
collectionstr: Which collection to get the array elements from: i.e. telescopes, calibration_devices
db_namestr: Database name

Returns:

list: List of all array element names found in collection

get_collections(db_name=None)[source]#

List of collections in the DB.

Parameters:

db_name: str: Database name.

Returns:

list: List of collection names

get_derived_values(site, telescope_model_name, model_version)[source]#

Get all derived values from the DB for a specific telescope.

Parameters:

site: str: Site name.
telescope_model_name: str: Name of the telescope model (e.g. MSTN, SSTS).
model_version: str: Version of the model.

Returns:

dict containing the parameters

get_model_parameters(site, telescope_model_name, model_version, collection='telescope', only_applicable=False)[source]#

Get parameters from MongoDB or simulation model repository for an array element.

An array element can be e.g., a telescope or a calibration device. Read parameters for design and for the specified array element (if necessary). This allows to overwrite design parameters with specific telescope parameters without having to copy all model parameters when changing only a few.

Parameters:

site: str: Site name.
telescope_model_name: str: Name of the telescope model (e.g. LSTN-01, MSTS-design)
model_version: str: Version of the model.
collection: str: collection of array element (e.g. telescopes, calibration_devices)
only_applicable: bool: If True, only applicable parameters will be read.

Returns:

dict containing the parameters

get_sim_telarray_configuration_parameters(site, telescope_model_name, model_version)[source]#

Get sim_telarray configuration parameters from the DB for a specific telescope.

Parameters:

sitestr: Site name.
telescope_model_namestr: Name of the telescope model (e.g. MSTN).
model_versionstr: Version of the model.

Returns:

dict: Configuration parameters for sim_telarray

get_site_parameters(site, model_version, only_applicable=False)[source]#

Get parameters from either MongoDB or simulation model repository for a specific site.

Parameters:

site: str: Site name.
model_version: str: Version of the model.
only_applicable: bool: If True, only applicable parameters will be read.

Returns:

dict containing the parameters

get_telescope_db_name(telescope_name, model_version, collection='telescopes')[source]#

Translate telescope name to the name used in the DB.

This is required, as not all telescopes are defined in the database yet. In these cases, use the “design” telescope.

Parameters:

telescope_name: str: Name of the telescope model (e.g. MSTN-01)
model_version: str: Which version to get the telescopes
collection: str: collection of array element (e.g. telescopes, calibration_devices)

Returns:

str: Telescope model name as used in the DB.

Raises:

ValueError: If the telescope name is not found in the database.

get_telescope_list_for_db_query(telescope_model_name, model_version)[source]#

Return a list of telescope names to be used for querying the database.

The first entry of the list is the design telescope (if it exists in the DB), followed by the actual telescope model name.

Parameters:

telescope_model_name: str: Name of the telescope model (e.g. MSTN-01).
model_version: str: Model version.

Returns:

list: List of telescope model names as used in the DB.

insert_file_to_db(file_name, db_name=None, **kwargs)[source]#

Insert a file to the DB.

Parameters:

file_name: str or Path: The name of the file to insert (full path).
db_name: str: the name of the DB
**kwargs (optional): keyword arguments for file creation.: The full list of arguments can be found in, https://docs.mongodb.com/manual/core/gridfs/#the-files-collection mostly these are unnecessary though.

Returns:

file_iD: GridOut._id: If the file exists, return its GridOut._id, otherwise insert the file and return its” “newly created DB GridOut._id.

model_version(version='Released', db_name=None)[source]#

Return the model version for the requested tag.

Resolve the “Released” or “Latest” tag to the actual version name.

Parameters:

versionstr: Model version.
db_namestr: Database name.

Returns:

str: Model version.

Raises:

ValueError: if version not valid. Valid versions are: ‘Released’ and ‘Latest’.

read_mongo_db(db_name, telescope_model_name, model_version, run_location, collection_name, write_files=True, only_applicable=False)[source]#

Build and execute query to Read the MongoDB for a specific telescope.

Also writes the files listed in the parameter values into the sim_telarray run location

Parameters:

db_name: str: the name of the DB
telescope_model_name: str: Name of the telescope model (e.g. MSTN-design …)
model_version: str: Version of the model.
run_location: Path or str: The sim_telarray run location to write the tabulated data files into.
collection_name: str: The name of the collection to read from.
write_files: bool: If true, write the files to the run_location.
only_applicable: bool: If True, only applicable parameters will be read.

Returns:

dict containing the parameters

Raises:

ValueError: if query returned zero results.

update_parameter_field(db_name, model_version, parameter, field, new_value, telescope=None, site=None, collection_name='telescopes')[source]#

Update a parameter field value for a specific telescope/version.

This function only modifies the value of one of the following DB entries: Applicable, units, Type, items, minimum, maximum. These type of changes should be very rare. However they can be done without changing the Object ID of the entry since they are generally “harmless”.

Parameters:

db_name: str: the name of the DB
model_version: str: Which model version to update
parameter: str: Which parameter to update
field: str: Field to update (only options are Applicable, units, Type, items, minimum, maximum). If the field does not exist, it will be added.
new_value: type identical to the original field type: The new value to set to the field given in “field”.
telescope: str: Which telescope to update, if None then update a site parameter
site: str, North or South: Update a site parameter (the telescope argument must be None)
collection_name: str: The name of the collection in which to update the parameter.

Raises:

ValueError: if field not in allowed fields

db_from_repo_handler#

Module to mimic DB interaction for simulation model DB development. Read simulation model values from files in simulation model repository.

db.db_from_repo_handler.update_model_parameters_from_repo(parameters, site, telescope_name, parameter_collection, model_version, db_simulation_model_url, db_simulation_model='verified_model')[source]#

Update model parameters with values from a repository. Existing entries will be updated, new entries will be added.

Parameters:

parameters: dict: Existing dictionary with parameters to be updated.
site: str: Observatory site (e.g., South or North)
telescope_name: str: Telescope name (e.g., MSTN-01, MSTN-DESIGN)
parameter_collection: str: Collection of parameters to be queried (e.g., telescope or site)
model_version: str: Model version to use.
db_simulation_model_url: str: URL to the simulation model repository.

Returns:

dict: Updated dictionary with parameters.

Database support

Contents

Database support#

db_handler#

db_from_repo_handler#