CHANGELOG#
All notable changes to the simtools project will be documented in this file. Changes for upcoming releases can be found in the docs/changes directory.
This changelog is generated using Towncrier, see the developer documentation for more information.
v0.25.0 - 2025-11-19#
Bugfixes#
Bugfix in comparison of sim-telarray configuration files: ignore system dependent metadata values. (#1873)
Fix thread-safe singleton check in
mongo_db. Add{"maxIdleTimeMS": 10000}to close idle connections after 10s. Add debugging tool for connections. (#1899)Fix simulation model production table upload: full updates tables need not to search through the model history. This led to the addition of deprecated parameters back into the production. (#1913)
New Features#
Add pulse shape table writer for light emission flasher simulations. (#1851)
Add CORSIKA executable compiled with CURVED option to docker images (
corsika-curved). (#1859)Improve array plotting with more custom configuration options. (#1860, #1862)
Add tests for certain strings in the CORSIKA and sim_telarray log files. (#1882)
Verify that the request and actually simulated number of events is identical for CORSIKA and sim_telarray simulations. (#1883)
Add tests that certain strings do not appear in the CORSIKA and sim_telarray log file (see also PR #1882) (#1885)
Added functionality to plot mirrors and implemented auto plotting of mirrors for the reports. (#1886)
Add test to verify that number of CORSIKA events is consistent with requested number of events. (#1890)
Add application for pulse shape parameter derivation and move existing functionality there. (#1894)
Add possibility to run sim_telarray only using an existing CORSIKA file with
simtools-simulate-prod. (#1896)
Maintenance#
Refactor
simtools-validate-file-using-schemaand improve unit testing of code. (#1857)Introducing a new psf optimizer class and improving exception handling. (#1864)
Update model version to ensure reproducibility if given as MAJOR.MINOR. Revert testing to exact semver version. (#1870)
Improve description of
submit_array_layouts. Fix model parameter writing of instrument for site parameters. (#1872)Introducing a data class and capping maximum learning rate in PSF optimization modules. (#1875)
Remove unused command line parameter
data_directoryfrom simtools-simulate-prod. (#1891)Remove unused command line parameter
number_of_runsfrom simtools-simulate-prod. Simulate always a single run number. (#1893)Introduce consistent naming for pedestal runs and output files: always start with
pedestal...(e.g.,pedestals_dark). (#1904)Suppress mongoDB connection close debug message in unit test. (#1906)
Documentation is now generated for each push. Deployment changed to either the release of a version, merge to main, or manual trigger. (#1907)
Update ignore pattern for log file inspector to ignore warnings on
getpwuidobtained in AIV environment. (#1912)
Simulation model#
Merge all flasher pulse shape parameters (shape, width, exp decay) into a single parameter
flasher_pulse_shape. (#1895)
v0.24.0 - 2025-10-28#
Bugfixes#
Bugfix in validation of generated sim_telarray configuration: missing assert on the outcome of the tests (tests always passed, even with differences). (#1856)
Fixed array trigger file generation to properly combine non-hardstereo telescopes. Previously, each telescope type generated separate trigger lines, preventing software stereo triggers between different telescope types (e.g., SST+MST). Now hardstereo lines remain separate while non-hardstereo telescopes are combined into a single line, enabling cross-telescope-type triggering. (#1863)
New Features#
Add new simulation model parameter
transit_time_random. (#1808)Implement version check for model parameters for any software type. (#1842)
Add possibility to define new base models taking into account the correct history of models for
simtools-maintain-simulation-model-add-production. (#1844)Add possibility to change model parameters from command line using the
--overwrite_model_parameterscommand line parameter. (#1850)
Maintenance#
Change source of simulation model database in CI to ‘main’ branch of simulation model repository. Tests against the latest release versions for simtools release candidates (
1.2.3-rc). Allow to configure simulation model repository branches for manually triggered workflows. (#1846)Review and update all XPASS unit tests. (#1855)
Simulation model#
Change default simulation models database version to 0.11.0. (#1852)
v0.23.0 - 2025-10-13#
Bugfixes#
Bugfix in comparing model versions in integration tests to decide which tests should run. Bugfix in getting model parameter values with units for the case of value-unit pairs consisting of lists with None entries. (#1805)
Fix a bug where hard stereo trigger was set every time the hard_stereo key existed in dictionary, rather than checking its value. Now it only sets hardstereo if the value is True. (#1813)
Bugfix packing model files: model version was not taken into account correctly. (#1818)
Bugfix in preparing sim_telarray command for shower simulations: power-law configuration (used to fill histograms) set at the wrong place. (#1825)
Bugfix in writing sim_telarray configuration option: “-C show=all” should be last in command. (#1828)
New Features#
Add new CORSKIA / sim_telarray image build for:
CORSIKA 7.8010
Bernloehr package 1.70
sim_telarray 250903
Improved downloading script and handling of QGSJet tables.
(#1824)
Fill telescope design types into sim_telarray metadata field for
camera_config_variantandoptics_config_variant. (#1826)Allow to highlight or grey out telescopes in layout plotting application. (#1829)
Allow to provide integers in scientific notation in yaml configuration files. Currently only implemented for the –number_of_photons argument of calculate_incident_angles.py. (#1837)
Maintenance#
Add application control functionality for all simtools applications. (#1809)
Adding –all_model_versions flag for calibration and software report generation. (#1815)
Replace superlinter in linter CI. Add toml linter and shellcheck to precommit. (#1838)
Update one function used in the tests to comply with version 2.x of pyeventio. (#1840)
v0.22.0 - 2025-09-30#
Bugfixes#
Documentation#
Improve documentation on DB upload. (#1775)
New Features#
Add software version range validation. Introduce software version for updated or new flasher simulation model parameters. (#1763)
Implement simulation models production table and model parameter update using the
simtools-maintain-simulation-model-add-productionapplication. (#1765)Schema for info.yml files used in simulation models. (#1790)
Add flat-fielding devices for all telescope types. (#1798)
Add new application
simtools-db-upload-model-repositoryfor simplified upload of repository data to the database. (#1801)Add flasher exponential decay pulse shape parameter. (#1802)
Maintenance#
Linked parameter plots to parameter comparison reports. (#1661)
Added CLI argument to flexibly change PSF containment percentage as needed. (#1766)
Remove ‘LATEST’ options when setting the
DB_SIMULATION_MODEL_VERSIONconfiguration option (require explicit setting of the DB version). (#1785)Introduce a simpler naming for the database (CTAO-Simulation-Model instead of CTAO-Simulation-ModelParameters). (#1787)
Remove
use_plain_outputoption and generalize output direction generation. (#1792)Simplify generation of production image and remove outdated
build-corsika-simtelarraytarget. (#1794)Increase robustness for unit and integration tests when cloning remote Git repositories. Add a 3x retry. (#1796)
Split CORSIKA/sim_telarray and simtools images into two. (#1797)
Change minimum python version requirement to 3.12 (following CTAO recommendation). (#1799)
Remove ‘instrument:type’ and ‘instrument:site’ in model parameter schema files. (#1803)
v0.21.0 - 2025-09-23#
Bugfixes#
New Features#
Implemented gradient descent minimisation for psf parameter optimisation, KS statistic as a sanity check, and monte carlo error on the simulated psf. (#1717)
Add functionality to work with delta-type production tables of the simulation model for patch updates. (#1758)
Automatic retrieval of the latest patch version for the simulation model. Allow
--model_version=6.0being resolved to the newest version. (#1768)
v0.20.0 - 2025-09-12#
Bugfixes#
Fix warning messages in simtel metadata reader to avoid the log-file testing routines to fail. (#1712)
Developer image is using an old base image with CORSIKA and sim_telarray. Fix naming of containers used in developer image to ensure the most recent are used. (#1716)
Introduce
DB_SIMULATION_MODEL_VERSIONto fix instability of environment when introducing new model parameters. (#1729)Fix reduce complexity issue raised in sonar in plot simtel events. (#1735)
Fix bug in schema file for model parameters regarding schema version. Improved printing on error. (#1742)
New Features#
Add new application to calculate telescope and array trigger rates from reduced event lists. (#1685)
Add application to calculate photon incident angles at the camera plane. (#1687)
Add sim_telarry executable compiled with debug options to containers. Allows to generate additional output columns for e.g. sim_telarray ray tracing. (#1715)
Add / update
nsb_spectrummodel parameter and calculate integrated NSB rate. (#1718)Add avx512f and sse4 vector optimization images for production images. (#1741)
Maintenance#
Remove calculate trigger rate application and associated code. Replaced by simtools-derive-trigger-rate (see PR #1685). (#1693)
Improve robustness of image building and downloading autoconf. (#1722)
Improvement of CI-workflow efficiency and documentation. (#1723)
Improve handling of secrets in CI workflows. (#1728)
Add new field
production_table_nameto production table schema. (#1756)Remove Codecov from CI (covered by SonarQube). (#1759)
v0.19.0 - 2025-08-27#
Bugfixes#
Fix bug in reading list of array elements from file. Only layout names were used and the list was read not from file but from the model parameters DB. (#1658)
Ensure that run numbers start at 1 for cases no offset if given (HT Condor job submission). (#1686)
Fix array trigger writing for single telescope simulations. (#1690)
Bugfix in shower core calculation in shower coordinates for reduced event data. (#1703)
New Features#
Add additional plots for the derivation of CORSIKA limits. (#1632)
Add a warning to derived corsika limit if they are close to the limits of the simulations. (#1643)
Add an application that merges derived corsika simulation limits for each available grid point. (#1649)
Add curvature radius to testeff command. User for calculation of NSB rates per pixel. (#1651)
Allow to run applications in container environments using simtools-run-application. Introduce separate module (simtools-runner) and improve support for workflow running. (#1657)
Add application to update, modify, and compare production tables. Improve documentation for simulation repository maintenance. (#1660)
Add application to simulate calibration events (pedestals, dark pedestals, flasher). (#1663)
Add functionality to simtools-validate-camera-efficiency to write out model parameter JSON with nsb_pixel_rate. (#1665)
Add functionality to compare derived model parameters with DB values to integration tests (examples for simtools-validate-camera-efficiency). (#1666)
Add log file inspector for integration tests to fail tests when errors or runtime warnings are reported. (#1674)
Add application to simulate flasher with the light emission package. (#1676)
Add sim_telarray model files to output of each production run. This asserts reproducibility and simplifies debugging. (#1689)
Add MongoDB compound indices to improve query efficiency. (#1691)
Add possibility to set random seed for random focal length settings in simtools-derive-mirror-rnda. (#1702)
Maintenance#
Apply more consistent key-name convention with lower-case snake_make to application workflow configurations. (#1636)
Clarify input data to derivation of CORSIKA limits with
simtools-production-derive-corsika-limit. (#1641)Change metadata to lower-case snake make. More consistent usage of schema versions. (#1645)
Add missing plotting configurations for tabular data. (#1647)
Plots are now auto-generated alongside reports. (#1653)
Refactoring of ASCII input io operations. (#1667)
Add a generic writer for structured data for JSON and YAML format. (#1668)
Simplification of legend_handlers class. (#1670)
Change units of shower- and pointing directions in reduced event data from radians to degrees. (#1672)
Generalization of reduced event-data histogram filling (moved from CORSIKA limits derivation modules to generic module). (#1675)
Fix hadolint issue and using parameter expansion for replacement in Dockerfile. (#1680)
Refactor PSF parameter optimization workflow and improve output. Added a flag to export best parameters as JSON files. (#1681)
Revision of main repository readme file and removed duplications. (#1684)
Remove DB access of unit test for the
docs_read_parametersmodule. (#1688)Maintenance pass through integration test to remove duplicated tests and improve test efficiency. (#1694)
Improve code quality by addressing SonarQube reliability issues. (#1697)
Unit tests will now fail in the CI for warnings. (#1699)
Fix some security hotspots reported by SonarQube. (#1700)
Improve robustness of Linter CI by caching pre-commit installation. (#1704)
Simulation model#
Addition of FADC long-sum model parameters. (#1659)
v0.18.0 - 2025-07-08#
Bugfixes#
Documentation#
Reorganisation and improvement of the documentation in the GitHub pages. (#1611)
New Features#
Use CTAO telescope names when writing out telescope list in reduced event data tables. (#1616)
Allow CORSIKA limits to use layouts defined in simulation models database. (#1619)
Add application / API to plot tabulated model parameters using default plotting configurations defined in model parameters schemas. (#1628)
Allow to use current prod6 simulations for reduced event data generation. (#1631)
Add common array elements ID. Add list with triggered common telescope IDs to reduced event data tables. (#1637)
Maintenance#
Removed unused ‘critical’ log level. (#1604)
Add JSON checker to pre-commit linter. (#1615)
Update test files with proton simulations (simtel output) and add provenance information. (#1617)
Improve linter CI to print file containing non-ascii characters. (#1620)
Replace file lists
tests/resources/simtel_output_files.txtused for testing by explicitly listing of files. (#1629)Add default plotting configuration for tabular data. Introduce consistent lower-cases for keys in plotting configuration schema. (#1635)
0.17.0 - 2025-06-10#
Bugfixes#
Documentation#
New Features#
Add pixel layout plotting. (#1541)
Enable docker-image generation for vector-optimized CORSIKA 7.8. (#1550)
Add the sequential option to simulations using multipipe. With this option the CORSIKA and sim_telarray instances run in sequential order as far as possible. This mode is useful particularly on the grid, where typically we request a single core per job. (#1561)
Add reading and conversion of primary_id from sim_telarray output. (#1570)
Update
simtools-generate-simtel-event-datato use astropy tables. (#1573)Generalize table reading and writing. Add application to merge tables. (#1580)
Write reduced event data file as part of the cleanup stage of the simtools-simulate-prod application. (#1588)
Use patches for optimized CORSIKA from GitLab repository instead of cloud download. (#1589)
New utility simtools-verify-simulation-model-production-tables to verify completeness of model parameter repository. (#1592)
Add custom copilot instructions to support AI used by developers. (#1594)
Add new application
simtools-submit-array-layoutsto submit new array-layouts definition and validate the telescopes listed. (#1596)Improve layout plotting; adding plotting of background layout. Add plotting of all layouts. (#1597)
Improve printout for assertion error for testing. (#1598)
Add production version and build options to sim_telarrary metadata. (#1601)
Maintenance#
Add unit tests to increase coverage in interpolation handler. (#1504)
Docker file improvements: linter application; bugfix in extra flags for optimized CORSIKA compilation. (#1544)
Include stdout and stderr in error message of integration test. (#1555)
Add pytest-retry package for ‘flaky test’. (#1558)
Reduce number of log-messages (INFO) for uploaded model parameters to DB. (#1559)
Replace the “run_number_start” argument with “run_number_offset” (clearer) and re-introduce the “run_number” argument to allow specifying a run number (to which the offset is added). (#1562)
Use telescope position files for setup in illuminator application and set correct obs level. (#1566)
Improve the names of the various simulation output files. They now follow the current convention in current productions, they contain a bit more information (e.g., the model version where it wasn’t there before), and they are more consistent with each other. (#1568)
Improved interface and interpolation routines to derive production configuration. This creates a pipeline for lookup table generation, production grid generation, creating a grid with limits, which is then further complemented by the production statistics per grid point. (#1569)
Fix unit test to run with astropy 7.1 (and earlier versions) (see #1574). (#1575)
Simplified Sonar setup by moving properties into GitHub workflow. (#1578)
Introduce simplified Codecov configuration. (#1579)
Copy the CORSIKA log in binary mode instead of text mode. (#1582)
Protect against non-unicode characters in CORSIKA log file. (#1583)
Remove obsolete code on submission systems in JobManager. (#1586)
Change dummy trigger thresholds to easier-to-spot values. (#1591)
Fix duplication of some sim_telarray test files in ‘./tests/resources’ and corresponding test fixtures. (#1593)
v0.16.0 - 2025-05-05#
Documentation#
New Features#
Add grid generation production application to generate the grid points for Corsika simulations, with energy thresholds, viewcone and radius interpolation. (#1482)
Add support for multiple versions and the multipipe functionality of sim_telarray (#1496)
Add validation metadata written by sim_telarray with array model parameters. (#1505)
Improve robustness of sim_telarray configuration by adding an invalid (dummy) telescope configuration. (#1509)
Adding functionality for producing simulation configuration reports. (#1510)
Add random seeds from file for instrument configuration. (#1511)
Adapt Docker files and building workflows to CORSIKA 7.8000. (#1512)
Add sim_telarray random seed for instrument configuration to metadata (and metadata tests). (#1515)
Integration test with multipipe and support for packing all output files in grid productions. (#1520)
Add versioning to model parameter schema files. Implement example for
corsika_starting_grammage. (#1523)Added an application to produce calibration device reports. (#1530)
Allow
corsika_starting_grammageto be telescope-type dependent. (#1539)
Maintenance#
Separate TelescopeModel and SiteModel for improved readability in
sim_telarrayconfiguration writer. (#1489)Fix licence definition in pyproject.toml to follow PEP 639. (#1493)
Improve consistency of spelling of
sim_telarray. (#1494)Remove test condition in simulator light emission and mock configuration file access. (#1495)
Refactoring of event scaling application and moving most of the functionality into a new module. (#1498)
Renaming of event scaling methods to production statistics derivation methods. (#1502)
Improved naming and docstrings in simulation runners. (#1519)
Remove duplication of reading yaml/JSON files in simtools.utils.general. (#1522)
Remove duplication in
make_run_commandincorsika_simtel_runnerandsimulator_array. (#1525)Collect pyeventio related functions in new module
simtel_io_file_info. (#1531)Add new field model_parameter_schema_version to model parameter test files. (#1535)
Simulation model#
Change list of applicable array elements and class for
pedestal_eventsmodel parameter. (#1490)
v0.15.0 - 2025-04-04#
Maintenance#
Skip testing of DB upload for simpipe integration tests. (#1488)
0.14.0 - 2025-04-03#
Bugfixes#
Fix integration tests uploading values / files to sandbox DB. (#1453)
Fix building of production images from branches other than
main. (#1467)Add missing
parameterdefinition in model parameter schema. (#1470)Primary degraded map is a parameter applied for all telescope types. (#1475)
Fixes a bug in the selection of the triggered events by telescope IDs for the limits calculation. (#1483)
Fix how integration tests are marked for use cases (used in SimPipe AIV). (#1484)
New Features#
Added functionality to automate report generation for array elements and parameters. (#1436)
Add full set of array pointings to reduced mc event data file if available in the mc header. (#1439)
Add fitting of afterpulse spectrum to derive single pe application. (#1446)
Run scheduled nightly integration and unit tests on production DB. (#1454)
Generalize extraction of MC event tree from sim_telarrary files; generic writer and reader. (#1458)
Add functionality to generate reports for site-specific parameters. (#1459)
Allow to skip integration tests for production DB. (#1460)
Add an application to print the versions of simtools, the DB, sim_telarray, and CORSIKA (
simtools-print-version). (#1461)Use files to pass filepaths for simtel event data files and telescope IDs in corsika limits derivation tool. (#1464)
Retrieve CTAO layout definitions. Merge with model parameter value and write to disk. (#1465)
Add tests to ensure that model table files uploaded to DB follow Unicode. (#1473)
Add writing of metadata into sim_telarray configuration files. (#1474)
Update model parameter definitions to reflect those from most recent sim_telarray release. (#1476)
Maintenance#
Remove Docker prod image generation; documentation update and improved workflow naming. (#1447)
Change requirement of arguments in event scaling application and improve documentation. (#1449)
Refactor light emission application and move logic to helper functions. (#1450)
Use consistently numpy dtypes for model parameters. (#1451)
Remove h5py dependency. (#1456)
Store arrays in model parameters as arrays and not as strings. (#1466)
0.13.0 - 2025-03-19#
Bugfixes#
Fix
nullvalues in YAML-style model parameter schema files. (#1437)
New Features#
Improvements for single pe setting workflow including plotting and metadata handling. (#1398)
Add an application that allows to extract array and shower data and to save them in hdf5 format for the later calculation of production configuration limits. (#1402)
Add integration tests to production image generation. (#1420)
Improve reading of single parameter and files from database. Add plotting of tabular data using
parameter_version. (#1426)Improved plotting of tabular data; adding error bars and schema validation for plotting configuration. (#1429)
Add function to get all model parameter data for all model versions. (#1432)
Maintenance#
Run unit and integration test on database setup in local test environment. (#1424)
Remove tests from pypi package. Remove obsolete
__init__.pyfiles. (#1430)Change base python version from 3.11 to 3.12. (#1433)
Update exporting of sim_telarray model to simtools taking into account the updated file naming (included parameter versions). (#1438)
Robust getting of username in metadata collector in case no user is defined on system level. (#1442)
0.12.0 - 2025-03-11#
Bugfixes#
0.11.0 - 2025-03-05#
Bugfixes#
New Features#
Adding two new applications for generating model parameter documentation reports for a given array element either for a given production, or across multiple productions. (#1185)
Add building of optimized corsika binaries (e.g., avx2, avx512) and building of images for different corsika versions. (#1355)
Add a simtool to derive limits for energy, viewcone and radius for a simulation. (#1356)
Add generic
simtools-run-applicationto run one or several simtools using a single configuration file. (#1379)Add reading of database collections from model parameter schemas. (#1380)
Add module for simtools dependency management. Allows to retrieve e.g. sim_telarray and CORSIKA versions. (#1383)
Expanded functionality of application configuration workflows based on the integration test configurations. (#1389)
Set user in Docker images and add possibility to add user information from env/command line. (#1400)
Allow to mark integration tests in their use case config with a corresponding CTAO use case or requirement ID. (#1405)
Run check for changelog entry only on PRs ready for review. (#1416)
Maintenance#
Maintenance in production configuration removing not needed flags and only use metric files. (#1376)
Rename derive-limits application to production-derive-limits. (#1378)
Add YAML file with build opts into CORSIKA/sim_telarray images. (#1382)
Add unit tests for version.py. (#1384)
Improve function naming and unit test coverage in
simtools.utils.names. (#1390)Change checks for software updates from weekly to monthly. (#1394)
Remove duplication in converting sim_telarray string-lists (values, units) to python lists. (#1395)
Improve reading of design models; refactoring and improved docstrings for
simtools.utils.names. (#1396)
Simulation model#
Introduce MSTx-NectarCam and MSTx-FlashCam design models for MSTs (allow both sites). (#1362)
0.10.0 - 2025-02-17#
Bugfixes#
Schema validation bugfixes. (#1330)
New Features#
Add a simtool to derive (normalize) the single p.e. amplitude spectrum using the
sim_telarraytoolnorm_spe. (#1299)Add validation of simtools-generated configuration files for sim_telarray with reference files. (#1322)
Add schema for fake mirror list. (#1338)
Add additional queries to db handler to retrieve model versions etc. (#1349)
Add fixed database version for testing (instead of
LATEST). (#1365)
Maintenance#
0.9.0 - 2025-01-22#
Bugfixes#
New Features#
Add an application to plot tables from a file (from file system or for a model parameter file downloaded from the DB). (#1267)
Enhancements to file testing within integration tests. (#1279)
Add a tool to prepare job submission with
simulate_prodfor a HTCondor system. (#1290)Remove functionality in
db_handlerto read from simulation model repository. (#1306)
Maintenance#
0.8.2 - 2024-12-03#
Maintenance#
Move simtools package layout to src-layout. (#1264)
v0.8.1 - 2024-11-26#
Bugfixes#
Fix authentication and writing permits for deployment of simtools to pypi. (#1256)
v0.8.0 - 2024-11-26#
API Changes#
Bugfixes#
Fix triggering of telescopes in light emission package. (#1128)