Programming interfaces#

The format is available in C++, Python and C. Record types and fields are in Photon-stream format.

C++#

#include "obdeect/stream/stream.hpp"

#include <fstream>

std::ofstream file("run.pst", std::ios::binary);
obdeect::stream::Writer writer(file, "my-module");           // writes the stream record
obdeect::stream::PhotonChunk chunk;
chunk.telescope_id = 1;
chunk.frame = obdeect::stream::Frame::telescope;
chunk.resize(3, obdeect::stream::required_columns);          // x y cx cy t w
chunk[obdeect::stream::Column::x] = {0.0F, 1.0F, 2.0F};      // fill the columns, then:
writer.write_chunk(chunk);
writer.finish();                                      // writes stream_end

std::ifstream in("run.pst", std::ios::binary);
obdeect::stream::Reader reader(in);
obdeect::stream::Record record;
while (reader.next(record)) {
  if (record.type == obdeect::stream::RecordType::photons) {
    // record.chunk.rows(), record.chunk[obdeect::stream::Column::w], ...
  }
}

Python#

import numpy as np
import obdeect.stream

with obdeect.stream.Writer("run.pst", "my-module") as writer:
    writer.write("run", {"run_number": 1, "telescope_x_m": [0.0, 10.0]})
    writer.write_photons(
        1,
        "telescope",                                  # or 2
        0,
        {"x": np.zeros(3, np.float32), "y": np.zeros(3, np.float32),
         "cx": np.zeros(3, np.float32), "cy": np.zeros(3, np.float32),
         "t": np.zeros(3, np.float32), "w": np.ones(3, np.float32)},
    )
    writer.write_blob(1204, 0, b"...")                # raw EventIO block

for record in obdeect.stream.Reader("run.pst"):
    print(record["type"])                             # stream, run, photons, blob, ...
    if record["type"] == "photons":
        w = record["columns"]["w"]                    # numpy float32 array

Metadata values: int, float, str, or a list or array of floats. Photon columns must be float32 arrays. Names: obdeect.stream.COLUMNS, obdeect.stream.FRAMES. Reader("-") reads stdin. Reader(path, photon_telescopes) with a list of telescope ids skips the photons of all other telescopes without decoding them: their records come back with columns empty.

The module also provides obdeect.stream.corsika_to_stream, stream_to_simtel and stream_to_grisu, the Python counterparts of the commands.

CMake package#

cmake --install puts the headers and a CMake package in the install prefix. A wheel carries the same files under obdeect/stream/include and obdeect/stream/cmake:

$ python -m obdeect.stream --cmake-dir      # directory of obdeect_stream-config.cmake
$ python -m obdeect.stream --include-dir    # directory of the obdeect/stream/ headers

A consumer links the header-only core:

find_package(obdeect stream 0.1 CONFIG REQUIRED)   # SameMinorVersion: any 0.1.x
target_link_libraries(<target> PRIVATE obdeect::stream)

Configure with -Dobdeect_stream_DIR=$(python -m obdeect.stream --cmake-dir), or with -Dobdeect_stream_DIR=<prefix>/lib/cmake/obdeect_stream for a cmake --install prefix. Other consumers and build options: Installation.

The version comes from the git tag (setuptools-scm, passed by scikit-build-core as SKBUILD_PROJECT_VERSION). Other builds take the newest v* tag, or 0.0.0 without one.

Other languages and custom stages#

libobdeect_stream_c exposes the reader and writer through a plain C ABI (obdeect/stream/obdeect_stream.h, obdeect::stream_c in the CMake package). Rust (bindgen or extern "C"), Julia (ccall) and Go (cgo) can use it. Returned pointers, including photon columns, belong to the reader and are valid until the next obdeect_stream_reader_next(); nothing is copied. Errors are negative codes with a message in obdeect_stream_last_error().

The optics step, which turns ground-frame photons into camera-frame photons, is obdeect ray-tracing trace-stream (Photon streams). A Python ray tracer uses the same stage interface:

import numpy as np
from obdeect.stream.stage import transform

def trace(telescope_id, columns, metadata):
    # columns: x, y, cx, cy, t, w, ... as float32 arrays (ground frame, metres)
    x, y = columns["x"], columns["y"]
    hit = x**2 + y**2 < 12.0**2                  # your optics here
    out = {name: values[hit] for name, values in columns.items()}
    out["x"], out["y"] = out["x"] * 0.01, out["y"] * 0.01   # focal plane
    return out                                   # float32 arrays of equal length

transform("ground.pst", "camera.pst", trace, stage_id=0x444C0010, frame_out="camera",
          efficiency_status={"mirror": {"state": "applied", "source": "my-model-v1"}})

transform copies all other records unchanged (records of unknown type are dropped), sets the frame of the output chunks and telescope records, and appends one efficiency_status record per run. It refuses a factor that is applied in any input efficiency status.

The same in C (error handling shortened):

#include <obdeect/stream/obdeect_stream.h>

obdeect_stream_reader *in;  obdeect_stream_writer *out;
obdeect_stream_reader_open("ground.pst", &in);
obdeect_stream_writer_open("camera.pst", "my-tracer", &out);
int type;
while (obdeect_stream_reader_next(in, &type) == OBDEECT_STREAM_OK) {
  if (type == OBDEECT_STREAM_RECORD_PHOTONS) {
    uint32_t telescope, rows;  uint64_t first;
    obdeect_stream_reader_photons(in, &telescope, NULL, &first, &rows, NULL);
    const float *columns[OBDEECT_STREAM_N_COLUMNS];
    for (int c = 0; c < OBDEECT_STREAM_N_COLUMNS; ++c)
      columns[c] = obdeect_stream_reader_column(in, c);        /* NULL if absent; trace here */
    obdeect_stream_writer_write_photons(out, telescope, OBDEECT_STREAM_FRAME_CAMERA, first, rows, columns);
  }                       /* copy other records with obdeect_stream_reader_meta_* / write_record */
}                         /* the loop ends with OBDEECT_STREAM_END, anything else is an error */
obdeect_stream_writer_close(out);  obdeect_stream_reader_close(in);

Link with -L$(python -m obdeect.stream --lib-dir) -lobdeect_stream_c, or in CMake with target_link_libraries(tracer PRIVATE obdeect::stream_c).

Conformance streams#

cpp/tests/data/conformance/ holds valid.pst (every record type, plus a record of unknown type that must be skipped), four invalid streams (bad magic, unknown major version, truncated chunk, more than 2^16 rows) and expected.json with the decoded content and the expected error. make_conformance.py regenerates them with the standard library only. An independent implementation of Photon-stream format should decode valid.pst to expected.json and reject the others.

End-to-end check#

The end-to-end check of the EventIO codec lives in the obdeect-validation repository: case cases/stream/corsika-roundtrip.json (suite stream). In the ghcr.io/gammasim/simtools-dev container it runs obdeect stream from-corsika | obdeect stream to-simtel on a CORSIKA IACT file, compares bunches and sim_telarray camera images with the direct run, and checks the stdin/tee variants. It needs CORSIKA and sim_telarray and is not part of the automated tests.