A C++20 binary event recorder built around a bounded producer-consumer queue, a dedicated background writer, an integrity-checked binary format, replay/export tools and an end-to-end benchmark.
The project behaves like a small software black box: several producer threads generate or submit technical events, AsyncRecorder applies backpressure through a bounded queue, and a single writer thread serializes the accepted events to a .hpblr file. Companion tools can inspect, filter, dump and export the recorded stream.
- modern C++20 with RAII,
std::jthread, move semantics and exception-safe shutdown; - producer-consumer concurrency with
std::mutex,std::condition_variableand atomics; - explicit little-endian binary serialization independent of native structure layout;
- CRC-32 validation for the file header, record headers and payloads;
- bounded-memory backpressure rather than an unbounded logging queue;
- replay with corruption and truncation detection;
- CSV and JSON export from validated records;
- reproducible CMake/CTest builds and end-to-end benchmark tooling.
.
├── apps/ # hpblr_record, hpblr_tool and hpblr_bench
├── docs/ # Format, robustness, design and benchmark documentation
├── include/hpblr/ # Public C++ headers
├── scripts/ # End-to-end demo script
├── src/core/ # Core library implementation
└── tests/ # Unit and integration-style core tests
- CMake 3.20 or newer;
- a C++20 compiler and standard library with
std::jthreadsupport; - a normal filesystem with binary file I/O support;
- Bash only for
scripts/demo.sh.
The project is developed as a Linux-oriented CLI project. The core C++ code does not depend on POSIX-only headers.
From the repository root:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failureThe three executables are generated directly in build/:
build/hpblr_record
build/hpblr_tool
build/hpblr_bench
GCC/Clang targets are built with -Wall -Wextra -Wpedantic -Wconversion -Wshadow. The following configuration additionally treats those warnings as errors:
cmake -S . -B build-werror \
-DCMAKE_BUILD_TYPE=Release \
-DHPBLR_WARNINGS_AS_ERRORS=ON
cmake --build build-werror --parallel
ctest --test-dir build-werror --output-on-failureTo build only the library and CLI applications without the test target:
cmake -S . -B build -DHPBLR_BUILD_TESTS=OFF
cmake --build build --parallel./build/hpblr_record \
--output sample.hpblr \
--producers 4 \
--events 50000 \
--payload-size 64./build/hpblr_tool inspect --input sample.hpblrThe command validates the complete file while reporting its version, creation time, event count, payload volume, timestamp range and severity distribution.
./build/hpblr_tool dump \
--input sample.hpblr \
--severity warning \
--limit 5./build/hpblr_tool export \
--input sample.hpblr \
--format csv \
--producer 2 \
--output producer2.csv./build/hpblr_tool export \
--input sample.hpblr \
--format json \
--severity error \
--output errors.json./build/hpblr_bench \
--output bench.hpblr \
--report bench_report.json \
--producers 8 \
--events 1000000 \
--payload-size 128Generates synthetic events from several producer threads and persists them through AsyncRecorder.
--output <file> Output .hpblr file (default: recording.hpblr)
--producers <n> Number of producer threads (default: 4)
--events <n> Total events; 0 means run until SIGINT/SIGTERM
--payload-size <bytes> Payload size per event, maximum 16777216
--queue-capacity <n> Bounded queue capacity (default: 8192)
--flush-bytes <n> Writer flush threshold (default: 1048576)
--sleep-us <n> Optional producer pacing delay in microseconds
--log-file <file> Append diagnostics to a text file
When the queue is full, submit() blocks until the writer frees capacity. On shutdown, accepted events are drained and the output file is closed before the writer thread is joined. --output and --log-file are rejected if they resolve to the same file.
Reads .hpblr files and validates their structure and CRCs during replay.
inspect --input <file>
dump --input <file> [filters] [--limit <n>]
export --input <file> --format csv|json [--output <file>] [filters] [--limit <n>]
Available replay filters:
--producer <id>
--type <id>
--severity trace|debug|info|warning|error|critical
--from-ns <timestamp>
--to-ns <timestamp>
--from-ns and --to-ns are inclusive Unix timestamps expressed in nanoseconds. File export refuses an --output path that resolves to the same file as --input, preventing accidental truncation of the recording.
Measures the complete generation -> queue -> serialization -> filesystem path.
--output <file> Output .hpblr file (default: benchmark.hpblr)
--report <file> Write the benchmark report as JSON
--json Print the JSON report to stdout
--producers <n> Producer threads (default: 4)
--events <n> Total events, must be greater than 0
--payload-size <bytes> Payload size per event, maximum 16777216
--queue-capacity <n> Queue capacity (default: 32768)
--flush-bytes <n> Writer flush threshold (default: 4194304)
The benchmark reports written events, duration, events/s, file size and MiB/s. --output and --report must resolve to different files. Results should only be compared with equivalent hardware, storage and build settings.
The .hpblr format is documented in docs/BINARY_FORMAT.md. Important properties are:
- format version
1; - fixed-size 36-byte file header;
- fixed-size 48-byte record header;
- explicit little-endian integers;
- CRC-32 over the file header;
- CRC-32 over each record header with its CRC field zeroed;
- CRC-32 over each payload;
- maximum payload size of 16 MiB per record.
CRC-32 detects accidental corruption; it is not a cryptographic authenticity mechanism.
The test suite covers the queue, CRC implementation, binary round trips, truncated input, payload corruption, first-timestamp edge cases, invalid event severity, concurrent asynchronous recording, logger synchronization and end-to-end CLI smoke scenarios including destructive path-collision protection.
Run it with:
ctest --test-dir build --output-on-failureThe reader is intentionally fail-fast: malformed metadata, unsupported versions, oversized payloads, truncation and CRC mismatches produce explicit exceptions instead of attempting heuristic recovery.
The asynchronous writer stores failures from its background thread and rethrows them from synchronous control paths such as stop(). Applications that need to observe final persistence errors should therefore call stop() explicitly rather than relying only on destructor cleanup.
This command configures a separate build tree, runs the tests, records a sample, inspects it, exports CSV and runs a small benchmark:
bash scripts/demo.shGenerated demo artifacts are written to demo-output/.
docs/TECHNICAL_DESIGN.md— architecture and threading model;docs/BINARY_FORMAT.md— byte-level file format;docs/ROBUSTNESS.md— failure handling and shutdown behavior;docs/BENCHMARKING.md— benchmark methodology and suggested experiments.
See LICENSE.