A masterless, fault-tolerant, high-performance distributed computation system for Clojure.
This is a community-maintained fork of onyx-platform/onyx (now archived), updated to run on modern JDKs.
Add to your project.clj:
[com.github.clojure-finance/onyx "0.16.0"]Clojars also lists 0.1.0 and 0.2.0 from the fork's earlier version scheme. 0.2.0 has the same content as 0.15.0; prefer 0.15.0 or later.
JDK 17 or newer is required (tested on JDK 21). Onyx uses Aeron 1.53.x for messaging, and the Agrona library bundled with it reads jdk.internal.misc.Unsafe, so the JVM that starts peers needs this flag:
--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
In Leiningen:
:jvm-opts ["--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"]Without this flag, the embedded media driver fails with:
IllegalAccessError: class org.agrona.UnsafeApi ... cannot access class jdk.internal.misc.Unsafe
On JDK 24 and newer, also pass --enable-native-access=ALL-UNNAMED. Otherwise the JVM warns that nippy's LZ4 compression calls a restricted method (System.load) that will be blocked in a future release. The flag exists since JDK 17, so it is safe to add unconditionally.
Onyx is a distributed computation system written in pure Clojure. It features:
- Masterless architecture — no single point of failure
- Cloud-scale fault tolerance and high performance
- Hybrid batch/stream processing model
- Information model for describing and constructing distributed workflows
It occupies the same space as Storm, Flink, Spark and similar systems.
- Real-time event stream processing
- Continuous computation and CQRS
- Extract, transform, load (ETL) pipelines
- Data transformation (map-reduce style)
- Data ingestion between storage systems
- Data cleaning and normalization
0.16.0 (Latest)
- ZooKeeper 3.9.5, Curator 5.9.0, slf4j 1.7.36
- ZooKeeper now logs via slf4j. Onyx ships
slf4j-nop, so add your own binding to see its logs.
0.15.0
- Aeron upgraded from 1.21.2 to 1.53.1
- JDK 17+ required (the previous
sun.nio.chandjava.langopens are no longer needed) - The coordination log version is now
0.15.0. It had stayed at0.14.6-SNAPSHOT, which silently skipped the cluster version compatibility check, so peers of different Onyx versions now refuse to share a tenancy. Use a new:onyx/tenancy-idwhen upgrading.
Earlier fork work (2023, before 0.15.0)
- Clojure 1.11, core.async 1.6.673, ZooKeeper 3.4.12 → 3.5.5, Curator 4.0.1 → 5.5.0
onyx.static.uuid/random-uuidrenamed toonyx-random-uuidto avoid clashing withclojure.core/random-uuid(Clojure 1.11)
See changes.md for the full changelog including upstream history.
Note: The upstream project is archived, so some external links may be outdated.
- User Guide — comprehensive documentation (sources in
doc/user-guide) - API reference — generated HTML is in
doc/api; the public API lives insrc/onyx/api.clj - Data model reference — every job, catalog, lifecycle and peer-config key is described in
src/onyx/information_model.cljc - Onyx Starter — quick start project with walkthrough
- Onyx Template — Leiningen application template
Built into Onyx core:
Official plugins from upstream (may require updates for this fork):
Plugin template for building your own: onyx-plugin
Install into your local Maven repository:
lein installRun the test suite:
lein testThe default selector skips tests tagged ^:broken or ^:stress. Run namespaces one at a time if you encounter issues — embedded ZooKeeper and the Aeron directory under /dev/shm can collide across parallel runs.
Onyx was created by Michael Drogalis and developed by the upstream contributors.
Some code incorporated from:
Copyright © 2017 Michael Drogalis
Distributed under the Eclipse Public License, the same as Clojure.