A high-performance rules engine for IFTTT-style automation in Rust with zero-overhead JSONLogic evaluation.
Two chained rules in the visual debugger: IF the condition matches, THEN the next rule runs. Try it live in your browser →
Dataflow-rs is a lightweight, embeddable rules engine that lets you define IF → THEN → THAT automation in JSON. The engine compiles every rule's JSONLogic once at startup, so evaluation does no parsing at runtime, and actions execute asynchronously. Use it to route events, validate data, or build multi-step automation pipelines.
Dataflow-rs targets high-throughput hot paths. The engine compiles all JSONLogic expressions once at startup, so runtime evaluation runs with zero allocations, zero parsing overhead, and predictable latency.
A multi-threaded benchmark (1,000,000 concurrent events) on a 10-core Apple M2 Pro yields:
- Throughput: ~630,000 messages/sec
- Median (P50) Latency: 6 μs
- Tail (P99) Latency: 52 μs
- Tail (P99.9) Latency: 94 μs
The same rule definitions run across your stack, not only in backend services:
- Rust Backend: Run natively with maximum speed and concurrency using
dataflow-rs. - Browser & Edge: Run client-side validations or edge routing using WebAssembly bindings via @goplasmatic/dataflow-wasm.
- React UI Admin Portal: Let your team and your users visualize, edit, and step-by-step debug rules using @goplasmatic/dataflow-ui.
┌─────────────────────────────────────────────────────────────────┐
│ Rule (Workflow) │
│ │
│ IF condition matches → JSONLogic against any field │
│ THEN execute actions (tasks) → map, validate, custom logic │
│ THAT chain more rules → priority-ordered execution │
└─────────────────────────────────────────────────────────────────┘
Example: IF order.total > 1000 THEN apply_discount AND notify_manager
| Rules Engine | Workflow Engine | Description |
|---|---|---|
| Rule | Workflow | A condition + actions bundle: IF condition THEN execute actions |
| Action | Task | An individual processing step (map, validate, or custom function) |
| RulesEngine | Engine | Evaluates rules against messages and executes matching actions |
Both naming conventions work; use whichever fits your mental model.
If you need dynamic business rules or user-customizable workflows, writing manual if/else checks makes your code rigid, while running full orchestrators (like Temporal or Zeebe) adds heavy infrastructure overhead and milliseconds of network latency. Dataflow-rs sits between the two:
| Capability | Hardcoded Rust | dataflow-rs | Heavy Orchestrators (Temporal/Zeebe) |
|---|---|---|---|
| Hot Reload Rules | Recompile & redeploy | Instant JSON update | Deploy new worker code |
| Execution Overhead | None | Zero (pre-compiled JSONLogic) | DB reads/writes (tens of ms) |
| Browser Execution | Compile full app to WASM | Run same rules in JS via WASM | Network round-trip required |
| Visual Debugger | Build your own UI | Included React UI components | Included dashboard |
| Infrastructure | None | None (embeddable library) | Requires server clusters & DBs |
[dependencies]
dataflow-rs = "3.14"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
serde_json = "1.0"JSONLogic's extended operator families are opt-in; the default build ships core JSONLogic only. Add the families your rules use:
[dependencies]
dataflow-rs = { version = "3.14", features = ["ext-string", "ext-control"] }Read JSONLogic → Operator Families first: enabling a family can change how an existing rule behaves.
A message arrives with its body in payload. The engine evaluates conditions
and mappings against data, so the first rule loads the payload into data,
and the second rule acts on it. Chaining, the THAT in IF → THEN → THAT,
means rules run in order and each one sees what the previous rules wrote.
{
"id": "order_intake",
"name": "Order Intake",
"tasks": [
{
"id": "load_order",
"name": "Load payload into data.order",
"function": {
"name": "parse_json",
"input": {"source": "payload", "target": "order"}
}
}
]
}{
"id": "premium_order",
"name": "Premium Order Processing",
"condition": {">=": [{"var": "data.order.total"}, 1000]},
"tasks": [
{
"id": "apply_discount",
"name": "Apply Premium Discount",
"function": {
"name": "map",
"input": {
"mappings": [
{
"path": "data.order.discount",
"logic": {"*": [{"var": "data.order.total"}, 0.1]}
},
{
"path": "data.order.final_total",
"logic": {"-": [{"var": "data.order.total"}, {"*": [{"var": "data.order.total"}, 0.1]}]}
}
]
}
}
}
]
}The engine evaluates a rule's condition before any of its own tasks run. A condition can only read what earlier rules produced, never what its own tasks are about to write. The parse therefore lives in its own rule here rather than as a first task on
premium_order.
use dataflow_rs::{Engine, Workflow};
use dataflow_rs::engine::message::Message;
use serde_json::json;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Rule 1 from Step 2 — always runs, moves the payload into `data.order`.
let intake = Workflow::from_json(r#"{
"id": "order_intake",
"name": "Order Intake",
"tasks": [
{
"id": "load_order",
"name": "Load payload into data.order",
"function": {
"name": "parse_json",
"input": {"source": "payload", "target": "order"}
}
}
]
}"#)?;
// Rule 2 from Step 2 — runs only when the condition matches.
let premium = Workflow::from_json(r#"{
"id": "premium_order",
"name": "Premium Order Processing",
"condition": {">=": [{"var": "data.order.total"}, 1000]},
"tasks": [
{
"id": "apply_discount",
"name": "Apply Premium Discount",
"function": {
"name": "map",
"input": {
"mappings": [
{
"path": "data.order.discount",
"logic": {"*": [{"var": "data.order.total"}, 0.1]}
},
{
"path": "data.order.final_total",
"logic": {"-": [{"var": "data.order.total"}, {"*": [{"var": "data.order.total"}, 0.1]}]}
}
]
}
}
}
]
}"#)?;
// Create engine — all JSONLogic compiled once here
let engine = Engine::builder()
.with_workflows(vec![intake, premium])
.build()?;
// Process a message. `from_value` sets the *payload*; `parse_json` in the
// first rule is what lands it in `data`.
let mut message = Message::from_value(&json!({"total": 1500}));
engine.process_message(&mut message).await?;
assert_eq!(message.data()["order"]["discount"].as_f64(), Some(150.0));
assert_eq!(message.data()["order"]["final_total"].as_f64(), Some(1350.0));
println!("Discount: {}", message.data()["order"]["discount"]); // 150
println!("Final Total: {}", message.data()["order"]["final_total"]); // 1350
Ok(())
}process_message reports errors through two channels:
Result::Errsignals that the engine stopped early (a task failed withoutcontinue_on_error, or an engine-level error occurred).message.errors()always contains every error encountered, including errors from tasks that ran withcontinue_on_error = trueand so didn't short-circuit the workflow.
A short-circuit ? surfaces only the first kind. To cover both, check each:
use dataflow_rs::{Engine, Workflow};
use dataflow_rs::engine::message::Message;
use serde_json::json;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let engine = Engine::builder()
.with_workflow(Workflow::from_json(r#"{ ... }"#)?)
.build()?;
let mut message = Message::from_value(&json!({"order": {"total": 1500}}));
// `continue_on_error` tasks may record errors here without returning Err.
if let Err(e) = engine.process_message(&mut message).await {
eprintln!("engine halted: {e}");
}
// Always iterate `message.errors()` to see everything that went wrong.
for err in message.errors() {
eprintln!(
"[{workflow_id}/{task_id}] {msg}",
workflow_id = err.workflow_id.as_deref().unwrap_or("-"),
task_id = err.task_id.as_deref().unwrap_or("-"),
msg = err.message,
);
}
Ok(())
}Both channels above are host-side. A rule's condition can't reach them: the
JSONLogic context is data, metadata and temp_data, and the error list isn't
in it. Point the engine at a context path and it mirrors each failure's code
there as it happens:
let engine = Engine::builder()
.with_workflows(workflows)
.with_error_context_path("metadata.errors")
.build()?;Each record is {workflow_id, task_id, code, status}, so a later rule can treat
a transient failure differently from a permanent one:
{"in": [{"var": "metadata.errors.0.code"}, ["TIMEOUT_ERROR", "IO_ERROR"]]}The engine writes nothing there unless you call with_error_context_path.
Coverage matches message.errors(): handler Errs, 5xx outcomes, every failing
validation rule, and anything a handler records itself. The records
deliberately leave out the error text and the operator-only detail, since the
context is serialized back to callers. See the
error handling guide
for the full rules.
The built-in error variants describe engine concerns: a missing function, a failed condition, a bad path. When a handler fails for a reason only your service understands (a circuit breaker opened, a tenant hit a rate limit), classify it yourself instead of inventing a parallel error channel:
use dataflow_rs::DataflowError;
DataflowError::service("circuit_open", "upstream unavailable")
.detail("connector 'billing' breaker open since 12:04")
.retryable(true)
.build()kind becomes ErrorInfo::code verbatim (not upper-cased, so the string your
service writes is the string it switches on). detail is an operator-only field
that Display never renders, so to_string() stays safe for an untrusted caller.
retryable is declared rather than inferred from the variant. The engine
never interprets any of it: continue_on_error, the audit entry, and the
Result::Err short-circuit behave exactly as for any other error.
use dataflow_rs::{RulesEngine, Rule, Action};
// These are type aliases — same types, rules-engine terminology
let rule = Rule::from_json(r#"{ ... }"#)?;
let engine = RulesEngine::builder().with_workflow(rule).build()?;- IF → THEN → THAT Model: Define rules with JSONLogic conditions, execute actions, chain with priority ordering.
- Zero Runtime Compilation: All JSONLogic expressions compiled once at startup, never per message.
- Full Context Access: Conditions can read any field in
data,metadata, andtemp_data. - Secrets Outside the Record:
{"secret": "name"}reads an engine-scoped store that no trace, snapshot or serialized message ever contains, andbuild()refuses a workflow that would copy one into the message. - Async-First Architecture: Native async/await on the Tokio runtime.
- Execution Tracing: Step-by-step debugging with message snapshots after each action, bounded by
TraceOptions(snapshot budget, redaction, timings-only mode) when you need it in production. - Always-On Observability: Attach an
ExecutionObserverfor per-task timing, including the sync built-ins a trace or a wrapped handler can't reach on their own. - Built-in Functions: Parse, Map, Validate, Filter, Log, and Publish.
- Pipeline Control Flow: Filter/gate function to halt workflows or skip tasks based on conditions.
- Rejecting Assertions:
halt_on: "failure"ends a rule once an action has run and failed. It is the gate avalidationneeds, sincecontinue_on_errorcovers only5xxandErr. The task keeps its own status (a400stays a400). - Channel Routing: Route messages to specific workflow channels with O(1) lookup.
- Traffic Splits: Roll a new workflow version out to a percentage of a channel's traffic with bucket-range routing.
- Workflow Lifecycle: Manage workflow status (active/paused/archived), versioning, and tagging.
- Hot Reload: Swap workflows at runtime without re-registering custom functions.
- Extensible: Add custom async actions by implementing the
AsyncFunctionHandlertrait, with typed config fields that are themselves JSONLogic (Template). - Typed Integration Configs: Pre-validated configs for HTTP, Enrich, and Kafka integrations, with
resolve_*helpers and anHttpMethodenum your client can convert directly. - Service-Classified Errors: Handlers attach their own error
kind,detail, andretryableviaDataflowError::Service, without a parallel error channel. - Branch on Why a Task Failed: Opt in with
with_error_context_pathand the engine mirrors each failure's code into the message context, so a downstream rule can route a timeout differently from a rejected request. - WebAssembly Support: Run rules in the browser with
@goplasmatic/dataflow-wasm. - React UI Components: Visualize and debug rules with
@goplasmatic/dataflow-ui. - Auditing: An audit trail of every change as data flows through the pipeline.
- All JSONLogic expressions compiled once when the Engine is created
- Compiled logic cached with Arc for zero-copy sharing
- Validates all expressions early, failing fast on errors
- Engine evaluates each rule's condition against the message context
- Matching rules execute their actions with pre-compiled logic (zero compilation overhead)
process_message()for normal execution,process_message_with_trace()for debugging- Each action can be async, enabling I/O operations without blocking
- Optionally attach an
ExecutionObserverfor always-on per-task timing, or callprocess_message_with_trace_options()for a bounded, redactable trace
On a 10-core Apple M2 Pro processing 1,000,000 messages concurrently (Tokio multi-threaded runtime, --release; per message: 1 parse + 6 mappings + 3 validations). Medians of 12 interleaved runs:
| Metric | Value |
|---|---|
| Throughput | ~630,000 msg/sec |
| Avg Latency | 10 μs |
| P50 Latency | 6 μs |
| P90 Latency | 19 μs |
| P95 Latency | 29 μs |
| P99 Latency | 52 μs |
| P99.9 Latency | 94 μs |
Why it's fast:
- Pre-Compilation: All JSONLogic compiled at startup, zero runtime parsing
- Arc-Wrapped Logic: Zero-copy sharing of compiled expressions across threads
- Arena Evaluation: Consecutive sync tasks evaluate against one bump-arena view of the context; map writes are spliced into it in place instead of re-cloning the written subtree
- Precomputed Paths: Mapping, parse, and publish target paths are split and interned at compile time, so the hot path never re-parses a path string
- Async I/O: Non-blocking operations for external services via Tokio
Tuning tip: if you never read audit trails, build messages with
Message::builder().capture_changes(false). Skipping the per-mapping
old/new value snapshots is the largest single saving in mapping-heavy
workloads; in a looping workflow it also stops every sweep's copies from
being held in memory until the run ends. See the performance guide for more.
Run the benchmarks and examples yourself:
cargo run --example benchmark --release # Full throughput + latency percentiles
cargo run --example realistic_benchmark --release # ISO 20022 -> SwiftMT-style workload
cargo run --example micro_aggregate_bench --release # Aggregate-heavy (reduce/map) workload
cargo run --example hello_world # Minimal getting-started example
cargo run --example rules_engine # IFTTT-style rules engine demo
cargo run --example complete_workflow # Parse → Transform → Validate pipeline
cargo run --example custom_function # Extending the engine with custom handlers
cargo run --example error_handling # Error handling patterns
cargo run --example async_migration # Typed Input + TaskContext + TaskOutcome handler shapeTargeted microbenchmarks for profiling a specific hot path. The micro_* ones
run a tight current_thread loop so the signal isn't buried under Tokio
scheduling; the last two measure throughput on a multi-threaded runtime:
cargo run --example micro_cond_bench --release # Condition-eval / trivially-true folding
cargo run --example micro_multiworkflow_bench --release # Chained workflows, per-workflow arena cost
cargo run --example micro_subtree_write_bench --release # Same-subtree map-write scaling
cargo run --example async_handler_benchmark --release # Marginal cost of one custom async handler
cargo run --example map_performance_test --release # Sequential map mappingsExtend the engine with your own async actions. Each handler declares a typed
Input (deserialized once at engine init), receives a TaskContext that
records audit-trail changes automatically, and returns a TaskOutcome:
use async_trait::async_trait;
use dataflow_rs::{AsyncFunctionHandler, Engine, Result, TaskContext, TaskOutcome};
use dataflow_rs::datavalue::OwnedDataValue;
use serde::Deserialize;
use serde_json::json;
/// Typed config for the handler — fails at `Engine::new()` if malformed,
/// not on first message.
#[derive(Deserialize)]
pub struct NotifyInput {
pub channel: String,
}
pub struct NotifyManager;
#[async_trait]
impl AsyncFunctionHandler for NotifyManager {
type Input = NotifyInput;
async fn execute(
&self,
ctx: &mut TaskContext<'_>,
input: &NotifyInput,
) -> Result<TaskOutcome> {
// Your custom async logic here (HTTP calls, DB writes, etc.)
ctx.set(
"data.notified_channel",
OwnedDataValue::from(&json!(input.channel)),
);
Ok(TaskOutcome::Success)
}
}
// Register handlers via the builder. `.register("name", h)` accepts any
// `AsyncFunctionHandler` and boxes it internally.
fn build(workflows: Vec<dataflow_rs::Workflow>) -> dataflow_rs::Result<Engine> {
Engine::builder()
.with_workflows(workflows)
.register("notify_manager", NotifyManager)
.build()
}Any config field may be authored as JSONLogic; since 3.9, every parameter of
every built-in works that way. Declare it as Template and compile it once
via the compile_input hook instead of hand-rolling the raw/compiled pair:
#[derive(Deserialize)]
struct GreetingInput {
// Authored in the workflow as JSONLogic: {"cat": ["hello, ", {"var": "data.name"}]}
greeting: Template,
}
impl AsyncFunctionHandler for GreetingHandler {
type Input = GreetingInput;
// Called once per task at build time, right after `parse_input`. The
// default is a no-op, so a handler with no `Template` field needs no override.
fn compile_input(input: &mut Self::Input, c: &TemplateCompiler) -> Result<()> {
input.greeting.compile(c, "greeting")
}
// ...
}A malformed expression fails at build time rather than on the first message that reaches the task, matching this crate's own stance for the built-ins.
One handler type registered under several names (a plugin host, say)
overrides the receiver-taking twins parse_input_with / compile_input_with
instead, so which field is a template can come from per-registration data. See
One handler type, several registrations.
A JSON literal is JSONLogic for itself, so the static spelling an author
already writes ("data.out", 5000) folds to a constant at build time and is
evaluated once, not per message. One catch: a single-key object whose key names
an operator is that operator, so write {"$cat": …} for the literal object. See
Literal keys and the $ escape.
| Function | Purpose | Modifies Data |
|---|---|---|
parse_json |
Parse JSON from payload into data context | Yes |
parse_xml |
Parse XML string into JSON data structure | Yes |
map |
Data transformation using JSONLogic | Yes |
validation |
Rule-based data validation | No (read-only) |
filter |
Pipeline control flow: halt workflow or skip task | No |
log |
Structured logging with JSONLogic expressions | No |
publish_json |
Serialize data to JSON string | Yes |
publish_xml |
Serialize data to XML string | Yes |
The filter function evaluates a JSONLogic condition and controls pipeline execution:
{
"function": {
"name": "filter",
"input": {
"condition": {"==": [{"var": "data.status"}, "active"]},
"on_reject": "halt"
}
}
}on_reject: "halt"stops the entire workflow when the condition is falseon_reject: "skip"skips only the current task and continues
The log function outputs structured log messages using the log crate:
{
"function": {
"name": "log",
"input": {
"level": "info",
"message": {"cat": ["Processing order ", {"var": "data.order.id"}]},
"fields": {
"total": {"var": "data.order.total"},
"user": {"var": "data.user.name"}
}
}
}
}Log levels: trace, debug, info, warn, error. Messages and fields support JSONLogic expressions.
Route messages to specific workflow channels for O(1) dispatch:
// Workflows define their channel
// { "id": "order_rule", "channel": "orders", "status": "active", ... }
// Process only workflows on a specific channel
engine.process_message_for_channel("orders", &mut message).await?;Only active workflows are included in channel routing. Workflows default to the "default" channel.
Workflows support lifecycle management fields:
{
"id": "my_rule",
"channel": "orders",
"version": 2,
"status": "active",
"tags": ["premium", "high-priority"],
"created_at": "2025-01-15T10:00:00Z",
"updated_at": "2025-06-01T14:30:00Z",
"tasks": [...]
}| Field | Type | Default | Description |
|---|---|---|---|
channel |
string | "default" |
Channel for message routing |
version |
number | 1 |
Workflow version |
status |
string | "active" |
active, paused, or archived |
tags |
array | [] |
Arbitrary tags for organization |
rollout |
object | null |
Traffic split: {bucket_start, bucket_end} over 0..100 |
created_at |
datetime | null |
Creation timestamp (ISO 8601) |
updated_at |
datetime | null |
Last update timestamp (ISO 8601) |
All fields are optional and backward-compatible with existing configurations.
Give a workflow a slice of its channel's traffic with a half-open bucket range
over 0..100, so a new version can roll out gradually alongside the old one:
{
"id": "checkout_v2",
"channel": "checkout",
"rollout": { "bucket_start": 0, "bucket_end": 10 },
"tasks": [...]
}That workflow serves buckets 0..=9, 10% of traffic. bucket_start is
inclusive and bucket_end exclusive, so a {"bucket_start": 10, "bucket_end": 100}
sibling covers the remaining 90% with no overlap and no gap.
The engine does not derive the bucket. Set it per message with your own policy (a sticky hash of a user id, a random draw, round-robin):
let message = Message::builder().routing_bucket(7).build();Every workflow, split or not, admits a message with no bucket, so every caller that predates rollouts keeps working unchanged. That includes the WASM entry points, which have no way to set one. The engine skips an excluded workflow exactly like a false condition: no audit entry, and the gate runs before any other per-message work.
A signing key or partner token has to be readable by a condition and must never
appear in a trace. Message.context cannot express that, because everything in
it is recorded. Secrets therefore live in a store on the engine, and expressions
read them through one reserved operator:
let engine = Engine::builder()
.with_secrets_json(&json!({ "webhook_token": std::env::var("WEBHOOK_TOKEN")? }))
.with_workflows(workflows)
.build()?;{ "condition": { "==": [ { "var": "metadata.headers.x-token" }, { "secret": "webhook_token" } ] } }The value never enters a Message, so it cannot appear in anything derived
from one. build() refuses a map mapping or log expression that reads a
secret (SECRET_IN_MESSAGE_WRITE), and a name the engine does not declare
(UNKNOWN_SECRET). Derived values such as an HMAC belong in a custom handler
reading the key through a Template. See the
Secrets page.
Swap workflows at runtime without losing custom function registrations:
let new_workflows = vec![Workflow::from_json(r#"{ ... }"#)?];
let new_engine = engine.with_new_workflows(new_workflows);
// Old engine remains valid for in-flight messagesThe default process_message_with_trace() snapshots the full message after
every step. That suits a step debugger, but the trace is unbounded in size and
quadratic in task count. process_message_with_trace_options bounds capture at
the only point it can be bounded: you set snapshot size, path redaction, and
audit-trail scope up front, rather than trimming the trace afterward.
let trace = engine
.process_message_with_trace_options(&mut message, TraceOptions::timings_only())
.await?;
for step in &trace.steps {
if let Some(us) = step.duration_us {
println!("{}/{:?} took {us}us", step.workflow_id, step.task_id);
}
}For always-on aggregation instead of a per-request trace, attach an
ExecutionObserver. It fires once per dispatched task, including the sync
built-ins (map, validation, filter, parse_*, publish_*, log). The
executor dispatches those internally, where a wrapped handler cannot reach them:
impl ExecutionObserver for Metrics {
fn task_finished(&self, event: &TaskEvent<'_>) {
// Must be cheap and non-blocking — runs synchronously on the executor.
}
}
let engine = Engine::builder()
.with_workflow(workflow)
.with_observer(Arc::new(Metrics::default()))
.build()?;With neither attached, tracing and observation overhead, clock reads included, stays out of the dispatch path.
Because every rule is plain JSON, the React UI can render it: JSONLogic expressions become readable flow diagrams, and the debugger steps through execution with a message diff after every task.
| Package | Description | Install |
|---|---|---|
| dataflow-rs | Async rules engine in Rust (this crate) | cargo add dataflow-rs |
| @goplasmatic/dataflow-wasm | WebAssembly bindings: run rules in the browser or Node.js | npm i @goplasmatic/dataflow-wasm |
| @goplasmatic/dataflow-ui | React components for rule visualization, editing, and step-by-step debugging | npm i @goplasmatic/dataflow-ui |
| datalogic-rs | JSONLogic compiler/evaluator used internally | cargo add datalogic-rs |
📖 Documentation: User Guide & API Reference · Interactive Playground · Visual Debugger
Contributions are welcome. To get started:
- Fork the repository and clone your fork
- Run tests:
cargo test --workspace --all-features(what CI runs) andcargo test -p dataflow-rs(the default feature set) - Make changes and add tests for any new features
- Run the benchmark before and after:
cargo run --example benchmark --release - Submit a pull request with a clear description of your changes
See the CHANGELOG for recent changes and release history.
Dataflow-rs is developed by the team at Plasmatic. We build open-source tools for data processing and automation.
This project is licensed under the Apache License, Version 2.0. See the LICENSE file for more details.