Sentinel

Streaming anomaly detection on crypto markets. Kafka carries the trades, Spark turns them into rolling features, an Isolation Forest decides what looks wrong, and a dashboard shows you what it flagged.

It runs on either a simulator with anomalies deliberately injected (useful because you know the ground truth) or on live Binance trades over their public WebSocket.

There is a second, newer pipeline alongside it that does something no confirmed-block dataset can: it reads Solana transactions in propagation, before anyone knows whether they will land, and reconciles them against the chain afterwards. That gives you the transactions which were broadcast and never included: demand that existed and left no trace anywhere else. See solana-flow/, and the Solana Flow page in the dashboard.

Getting it up

Docker and Docker Compose is all you need.

docker compose up --build

For real market data instead of the simulator:

DATA_SOURCE=binance docker compose up --build

Give it two or three minutes on first start. Nothing is broken during that time; Spark has to accumulate enough windows to write features, the training job waits for those files to appear, and the API only picks up a model once training has finished. docker compose logs -f if you want to watch it happen.

Then: dashboard on 8501, API on 8000 (/docs for Swagger), Spark UI on 4040.

docker compose down -v when you're done, if you want the volumes gone too.

The pipeline

   generator --> Kafka --> Spark Streaming --> Parquet --> training --> API --> dashboard
  (sim or                  rolling stats,     partitioned  Isolation    /predict   Streamlit
   Binance)                z-scores           by symbol    Forest

Generating. The simulator produces BTC/ETH/BNB against USDT with configurable rates of price spikes, volume spikes and flash crashes, and labels each event so you can measure yourself afterwards. The Binance connector streams real trades and needs no API key. Both emit the same shape:

{
  "timestamp": 1710000000,
  "symbol": "BTC-USDT",
  "price": 43150.50,
  "volume": 12.534210,
  "log_return": 0.003521,
  "is_anomaly": false,
  "anomaly_type": null
}

Spark. Structured Streaming reads the topic and computes rolling mean and standard deviation for price, log return and volume over one-minute tumbling windows, then z-scores off those. Anywhere the standard deviation is zero the z-score returns 0 rather than a NaN. A flat minute is not an anomaly, and letting a NaN through poisons everything downstream. Output is Parquet partitioned by symbol.

Training. Isolation Forest, 200 estimators, 1% contamination, over the five z-score and volatility features, with a StandardScaler in front. On simulated data there's an 80/20 split and a classification report, since the labels exist. On real data it's fully unsupervised: there is nothing to score against, and pretending otherwise would be dishonest.

Serving. FastAPI comes up immediately and loads the model lazily, so the API is reachable while training is still running and reports model: not loaded rather than refusing to start. Endpoints:

curl http://localhost:8000/health
curl http://localhost:8000/stats
curl http://localhost:8000/model-info
curl "http://localhost:8000/latest-predictions?limit=50&symbol=BTC-USDT"

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"symbol":"BTC-USDT","z_score_price":4.5,"z_score_log_return":3.8,
       "z_score_volume":1.5,"rolling_price_std":0.008,"rolling_volume_std":25}'

/system-status reports on the other services too, which is what the dashboard's status page is built on.

Dashboard

Four pages in the sidebar.

System Status shows whether the API, Spark, Kafka and Zookeeper are up, with latency and the current model parameters. Start here if something looks wrong.

Live Feed auto-refreshes. Pick a symbol, a refresh interval and how far back to look, and it shows the score timeline and recent alerts.

Analytics is the retrospective view: per-symbol breakdown, feature correlations, score trend, and a filterable table you can export to CSV.

Manual Test lets you push feature values in by hand. The presets (Normal, Price Spike, Volume Spike, Flash Crash) fill the sliders with something representative, which is by far the quickest way to get a feel for where the decision boundary actually sits.

Solana pre-confirmation flow

The second pipeline. Attempts seen in propagation get joined against confirmed blocks, and every transaction lands in one of three buckets: landed, failed_onchain, or never_landed. The third one is the point, because it is invisible to any RPC, Dune query or analytics product, because once the slot has passed the evidence is gone.

From that you get things that cannot be computed downstream of a confirmed block: the empirical inclusion rate per priority fee (rather than everyone guessing), unmet demand per pool, and coordinated buying visible at the moment it is attempted rather than after only the survivors have settled.

It needs care in one place. Deciding something never landed is an argument from absence, and a dropped shred looks exactly like a lost race. So every row carries the coverage of its slot and anything reasoning about absence filters on it. The details, including the two independent coverage estimates and why they should agree, are in solana-flow/README.md.

shred-ingest/ is the Rust front of that pipeline: it reads shreds off a Jito proxy, rebuilds slots with Reed-Solomon when pieces are missing, decodes the swaps, and writes the NDJSON the rest of it reconciles. Its round-trip tests build real shreds, drop some, and demand the original bytes back, so the recovery path is checked rather than assumed. What it has never seen is a real shred: see its README for exactly what is verified and what is not.

The dashboard's Solana Flow page runs off the bundled simulator by default, so it works on a fresh clone with no Solana access at all:

FLOW_SOURCE=demo uvicorn main:app     # from api/
streamlit run app.py                  # from dashboard/

Configuration

Everything is environment variables, all with defaults that work under Compose:

Variable	Default
DATA_SOURCE	simulated	or binance
KAFKA_BOOTSTRAP_SERVERS	localhost:9092
KAFKA_TOPIC	crypto-market
EVENT_FREQUENCY_SECONDS	1	Simulator only
ANOMALY_PROBABILITY	0.01	Simulator only
MIN_PARQUET_FILES	3	How much data before training starts
MAX_WAIT_SECONDS	600	Give up waiting for it
MODEL_PATH / FEATURES_PATH		Where the model and features live
API_BASE_URL	http://localhost:8000	Used by the dashboard

Seven services come up: Zookeeper and Kafka, the generator, Spark, the training job (which exits once it's done), the API and the dashboard. They share three volumes: features written by Spark and read by training, Spark's checkpoints, and the model written by training and read by the API.

Tests

pip install -r tests/requirements.txt
pytest

Covers the simulator's event structure and log returns, the Binance connector's symbol mapping and message parsing, preprocessing (NaN handling, labelled and unlabelled paths), the API schemas and endpoints, and config defaults and overrides.

Layout

data-generator/ produces events, spark-java/ is the Maven-built streaming job, ml-python/ trains and evaluates, api/ serves predictions, dashboard/ is the Streamlit app, docker/ holds the Dockerfiles and tests/ the pytest suite.

There's more detail in docs/report.md, and docs/choices-en.md explains why things were built the way they were. That one is worth reading if you're wondering why Isolation Forest rather than something supervised.