AstroML

//WIP

Dynamic Graph Machine Learning Framework for the Stellar Network

AstroML is a research-driven Python framework for building dynamic graph machine learning models on the Stellar Development Foundation Stellar blockchain.

It treats blockchain data as a multi-asset, time-evolving graph, enabling advanced ML research on transaction networks such as fraud detection, anomaly detection, and behavioral modeling.

---

✨ Features

AstroML provides end-to-end tooling for:

---

🧠 Core Idea

Blockchain networks are naturally graph-structured systems:

Blockchain Concept	Graph Representation
Accounts	Nodes
Transactions	Directed edges
Assets	Edge types
Time	Dynamic dimension

Most analytics tools rely on static heuristics or SQL queries.

AstroML instead enables:

• Dynamic graph learning
• Temporal GNNs
• Representation learning
• Research-grade experimentation

---

🎯 Target Users

AstroML is designed for:

• ML researchers
• Graph ML engineers
• Fraud detection teams
• Blockchain data scientists

---

🏗 Architecture Overview

High-Level Pipeline

┌─────────────────────────────────────────────────────────────────────────┐
│                    AstroML: Ingestion → Graph → Train                   │
└─────────────────────────────────────────────────────────────────────────┘

                              ┌──────────────┐
                              │ Stellar      │
                              │ Ledgers      │
                              └──────┬───────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  1. INGESTION LAYER             │
                    │  ├─ Ledger backfill (Polars)   │
                    │  ├─ Incremental ingestion      │
                    │  ├─ State tracking (idempotent)│
                    │  └─ PostgreSQL storage         │
                    └────────────────┬────────────────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  2. NORMALIZATION LAYER         │
                    │  ├─ Raw Stellar schema          │
                    │  │  (Ledger, Transaction, Op)   │
                    │  ├─ Graph mirror layer          │
                    │  │  (GraphAccount, GraphEdge)   │
                    │  └─ Composite indexes           │
                    │     (account_id, timestamp)     │
                    └────────────────┬────────────────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  3. GRAPH BUILDING LAYER        │
                    │  ├─ Time-windowed snapshots     │
                    │  ├─ Edge construction           │
                    │  ├─ Node indexing               │
                    │  └─ Graph validation            │
                    └────────────────┬────────────────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  4. FEATURE ENGINEERING         │
                    │  ├─ Transaction frequency       │
                    │  ├─ Asset diversity             │
                    │  ├─ Structural importance       │
                    │  │  (degree, betweenness, PR)   │
                    │  ├─ Feature store & versioning  │
                    │  └─ Point-in-time queries       │
                    └────────────────┬────────────────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  5. TRAINING LAYER              │
                    │  ├─ Temporal train/test split   │
                    │  ├─ Link prediction task        │
                    │  ├─ Negative sampling           │
                    │  ├─ PyTorch Geometric models    │
                    │  │  (GCN, GraphSAGE, GAT)       │
                    │  └─ Early stopping              │
                    └────────────────┬────────────────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  6. BENCHMARKING & EVALUATION   │
                    │  ├─ Reproducible configs        │
                    │  ├─ Random seed tracking        │
                    │  ├─ Metric computation          │
                    │  │  (AUC, Precision, Recall)    │
                    │  ├─ Memory profiling            │
                    │  └─ Result persistence          │
                    └────────────────┬────────────────┘
                                     │
                              ┌──────▼──────┐
                              │ Baseline    │
                              │ Results     │
                              └─────────────┘

Data Flow Details

Stellar Ledger Data
    ↓
[Ingestion Service]
    ├─ Fetch ledgers (1000000-1100000)
    ├─ Track state (.astroml_state/ingestion_state.json)
    └─ Store in PostgreSQL
    ↓
[Database Schema]
    ├─ Raw Layer: Ledger, Transaction, Operation, Account, Asset
    ├─ Graph Layer: GraphAccount, GraphEdge, GraphTransactionDetail
    └─ Indexes: (account_id, timestamp) composite
    ↓
[Graph Snapshot]
    ├─ Query operations by time window
    ├─ Create Edge objects (src, dst, timestamp, asset, amount)
    ├─ Build node_index mapping
    └─ Validate graph (isolated nodes, self-loops, density)
    ↓
[Feature Store]
    ├─ Compute node features (frequency, diversity, centrality)
    ├─ Compute edge features (asset type, amount, direction)
    ├─ Version features with metadata
    └─ Store in SQLite + Parquet
    ↓
[Temporal Split]
    ├─ Sort edges by timestamp
    ├─ Split at cutoff (80% train, 20% test)
    └─ Ensure no future data leaks into training
    ↓
[Link Prediction Task]
    ├─ Context window: edges before cutoff
    ├─ Future window: edges after cutoff
    ├─ Positive labels: future edges
    ├─ Negative sampling: random non-edges
    └─ Binary classification objective
    ↓
[Model Training]
    ├─ LinkPredictor(encoder + decoder)
    ├─ Adam optimizer with early stopping
    ├─ Compute AUC, Precision, Recall, F1
    └─ Track training/validation losses
    ↓
[Benchmark Results]
    ├─ config.json (full configuration + seed)
    ├─ result.json (metrics + performance)
    └─ metadata.json (run_id, timestamp, linking files)

Module Organization

astroml/
├── ingestion/           # Ledger ingestion & state tracking
│   ├── service.py       # IngestionService (incremental, idempotent)
│   ├── state.py         # StateStore (tracks processed ledgers)
│   └── backfill.py      # Bulk ledger loading
├── db/                  # Database layer
│   ├── schema.py        # SQLAlchemy ORM models
│   └── session.py       # Database connection management
├── features/            # Feature engineering
│   ├── feature_store.py # Enterprise feature management
│   ├── graph/
│   │   └── snapshot.py  # Time-windowed graph construction
│   ├── frequency.py     # Transaction frequency features
│   ├── asset_diversity.py
│   └── gnn/             # Graph neural network layers
├── models/              # ML models
│   ├── link_predictor.py
│   ├── gcn.py
│   ├── sage.py
│   └── deep_svdd.py
├── tasks/               # Training tasks
│   └── link_prediction_task.py
├── training/            # Training utilities
│   ├── temporal_split.py # Prevent data leakage
│   └── train_link_prediction.py
├── benchmarking/        # Benchmarking framework
│   ├── core.py          # ModelBenchmark orchestrator
│   ├── config.py        # Configuration management
│   └── metrics.py       # Metric computation
├── quick_start.py       # Quick start pipeline
└── cli.py               # Command-line interface

---

🚀 Quick Start

Prerequisites

Before running AstroML ensure you have the following installed:

Requirement	Minimum version	Notes
Python	3.10+	3.11 recommended; 3.12 supported
Docker & Docker Compose	24+	Required for the database and cache containers
Git	any	For cloning the repository
Make	any	Optional but recommended; used for convenience targets
CUDA toolkit	11.8+	Optional; only needed for GPU-accelerated training

Verify your Python version before proceeding:

python --version   # must print 3.10 or higher
docker --version   # must be available

Requirements files

Three requirements files are provided — pick the one that matches your workflow:

File	When to use
requirements.txt	Default — full stack including training, without GPU
requirements-cpu.txt	CPU-only training on machines without a CUDA-capable GPU
requirements-train.txt	GPU training; includes PyTorch with CUDA support
requirements-api.txt	API server only; excludes heavy ML dependencies
requirements-dev.txt	Development + testing; adds linting/typing tools on top of requirements.txt
requirements-minimal.txt	Config parsing only — useful in CI stages that don't run training

Tip: If you only want to explore the quick start without training, requirements-minimal.txt + requirements-api.txt is the lightest combination.

See REQUIREMENTS.md for a detailed breakdown of every package.

Option 1: Using Make (Recommended)

# Run quick start with default settings (100 ledgers, 50 accounts, 10 epochs)
make quickstart

# Run with more data for thorough testing
make quickstart-verbose

Option 2: Using Python Module

# Run quick start with default settings
python -m astroml.quick_start

# Run with custom parameters
python -m astroml.quick_start --num-ledgers 200 --num-accounts 100 --epochs 20 --seed 42

Option 3: Using CLI

# Run quick start command
python -m astroml quickstart --num-ledgers 100 --num-accounts 50 --epochs 10 --seed 42

What Quick Start Does

The quick start pipeline:

1. Generates sample data: Creates 100 synthetic ledgers with 50 accounts and realistic transactions
2. Builds transaction graph: Constructs a time-windowed graph with ~2000 edges
3. Validates graph: Checks for isolated nodes, self-loops, and computes statistics
4. Trains baseline model: Trains a LinkPredictor model for 10 epochs
5. Saves reproducible results: Stores config, results, and metadata for reproducibility

Output:

benchmark_results/quickstart/
├── config.json          # Full configuration with random seed
├── result.json          # Training metrics and performance
└── metadata.json        # Run metadata linking config and result

Expected output (typical values — exact numbers vary by seed):

================================================================================
AstroML Quick Start: Ingestion → Graph → Train Pipeline
================================================================================

[Step 1/5] Generating sample ledger data...
Generated 100 ledgers with 50 accounts

[Step 2/5] Building transaction graph...
Built graph with 2000 edges and 50 nodes

[Step 3/5] Creating benchmark configuration...

[Step 4/5] Training baseline model...
Epoch 0: Train Loss = 0.6931, Val Loss = 0.6892
Epoch 5: Train Loss = 0.4521, Val Loss = 0.4612
Training complete. Best metrics: {'auc': 0.92, 'precision': 0.88, 'recall': 0.85}

[Step 5/5] Saving benchmark results...
Saved config to benchmark_results/quickstart/config.json
Saved result to benchmark_results/quickstart/result.json
Saved metadata to benchmark_results/quickstart/metadata.json

✓ Quick start completed successfully!
Results saved to: benchmark_results/quickstart
================================================================================

Expected metric ranges on 10 epochs with the default seed:

• AUC: 0.85 – 0.96
• Precision: 0.80 – 0.93
• Recall: 0.78 – 0.91
• Training time: 5 – 30 s (CPU) / 2 – 8 s (GPU)

Troubleshooting

"Port 8000 already in use"

Another process is bound to port 8000. Find and stop it:

# Find the process
lsof -i :8000          # macOS / Linux
netstat -ano | findstr :8000   # Windows

# Stop it, or change the API port in docker-compose.yml:
#   ports: ["8001:8000"]

"Database connection refused"

The PostgreSQL container is not running. Start it:

docker compose up -d db
# Wait ~10 seconds for Postgres to initialise, then retry
docker compose logs db

Also verify your DATABASE_URL in .env matches the container settings (default: postgresql://astroml:astroml@localhost:5432/astroml).

"Model training CUDA out of memory"

Reduce batch size or switch to CPU:

# configs/training/default.yaml
training:
  device: cpu          # force CPU
  batch_size: 256      # reduce from default 1024

Alternatively, use requirements-cpu.txt which installs a CPU-only PyTorch build.

"Module import errors" / ModuleNotFoundError

Ensure you are in the right virtual environment and have installed dependencies:

source venv/bin/activate          # or: conda activate astroml
pip install -r requirements.txt

If the error mentions astroml itself, install the package in editable mode:

pip install -e .

For GPU-related import errors (No module named 'torch_geometric'), install the full training requirements:

pip install -r requirements-train.txt

Quick-start produces no output / hangs

Check that the SQLite temp path is writable and that no previous benchmark result is locked:

rm -rf benchmark_results/quickstart/
make quickstart

---

🔄 Full Setup

Using Docker (Recommended)

For the quickest setup with all dependencies, use Docker:

# Clone and navigate to repository
git clone https://github.com/Traqora/astroml.git
cd astroml

# Start with Docker
cp .env.example .env
./scripts/docker-start.sh core

# Access services
curl http://localhost:8000            # API
open http://localhost:3000            # Grafana

📚 Full Docker Setup: See DOCKER.md for comprehensive documentation including:

• Docker Quick Reference - Quick commands and common tasks
• Environment Configuration - Configuration guide
• Production Deployment - Production setup
• Troubleshooting - Common issues and solutions

Local Development Setup

1. Clone the repository

git clone https://github.com/Traqora/astroml.git
cd astroml

2. Create environment

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Note: Three requirements files are available. See REQUIREMENTS.md for guidance on which to use based on your environment (GPU training, CPU-only, or minimal config-only).

3. Configure database

A lightweight Docker Compose setup is provided to spin up PostgreSQL and Redis with persistent volumes. Simply run:

docker compose up -d

This starts only the database and cache, letting you run Python scripts and training natively on your machine. Alternatively, you can configure your own database and update config/database.yaml.

---

🤖 LLM Agent Framework

AstroML includes an LLM agent framework for multi-step reasoning and autonomous task execution over the graph pipeline. It is provider agnostic, dependency light (the core loop is standard library only) and traces every step so runs stay auditable.

# Offline smoke test with the deterministic echo provider
python -m astroml.agent "Summarise this transaction graph"

# Against a local Ollama server, with task decomposition
python -m astroml.agent --provider ollama --model llama3.1 --plan \
  "Rank the busiest accounts and flag anything unusual"

# Analyse a graph file and print the full trace as JSON
python -m astroml.agent --edges data/edges.json --json "How many accounts?"

from astroml.agent import (
    AgentConfig,
    AgentExecutor,
    build_default_registry,
    provider_from_env,
)

agent = AgentExecutor(
    llm=provider_from_env(),           # echo | scripted | openai | ollama | ...
    tools=build_default_registry(),    # graph_overview, window_stats, ...
    config=AgentConfig(mode="auto", max_steps=8),
)
result = agent.run("Which accounts look unusual?")

print(result.answer)
print(result.trace.summary())

📚 Full guide: docs/agent-framework.md — providers, tools, memory, planning, CLI flags and design notes.

---

📥 Data Ingestion

Backfill ledgers:

python -m astroml.ingestion.backfill \
  --start-ledger 1000000 \
  --end-ledger 1100000

---

🕸 Build Graph Snapshot

Create a rolling time window graph:

python -m astroml.graph.build_snapshot --window 30d

---

🧪 Synthetic Fraud Pattern Injection

Create benchmark datasets by injecting controlled fraud structures into a clean ledger copy:

python -m astroml.ingestion.synthetic_fraud_injector \
  --input data/clean_ledger.jsonl \
  --output data/ledger_with_fraud.jsonl \
  --summary outputs/fraud_injection_summary.json \
  --sybil-clusters 3 \
  --sybil-cluster-size 8 \
  --wash-loops 2 \
  --wash-loop-size 5

The injector appends transactions tagged with synthetic_fraud=true and fraud_pattern (sybil_cluster or wash_trading_loop) for downstream benchmarking.

---

🤖 Train Baseline GCN

python -m astroml.training.train_gcn

---

📊 Example Use Cases

• Liquidity Monitoring for the Stellar Community Fund
• Fraud / scam detection
• Account clustering
• Transaction risk scoring
• Temporal behavior modeling
• Self-supervised embeddings
• Network anomaly detection

---

🔬 Research Goals

AstroML emphasizes:

• Reproducibility
• Modular experimentation
• Scalable ingestion
• Temporal graph learning
• Production-ready ML pipelines

---

🛠 Tech Stack

• Python
• PyTorch / PyTorch Geometric
• PostgreSQL
• NetworkX / graph tooling

---

📌 Roadmap

• Real-time streaming ingestion
• Temporal GNN models
• Contrastive learning pipelines
• Feature store
• Model benchmarking suite
• Docker deployment

---

🤝 Contributing

Contributions are welcome!

fork → branch → commit → PR

Please open issues for bugs or feature requests.

---

📜 License

MIT License