molq

Unified job queue — one submission API for local, SLURM, PBS, and LSF

Documentation  ·  Quick start  ·  Ecosystem

molq is a unified job queue for Python workloads that need the same submission API on a laptop, a workstation, or an HPC cluster. A Cluster says where jobs run, a Submitor tracks how they progress — and the same code runs against local subprocesses or remote schedulers over SSH.

Under active development. Public APIs may change between minor releases.

Capabilities

Module	Capability
cluster	Cluster — destination spec: scheduler kind × transport × scheduler options, plus live queue snapshots
submitor	Submitor + JobHandle — single entry point for submitting, tracking, and waiting on jobs
scheduler	Scheduler protocol plus one module per backend (shell, slurm, pbs, lsf), each routing shell calls through a transport and owning its own directive and dependency syntax
transport	LocalTransport / SshTransport — runs shell and file ops here or on a remote host via OpenSSH, sharing one multiplexed connection
store	JobStore — SQLite persistence with WAL mode, UUID job identity, schema versioning, and v1 auto-migration
validation	Checks a request against the backend's declared capability matrix before submitting
jobpaths	Job directory layout, log path resolution, script staging, and manifests
dependencies	Resolves molq job ids + conditions into the backend's submit syntax
retry / retention	Retry eligibility and backoff; expiry of job directories and old records
artifacts	Pulls logs and job directories back from the cluster
reconciler	JobReconciler — batch-queries schedulers, diffs against the store, syncs job state
monitor	Blocking waits and polling engine driven by pluggable strategies
strategies	Pluggable polling strategies; exponential backoff by default
callbacks	EventBus — synchronous pub/sub for job lifecycle events with handler isolation
models	Job data models — JobRecord, RetryPolicy, RetentionPolicy, JobDependency, SubmitorDefaults
types	Frozen value types — Memory, Duration, Script, JobResources, JobScheduling, JobExecution
options	Per-scheduler frozen option dataclasses (Local, Slurm, PBS, LSF) — no untyped dicts
config	Profile and config loading from molcfg (~/.molcrafts/molq/config/config.toml, or MOLCRAFTS_HOME)
plugin	MolqPlugin host — official builtins + third-party entry points (molq.plugins)
plugins	Official plugins (e.g. nerve → local Nerve menu-bar status; fail-open)
workspace	Workspace / Project — directory handles over a cluster's filesystem (local or remote)
ssh_config	Surfaces ~/.ssh/config hosts as cluster candidates
serde	Serialization helpers for stored requests and config-driven values
errors	Unified MolqError exception hierarchy with typed context
status	JobState enum with terminal-state semantics
merge	Pure function that merges per-submit parameters with Submitor defaults
dashboard	Full-screen terminal dashboard for monitoring runs and jobs
testing	FakeScheduler and make_submitor for tests and runnable examples without a real cluster
cli	Typer + Rich CLI: jobs (submit/list/status/logs/cancel/…), live (watch/monitor/daemon), setup (clusters/workspace/plugins)

Install

pip install molcrafts-molq

Requires Python 3.12+. Depends on typer, rich, molcrafts-mollog, and molcrafts-molcfg.

Quick start

import molq as mq

# Cluster = destination (where to run). Submitor = lifecycle (how jobs are tracked).
cluster = mq.Cluster("devbox", "local")

with mq.Submitor(target=cluster) as queue:
    handle = queue.submit_job(
        argv=["python", "-c", "print('hello from molq')"]
    )
    record = handle.wait()

print(record.state)

Swap to a cluster by changing one line — mq.Cluster("hpc", "slurm", host="user@hpc.example.com") — and the rest of the code is unchanged. See the documentation for retries, dependencies, profiles, and the CLI.

Documentation

• Quickstart — install and complete a local job
• Mental model — understand destinations, tracking, and records
• Submit jobs — commands, resources, retries, and dependencies
• Clusters and schedulers — local, SSH, SLURM, PBS, and LSF
• Monitor jobs — status, logs, history, and dashboards
• Remote files — stage inputs and collect results
• Command line — task-oriented CLI workflows
• Plugins and Nerve — lifecycle observers and menu-bar status
• Configuration — profiles, defaults, and remote destinations
• Python API — generated, responsibility-based reference

MolCrafts ecosystem

Project	Role
molpy	Python toolkit — the shared molecular data model & workflow layer
molrs	Rust core — molecular data structures & compute kernels (native + WASM)
molpack	Packmol-grade molecular packing (Rust + Python)
molvis	WebGL molecular visualization & editing
molexp	Workflow & experiment-management platform
molnex	Molecular machine-learning framework
molq	Unified job queue — local / SLURM / PBS / LSF — this repo
molcfg	Layered configuration library
mollog	Structured logging, stdlib-compatible
molhub	Molecular dataset hub
molmcp	MCP server for the ecosystem
molrec	Atomistic record specification

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md for development setup.

License

MIT — see LICENSE.

---

_{Crafted with 💚 by MolCrafts}