pydecide

decide is one Python client for typed decisions - Choice, Score and Noul - over any "System One" decision model: TypeSafe's hosted Jev, OpenRouter's Decisions endpoint, the open-weight laya family (PyTorch and MLX), any sentence-transformers CrossEncoder, and a JSON-prompted LLM fallback. The core of the library is the fallback chain: a Client takes an ordered list of these backends and a confidence policy, and if a backend errors or answers with low confidence, the next backend in the chain is tried, so a cheap local model can back off to a hosted one only when it's unsure.

The library also ships a small HTTP server (see Server below) that speaks TypeSafe's wire protocol. laya now ships its own laya-serve as of 0.3.7, so this server isn't the reason to reach for decide if you only run laya

• its purpose is chaining several backends (local and hosted) behind one TypeSafe-compatible endpoint.

Install

uv tool install pydecide   # the decide CLI on your PATH, from any directory
uv add pydecide            # inside a project

With pip: pip install pydecide / pipx install pydecide. Either way, this includes the CLI, the hosted backends and the server.

Local model backends need an extra:

uv tool install "pydecide[all]"

With pip: pip install "pydecide[all]". Quote the brackets in zsh, since it otherwise tries to glob them. This pulls in PyTorch and MLX, about 2 GB.

Extra	Adds
server	Nothing further; kept as an empty extra so pip install "pydecide[server]" still works. FastAPI and uvicorn ship in the base install.
laya	laya backend (PyTorch)
mlx	laya_mlx backend (Apple Silicon; needs Python 3.11+)
st	crossencoder backend (sentence-transformers)
all	laya + mlx + st

The mlx extra depends on laya-mlx, which requires Python 3.11 or newer; on 3.10 the extra installs nothing and the laya_mlx backend is unavailable.

Quickstart: zero configuration

Install a local backend and decide ask works right away, with no environment variables at all:

uv tool install "pydecide[mlx]"    # Apple Silicon
# or
uv tool install "pydecide[laya]"   # any platform (PyTorch)

decide ask "I had a rough day, everything broke" --noul "Is the writer doing well?" --choice good,bad

NAME    TYPE    ANSWER  PROBABILITIES
choice  choice  bad     good=0.01 bad=0.99
noul    noul    false   0.01
backend=laya_mlx route=laya_mlx:ok latency=33.1ms

This is the real output of that command, run with laya_mlx installed and none of DECIDE_LOCAL_MODEL, TYPESAFE_API_KEY, OPENROUTER_API_KEY, OPENAI_API_KEY, DECIDE_LLM_BASE_URL or DECIDE_BACKENDS set.

The Python client works the same way. Client.from_env() builds a backend chain from whatever is installed and configured in the process environment; with a local backend installed, it needs no variables either. Pass an explicit env= mapping instead (e.g. in tests) to configure from something other than os.environ:

from decide import Client, Choice, Score, Noul

client = Client.from_env()

r = client.decide(
    state={"ticket": "I was charged twice, please refund."},
    questions={
        "team": Choice(
            "Which team handles this?",
            {
                "billing": "charges, refunds",
                "engineering": "bugs and outages",
                "sales": "pricing and upgrades",
            },
        ),
        "severity": Score("How severe is this?", ["minor", "degraded", "blocked"]),
        "refund": Noul("Does the customer ask for money back?"),
    },
)

r.choices["team"].choice  # "billing"
r.choices["team"].probabilities  # {"billing": 1.0, "engineering": 0.0, "sales": 0.0}
r.scores["severity"].score  # 1.5522  (expected level index, 0..len(levels)-1)
r.scores["severity"].probabilities  # [0.0197, 0.4083, 0.5719]
r.nouls["refund"].noul  # 0.9461
r.meta.backend  # "laya_mlx"
r.meta.latency_ms  # 37.1
r.meta.route  # ["laya_mlx:ok"]

This was run against the laya_mlx backend with no environment variables set at all, using its own default checkpoint; every value above is the real output of that run, not illustrative.

from_env picks the chain in this order, using whatever is installed and/or configured: laya_mlx or laya, whichever is importable, with its own default model unless DECIDE_LOCAL_MODEL overrides it; typesafe (if TYPESAFE_API_KEY is set); openrouter (if OPENROUTER_API_KEY is set); llm (if DECIDE_LLM_BASE_URL or OPENAI_API_KEY is set). A local backend, when installed, is tried first, with the hosted backends as fallback. Set DECIDE_BACKENDS="laya,typesafe" to override the order explicitly. If nothing is importable or configured, from_env raises ConfigError telling you to install a local backend or set a hosted one's API key.

AsyncClient has the same surface, awaited: await client.decide(...), await client.decide_batch(...), AsyncClient.from_env(...).

Backends

Module	Backend name	Extra	Notes
typesafe.py	typesafe	none (httpx only)	POST {base_url}/v1/systemone, bearer auth. Default base URL https://api.typesafe.ai, default model jev-latest.
openrouter.py	openrouter	none	Same wire shape as typesafe, POST https://openrouter.ai/api/alpha/decisions, default model typesafe/jev-latest.
laya.py	laya	pydecide[laya]	Local PyTorch laya.Agent, default model convaiinnovations/laya when constructed directly.
laya_mlx.py	laya_mlx	pydecide[mlx] (Python 3.11+)	Local MLX laya_mlx.Agent (Apple Silicon), default model aac6fef/laya-multilingual-mlx when constructed directly.
crossencoder.py	crossencoder	pydecide[st]	Local sentence_transformers.CrossEncoder. Not configurable from the environment; construct it directly and pass it to Client([...]).
llm.py	llm	none (httpx only)	Any OpenAI-compatible chat-completions server, default base URL https://api.openai.com/v1, default model gpt-4o-mini. The least trustworthy backend (see below).

crossencoder is built by hand, for example:

from decide import Client, Choice
from decide.backends.crossencoder import CrossEncoderBackend

backend = CrossEncoderBackend("cross-encoder/ms-marco-MiniLM-L6-v2")
client = Client([backend])

r = client.decide(
    "I was charged twice for the same order last week, please refund the duplicate charge.",
    {
        "team": Choice(
            "Which team should handle this?",
            {
                "billing": "Charges, invoices, payment problems, refunds",
                "eng": "Bugs, crashes, broken features",
                "shipping": "Delivery status, delays, lost packages",
            },
        )
    },
)
r.choices["team"].choice  # "billing"
r.choices["team"].probabilities  # {"billing": 0.947, "eng": 0.019, "shipping": 0.034}

Environment variables: overrides and hosted-backend keys

None of these are required to get started - see Quickstart: zero configuration above. They either override a local backend that from_env already auto-selects once it's installed, or supply the API key a hosted backend needs to be auto-selected at all.

Variable	Backend	Meaning
DECIDE_LOCAL_MODEL	laya, laya_mlx	Overrides the default Hugging Face repo id (or local path) of the model to load. Optional: omit it and the installed local backend auto-selects with its own default model.
TYPESAFE_API_KEY	typesafe	API key, sent as Authorization: Bearer. Required to auto-select typesafe.
TYPESAFE_BASE_URL	typesafe	Overrides the default https://api.typesafe.ai.
OPENROUTER_API_KEY	openrouter	API key, sent as Authorization: Bearer. Required to auto-select openrouter.
DECIDE_LLM_BASE_URL	llm	Base URL of an OpenAI-compatible chat-completions server. Setting it (or OPENAI_API_KEY) auto-selects llm.
OPENAI_API_KEY	llm	API key, sent as Authorization: Bearer, if the server needs one.
DECIDE_LLM_MODEL	llm	Model name to request; defaults to gpt-4o-mini.
DECIDE_BACKENDS	Client.from_env	Comma-separated backend names, overriding auto-detection entirely.
DECIDE_MIN_CONFIDENCE	Client.from_env (Gate)	Float threshold for the default Gate built by from_env, when no explicit policy is passed.
DECIDE_API_KEY	decide serve	Bearer token required to call the server, when --api-key is not passed. The flag takes precedence over this variable.

Gating and fallback

from decide import Gate

Gate(
    min_confidence=0.0,  # top probability of a Choice, max(noul, 1-noul) for a Noul
    per_question=None,  # optional {"question_name": threshold} overrides
    per_type=None,  # optional {"choice"/"score"/"noul": threshold} overrides
    on_error="next",  # or "raise" to stop the chain on the first BackendError
)

For a question with a threshold set at more than one level, the most specific one wins: per_question[name], else per_type[<the question's type>], else min_confidence. Score answers are never gated regardless of which of these sets a threshold for "score".

For each backend in order: call it. On BackendError with on_error="next", append "<name>:error" to meta.route and try the next backend (with on_error="raise", the error propagates immediately instead). If the response fails the gate, append "<name>:low_confidence:<question>=<value><threshold>" and try the next backend. If it passes, append "<name>:ok" and return. Score answers are never gated (there is no single confidence number for an expected value over levels).

If every backend is exhausted without a passing response, the best response seen so far (highest minimum confidence across its gated answers) is returned, with meta.route[-1] == "<name>:accepted_low_confidence". If no backend produced any response at all, Client.decide raises AllBackendsFailed(route, errors).

Route strings you will see in meta.route:

• "<name>:ok" - the backend answered and passed the gate.
• "<name>:error" - the backend raised a BackendError.
• "<name>:low_confidence:<question>=<value><threshold>" - the backend answered but at least one gated question fell below its threshold.
• "<name>:accepted_low_confidence" - appended once, at the end of the route, when no backend passed the gate and the best low-confidence response was returned instead.

Client.decide_batch/AsyncClient.decide_batch run the same chain per input state, preserving input order; a state that clears the gate on an earlier backend is not sent to later ones. Client.decide_batch uses a backend's real batch path when capabilities().batch is true, looping decide per request otherwise. AsyncClient.decide_batch always loops adecide per state, concurrently via asyncio.gather; it does not use a backend's batch path in v1. If any state in the batch is left unrouted, AllBackendsFailed carries partial (every Response that did resolve, keyed by input index) and failed (the route so far for every state that did not), so the resolved siblings are not silently lost.

Choosing a threshold

Jevals, an independent Jev benchmark, measured Jev's own accuracy against the confidence threshold it would take to reach it, over 1,500 decisions per task:

Banking77, a Choice task (overall accuracy 79.7%):

Threshold	Coverage	Accuracy
0.55	93%	83.4%
0.65	88%	85.1%
0.85	76%	88.6%
0.96	59%	94.3%

PubMedQA, a yes/no Noul task (overall accuracy 91.3%):

Threshold	Coverage	Accuracy
0.55	98%	92.0%
0.65	94%	92.7%
0.85	69%	96.4%
0.91	49%	98.6%

Two things follow from this. First, thresholds below about 0.85 filter almost nothing - most of the accuracy gain from gating shows up only once the threshold climbs well past the model's overall accuracy. Second, Jev clips its probabilities to the 0.01-0.99 range, so a yes/no answer rarely exceeds 0.98; a single min_confidence applied to both Choice and Noul answers therefore behaves very differently across the two - a 0.96 threshold still keeps 59% of choice answers but only about 11% of yes/no answers. Use per_type to set separate thresholds:

from decide import Gate

Gate(per_type={"choice": 0.85, "noul": 0.95})

Server: point TypeSafe's SDK at a local model

laya ships its own laya-serve command as of 0.3.7, so if you only run laya behind a single endpoint, that's the more direct option. Reach for decide serve instead when you want to chain several backends - local and/or hosted - behind one TypeSafe-compatible endpoint, with the same fallback and gating behavior as the Python client.

DECIDE_LOCAL_MODEL=aac6fef/laya-multilingual-mlx decide serve --backends laya_mlx --port 8811

GET /health reports liveness and the configured backend names; GET /v1/models lists them in an OpenAI-style shape; POST /v1/systemone takes a TypeSafe SystemOneRequest body and returns a SystemOneResponse-shaped body plus a decide extension carrying our own backend/route metadata. Errors come back as {"error": {"message": ..., "type": ...}} with a matching HTTP status.

If --api-key/DECIDE_API_KEY is set, requests must send a matching Authorization: Bearer <token> header; the key must be ASCII, since HTTP header bytes are latin-1 decoded by the server before comparison.

A raw request against the running server above:

curl -s -X POST http://127.0.0.1:8811/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"ticket": "I was charged twice, please refund."},
    "questions": {
      "team": {"type": "choice", "instructions": "Which team handles this?",
               "criteria": {"billing": "charges, refunds", "engineering": "bugs and outages", "sales": "pricing and upgrades"}},
      "severity": {"type": "score", "instructions": "How severe is this?", "criteria": ["minor", "degraded", "blocked"]},
      "refund": {"type": "noul", "instructions": "Does the customer ask for money back?"}
    }
  }'

{
  "model": "aac6fef/laya-multilingual-mlx",
  "answers": {
    "team": {"type": "choice", "choice": "billing", "confidence": 1.0,
              "probabilities": {"billing": 1.0, "engineering": 0.0, "sales": 0.0}},
    "severity": {"type": "score", "score": 1.5522, "confidence": 0.5719,
                 "legend": {"0": "minor", "1": "degraded", "2": "blocked"},
                 "probabilities": {"0": 0.0197, "1": 0.4083, "2": 0.5719}},
    "refund": {"type": "noul", "noul": 0.9461}
  },
  "usage": {"input_tokens": 0, "output_tokens": 0},
  "decide": {"backend": "laya_mlx", "latency_ms": 50.5, "route": ["laya_mlx:ok"]}
}

And the same request through TypeSafe's own SDK, pointed at the local server (no API key is needed since this server has none configured, but the SDK requires a non-empty string):

from typesafe_sdk import TypeSafeClient, Choice, Score, Noul

client = TypeSafeClient(api_key="anything", base_url="http://127.0.0.1:8811")

resp = client.system_one(
    state={"ticket": "I was charged twice, please refund."},
    questions={
        "team": Choice(
            instructions="Which team handles this?",
            criteria={
                "billing": "charges, refunds",
                "engineering": "bugs and outages",
                "sales": "pricing and upgrades",
            },
        ),
        "severity": Score(
            instructions="How severe is this?", criteria=["minor", "degraded", "blocked"]
        ),
        "refund": Noul(instructions="Does the customer ask for money back?"),
    },
)

model='aac6fef/laya-multilingual-mlx' usage=Usage(input_tokens=0, output_tokens=0)
answers={'team': ChoiceAnswer(type='choice', choice='billing', confidence=1.0,
  probabilities={'billing': 1.0, 'engineering': 0.0, 'sales': 0.0}),
 'severity': ScoreAnswer(type='score', score=1.5522, confidence=0.5719,
  legend={0: 'minor', 1: 'degraded', 2: 'blocked'},
  probabilities={0: 0.0197, 1: 0.4083, 2: 0.5719}),
 'refund': NoulAnswer(type='noul', noul=0.9461)}

TypeSafe's SDK parsed our server's response without modification: it is a genuine SystemOneResponse, not a hand-shaped dict.

CLI

--choice, --score and --noul question names are optional. An unnamed flag is auto-named after the flag itself (choice, score, noul); a second unnamed flag of the same type becomes choice2, score2, noul2, and so on:

decide ask "I had a rough day" --noul "Is the writer doing well?" --choice good,bad

NAME    TYPE    ANSWER  PROBABILITIES
choice  choice  bad     good=0.07 bad=0.93
noul    noul    false   0.02
backend=laya_mlx route=laya_mlx:ok latency=65.3ms

Name a question explicitly with NAME=... to control what shows up in the table and in --json; named and unnamed questions can be mixed on the same command line:

decide ask "I was charged twice, please refund." \
  --choice "team=billing,engineering,sales" \
  --score "severity=minor,degraded,blocked" \
  --noul "refund=Does the customer ask for money back?"

NAME      TYPE    ANSWER    PROBABILITIES
team      choice  billing   billing=0.93 engineering=0.02 sales=0.05
severity  score   degraded  1.09 (degraded)
refund    noul    true      0.87
backend=laya_mlx route=laya_mlx:ok latency=34.8ms

--json prints a machine-readable payload instead of the table, and writes only that JSON to stdout, so decide ask ... --json | jq . is safe to use in scripts. --min-confidence and --model are also available on ask. By default, ask and serve set HF_HUB_DISABLE_PROGRESS_BARS=1 (unless it's already set) so a local backend's model-loading progress bars don't mix into that output; pass --verbose to see them. When shown, that progress goes to stderr, never stdout.

decide backends

NAME          INSTALLED  CONFIGURED  INSTALL
typesafe      yes        no
openrouter    yes        no
laya          yes        default
laya_mlx      yes        default
crossencoder  yes        n/a
llm           yes        no

Every backend with INSTALLED no gets an INSTALL column naming the exact command to add it, e.g. pip install "pydecide[laya]". For laya and laya_mlx, CONFIGURED reads default when installed with no DECIDE_LOCAL_MODEL override (it will be auto-selected with its own default model), yes when DECIDE_LOCAL_MODEL is set, and no only when the backend isn't installed at all.

decide serve --backends a,b --host 127.0.0.1 --port 8811 [--api-key TOKEN] [--min-confidence FLOAT] runs the HTTP server described above.

What the probabilities mean

Every probability in a ChoiceAnswer, ScoreAnswer or NoulAnswer is whatever the backend reported; decide does not calibrate, smooth or verify it. What that means differs by backend: typesafe, openrouter and laya/laya_mlx are purpose-built decision models, but their outputs are still self-reported by the model and not audited by this library. The crossencoder backend turns a relevance reranker's raw logits into a softmax or sigmoid - the result is a normalized score forced to distribute mass over the supplied candidates, not a calibrated probability; a Choice will still pick a winner even when every candidate is a bad fit, and a Noul of 0.9 does not mean the condition holds 90% of the time. The llm backend is the least trustworthy of all: it prompts a general chat model to estimate its own confidence in JSON, with no guarantee the model attends to every candidate or keeps its numbers well calibrated. Treat all of this accordingly - as a signal to gate and fall back on, not as ground truth.

Status

pydecide is at 0.2.0. The public API may still change before a 1.0 release.

License

MIT, see LICENSE.