PyPlanner · Household Robot Task Planning

A pluggable, multi-backend LLM planning library for embodied AI — benchmarked end-to-end in AI2-THOR

---

What this project does

PyPlanner is a Python package that takes a natural-language task description and a list of visible objects, calls an LLM using one of seven planning algorithms, and returns a structured action plan ready to be executed by a household robot.

import pyplanner

planner = pyplanner.cot(provider="openai", model="gpt-4o-mini", api_key="sk-...")

steps, metrics = planner.generate_plan(
    task="make a cup of coffee",
    obs="Kitchen. Coffee machine on counter. Mug on shelf.",
    visible_objects=["CoffeeMachine", "Mug", "CounterTop", "Fridge"],
)

# steps →
# [{'action': 'Navigate', 'object': 'Mug',           'target': ''},
#  {'action': 'Grab',     'object': 'Mug',           'target': ''},
#  {'action': 'Navigate', 'object': 'CoffeeMachine', 'target': ''},
#  {'action': 'Place',    'object': 'Mug',            'target': 'CoffeeMachine'},
#  {'action': 'TurnOn',   'object': 'CoffeeMachine', 'target': ''}]

print(metrics.to_dict())
# {'method': 'CoT', 'latency_s': 2.3, 'llm_calls': 1,
#  'total_tokens': 580, 'num_steps': 5, 'parse_ok': True}

The companion thor_app connects PyPlanner to AI2-THOR, providing a full loop: natural language prompt → LLM plan → robot execution → goal verification → quantitative evaluation.

---

Live Demo

Try it in your browser — no local setup required

Task Planning Demo

Select a room task, choose a planning method and LLM provider, and watch the robot navigate and interact with objects in a photorealistic kitchen/living room/bedroom/bathroom. Bring your own OpenAI or Anthropic API key, or use the Ollama backend running on the demo server.

---

Repository layout

├── pyplanner/                  ← installable Python package (pip install -e ./pyplanner)
│   ├── __init__.py             ← public API: get(), list_methods(), factory functions
│   ├── base.py                 ← BasePlanner, PlanMetrics, LLMBackend
│   ├── direct.py               ← Method 1: Direct (baseline, 1 call)
│   ├── cot.py                  ← Method 2: Chain-of-Thought
│   ├── few_shot.py             ← Method 3: Few-Shot CoT
│   ├── self_refine.py          ← Method 4: Self-Refine (generate→critique→fix)
│   ├── react.py                ← Method 5: ReAct (Thought+Action per step)
│   ├── hierarchical.py         ← Method 6: Hierarchical (sub-goals → actions)
│   └── llm_router.py           ← Method 7: LLM Router (local generate + API verify)
│
└── thor_app/                    ← AI2-THOR demo application
    ├── app.py                   ← Streamlit UI
    ├── thor_server.py           ← AI2-THOR ZMQ server (skill primitives)
    ├── thor_client.py           ← ZMQ client
    ├── knowledge.py             ← 17 built-in task/scene definitions
    ├── make_dataset.py          ← static eval dataset (38 tasks, 3 difficulty levels)
    ├── make_dataset_from_sim.py ← ground dataset from real simulator state
    ├── record_reference.py      ← executable ground-truth plan recording
    ├── goal_checker.py          ← two-layer task-success verification
    ├── evaluate.py              ← offline benchmark (no simulator needed)
    └── evaluate_sim.py          ← online benchmark (plan + execute + verify)

---

PyPlanner — the package

Seven planning methods, one interface

#	Method	LLM calls	Strategy	When to use
1	Direct	1	Single prompt → JSON plan	Speed baseline
2	CoT	1	<reasoning> then <plan>	Logical multi-step tasks
3	Few-Shot CoT	1	3 built-in examples anchor the output	Consistent object naming
4	Self-Refine	1 + 2N	Generate → critique → fix, N rounds	Highest plan quality
5	ReAct	N	Thought + Action interleaved	Step-by-step grounding
6	Hierarchical	1 + N	Decompose to sub-goals, expand each	Complex multi-phase tasks
7	LLM Router	2	Local generates, frontier API verifies	Best of both, cost-efficient

Every method shares the same interface and returns the same PlanMetrics object.

Three LLM backends

# Ollama (local, free)
p = pyplanner.get("CoT", provider="ollama",    model="llama3.2",
                          host="http://localhost:11434")

# OpenAI
p = pyplanner.get("CoT", provider="openai",    model="gpt-4o-mini",
                          api_key="sk-...")

# Anthropic
p = pyplanner.get("CoT", provider="anthropic", model="claude-haiku-4-5-20251001",
                          api_key="sk-ant-...")

Or via shorthand factories:

pyplanner.direct()
pyplanner.cot()
pyplanner.few_shot()
pyplanner.self_refine(max_iterations=3)
pyplanner.react(max_steps=12)
pyplanner.hierarchical()
pyplanner.llm_router(verifier_backend="anthropic", anthropic_api_key="sk-ant-...")

Metrics every call returns

steps, m = planner.generate_plan(task, obs, visible_objects)

m.latency_s        # wall-clock seconds
m.llm_calls        # round-trips (1 for Direct/CoT, N for ReAct)
m.total_tokens     # input + output tokens
m.tokens_per_step  # efficiency metric
m.parse_ok         # True if output parsed successfully
m.to_dict()        # serialisable dict for CSV / logging

Replan on step failure

new_steps, m = planner.replan(
    task           = "make coffee",
    completed      = steps_executed_so_far,
    failed_step    = {"action": "Grab", "object": "Mug", ...},
    failure_reason = "Mug not reachable from current position",
    obs            = updated_observation,
    visible_objects= updated_visible,
)

---

thor_app — the demo application

Streamlit UI features

• Task browser — 17 built-in tasks across kitchen, living room, bedroom, bathroom; or free-form text input
• Scene picker — manual FloorPlan selector; scene auto-loads in AI2-THOR on change
• Planning config — method, provider, model, method-specific options (max steps, refine iterations)
• Direct command box — type Navigate apple / Place mug → CoffeeMachine, press Ctrl+Enter; object names resolve automatically (coffee_machine → CoffeeMachine)
• Live camera feed — frame from AI2-THOR updated after each executed step
• Execution log — per-step success/fail, reward, observation
• Benchmark panel — latency, LLM calls, tokens/step, parse rate across multiple runs

Setup

# 1. Install pyplanner
pip install -e ./pyplanner

# 2. Install application dependencies
pip install -r ./thor_app/requirements.txt

# 3. Start the AI2-THOR server (leave running in a separate terminal)
cd thor_app
python thor_server.py

# 4. Start the Streamlit UI
cd thor_app
streamlit run app.py

Windows/WSL note: if pip install -e fails, app.py contains an automatic sys.path fallback that resolves pyplanner at runtime without installation.

---

Evaluation pipeline

The ground-truth problem

LLM-generated or hand-written reference plans have no guarantee of executability. navigate coffee_machine fails if AI2-THOR expects CoffeeMachine. TurnOn CoffeeMachine succeeds as a step but doesn't make coffee if the mug was never placed.

This project addresses both problems with a three-stage pipeline.

Stage 1 — Record executable ground truth

python record_reference.py          # --ref-source manual (default) or llm
# outputs: eval_dataset_gt.json

Loads each scene in AI2-THOR, executes candidate steps, and records only steps that return success=True from the simulator. Object names are auto-resolved to real AI2-THOR types. The saved plan is what the robot actually did — not what the LLM suggested.

Stage 2 — Verify goal completion

python goal_checker.py --dataset eval_dataset_gt.json
# outputs: eval_dataset_verified.json

Step success ≠ task success. The checker uses two layers:

1. GoalCondition (deterministic) — per-task state assertions checked via the simulator. For "make coffee": mug placed on CoffeeMachine AND CoffeeMachine.isToggled == True. Fast, no LLM needed.
2. LLM judge (semantic fallback) — when state is ambiguous, asks the LLM: "Did these steps achieve the goal?" Returns success, confidence, and a one-sentence reason.

Stage 3 — Benchmark planners

# Offline: static plan quality metrics (no simulator)
python evaluate.py --methods Direct CoT "Self-Refine" Hierarchical \
                   --model llama3.2

# Online: execute plans in AI2-THOR, verify goal completion
python evaluate_sim.py --model llama3.2 --max-samples 10

# With OpenAI
python evaluate_sim.py --provider openai --model gpt-4o-mini --api-key sk-...

# Dry-run (no LLM, tests the pipeline)
python evaluate.py --dry-run

Metrics output (CSV):

Group	Metrics
Plan quality	executability rate, precondition score, completeness, redundancy, hallucination rate
Efficiency	latency (s), LLM calls, total tokens, tokens per step
Robustness	replan success rate, replan latency, step overlap ratio
Execution	task success rate, step success rate, cumulative reward
Aggregate	quality score, efficiency score, combined score (0–1)

---

Design decisions

Object name resolution — AI2-THOR uses CamelCase (CoffeeMachine); LLMs generate snake_case (coffee_machine). PyPlanner resolves this at execution time via a 4-step fuzzy matcher: exact match → CamelCase conversion → substring match → reverse substring. Applied to both LLM plans and direct user commands.

Simulator as ground truth — record_reference.py executes steps and records only those that succeed. This avoids the fundamental issue where LLM-generated reference plans may describe physically impossible or incorrectly sequenced actions.

Two-layer goal verification — goal_checker.py separates step success from task success, catching cases where all steps pass but the actual goal (e.g. coffee brewed, hands washed) was never reached.

Backend-agnostic core — LLMBackend in base.py routes to Ollama, OpenAI, or Anthropic through a single chat() interface. Every planner method and every planning algorithm works identically across all three backends with no code changes.

---

Quick performance comparison

import pyplanner

task  = "boil water in a pot on the stove"
obs   = "Kitchen. Stove visible. Pot on counter. Sink accessible."
vis   = ["Pot", "StoveBurner", "Sink", "Faucet", "CounterTop"]

for name in pyplanner.REGISTRY:
    p = pyplanner.get(name, model="llama3.2")
    steps, m = p.generate_plan(task, obs, vis)
    print(f"{name:15} {m.latency_s:5.1f}s  "
          f"{m.llm_calls} call(s)  {m.num_steps} steps  {m.total_tokens} tok")

Direct           1.8s  1 call(s)  6 steps   480 tok
CoT              2.9s  1 call(s)  7 steps   820 tok
Few-Shot CoT     3.1s  1 call(s)  6 steps   910 tok
Self-Refine      7.4s  5 call(s)  6 steps  2340 tok
ReAct           14.2s  6 call(s)  6 steps  1980 tok
Hierarchical     5.1s  3 call(s)  7 steps  1450 tok
LLM Router       4.3s  2 call(s)  6 steps   960 tok

---

Install

pip install -e ./pyplanner

Core requirements: Python 3.10+, ollama>=0.2.0, requests>=2.31.0

For thor_app: streamlit, ai2thor, pyzmq, pillow, opencv-python, pandas

---

_{Built with AI2-THOR · Ollama · Streamlit · Python 3.10+}