Skip to content

Run the test suites in CI, and make the fmt and clippy gates pass - #139

Closed
ParallelEntrepreneur wants to merge 9 commits into
mainfrom
colonizer/session-0ee041a4
Closed

ParallelEntrepreneur wants to merge 9 commits into
mainfrom
colonizer/session-0ee041a4

Conversation

@ParallelEntrepreneur

Copy link
Copy Markdown
Collaborator

README.md listed "CI running the Rust, runner and UI test suites" as PLANNED, and said
"No CI yet". Neither was quite true and neither was the whole story: release.yml already
builds, bundles and publishes to crates.io, but nothing ever ran a test. This adds the
workflow that does, and makes the lint gates it enforces actually pass.

What changed

.github/workflows/ci.yml (new). Runs on every pull request, on pushes to main, and on
workflow_dispatch, as three parallel jobs on ubuntu-24.04:

  • rust — cargo fmt --all -- --check, then cargo clippy --workspace --all-targets --locked -- -D warnings, then cargo test --workspace --locked.
  • node — the Claude Code runner's suite (npm ci + npm test) and the telemetry receiver's
    (node --test; it has no dependencies, so it needs no install step).
  • web — npm ci + npm run build (tsc --noEmit && vite build). There is no UI test suite,
    so this is a build check and the README says so.

It follows release.yml's conventions: actions pinned by commit SHA with a # vX.Y.Z comment
(the same pins already in use), rustup toolchain install stable --profile minimal rather than
a setup action, top-level permissions: contents: read, a concurrency group that cancels
superseded PR runs, and timeout-minutes on every job.

A genuinely broken test command, fixed. modules/agents/claude-code's test script was
node --test test/, which fails on Node 24 — the positional argument is treated as a literal
entry file, so it exits 1 in 25ms with Cannot find module '.../claude-code/test' without
running anything. npm test in that package was broken, as was the command documented in
README.md. Bare node --test discovers test/*.test.mjs correctly and passes all 58 tests.
Fixed in the script and the README.

cargo fmt --all across the workspace (26 files, formatting only) and cargo clippy --fix,
so the two new gates go green. The clippy pass touched 10 files: 13 collapsible_if collapses,
one while_let_on_iterator, one manual_contains, one items_after_test_module code motion,
and one hand-written fix for unnecessary_sort_by in memory.rs (sort_by →
sort_by_key(|p| Reverse(...))). No #[allow] was used to silence anything.

rust-version = "1.88" in both crate manifests. The collapsible_if fixes produce
let-chains, which need Rust 1.88 — edition 2024 alone only requires 1.85. Since both crates are
published to crates.io, this is now declared rather than left as an undocumented effective
minimum, so cargo gives consumers a clear error instead of a syntax failure.

README.md — the roadmap row flipped to SHIPPING and reworded to match what CI really does
(three test suites plus a UI build, not a "UI test suite"); the "No CI yet, and nothing is
published to crates.io or npm" bullet rewritten, since only the npm half was still true; and the
Development section now notes CI and shows the npm ci a fresh clone actually needs.

How this was verified

Everything below was run in the sandbox, and then re-run from scratch by a separate agent that
read ci.yml and executed each step's command verbatim:

Gate Result
cargo fmt --all -- --check exit 0
cargo clippy --workspace --all-targets --locked -- -D warnings exit 0, no warnings
cargo test --workspace --locked 100 passed, 0 failed
runner: npm ci + npm test 58 passed, 0 failed
telemetry: node --test 8 passed, 0 failed
web: npm ci + npm run build exit 0

actionlint v1.7.12 reports the workflow clean. The action SHAs were compared literally against
release.yml. The rustup ... --component rustfmt,clippy line was verified empirically rather
than by eye — clippy was removed from the toolchain and the exact workflow line re-run, which
reinstalled it. The old node --test test/ was run under Node 24 to confirm the failure is real
rather than inferred.

Worth a close look

This branch is based on a stale origin/main and will conflict on merge. It could not be
rebased here (git metadata is read-only in the colony). Simulating the merge in a throwaway
clone: HEAD already conflicts with origin/main on 5 files before any of this work —
Cargo.lock, both crate Cargo.tomls, web/src/components/ChatPanel.tsx and
web/src/sessionStream.ts. The workspace-wide reformat adds 5 more, all Rust: headroom.rs,
main.rs, memory.rs, sessions.rs and telemetry.rs (~28 marker blocks). Most are
style-versus-semantic and resolve by taking main's side and re-running cargo fmt, but
memory.rs needs real care — #97 rewrote 253 lines of it inside a region this reformat also
touched. Note that headroom.rs and telemetry.rs are byte-identical on both sides today; the
reformat alone is what breaks that identity. Rebasing onto origin/main and re-running
cargo fmt --all is likely to be much less work than resolving the reformat by hand.

The two Cargo.toml files that gained rust-version are themselves already in that baseline
conflict set.

The roadmap row is marked SHIPPING, which README.md:78-81 defines as "merged, in this
repository, and exercised on a real machine". That is true by the time this merges — GitHub runs
pull_request workflows from the PR's merge ref, so this PR exercises its own CI — but the
claim is slightly ahead of the evidence until that first run is green. Worth confirming the run
before merging.

The clippy autofixes are the only semantic changes in a very large diff, and each was reviewed
individually: the sort_by_key(Reverse(...)) rewrite preserves ordering and tie-breaking (both
are stable sorts on a Copy key), chars.by_ref() keeps the iterator usable after the loop as
the surrounding code requires, and every collapsed condition with an || was parenthesised to
preserve precedence. The remaining ~3900 lines of Rust diff are pure formatting churn and are
best reviewed with whitespace ignored.


🤖 Generated by Colonizer in a microVM

…settlers into a crew (#71)

Implements the Settler Showcase design (claude.ai/design): the ant, the card and
parallel settlers.

The ant (AntAvatar.tsx, index.css) is now a pixel-art worker in side profile,
26 by 20 pixels with crisp edges and a joint for every moving part. The ground,
thought pixels, scribble and lens move a pixel at a time.
- working: a tripod gait, carrying what the tool is (a lens to search or read, a
  leaf to edit, a block to build or install, a flag to test).
- thinking: looks up while a thought fills in, dot by dot.
- writing: nods over a line it writes and rewrites.
- done: hops once and rests with a check.
- paused: sits with its antennae drooped.
- A failed tool result makes it stumble over a pebble, then carry on.
- Each role wears its own mark: a Scout's pack, a Builder's hard hat, an
  Inspector's monocle, and so on.
- Reduced motion freezes every state in a readable pose.

The card (SettlerCard.tsx) keeps the settler's name, task and one status line.
It adds a step count and a trail along its foot that moves while the settler
works, and the frame turns green when the settler is done. Collapsed, a done
card shows the first sentence of its report. Opened, it shows the report as
markdown, then the steps: in plain words with the command or path each acted
on, or the raw transcript in the technical view.

Grouping: settlers the orchestrator sends out together are one crew. Their
events interleave while they work, and each settler now keeps one card for the
whole burst instead of splitting into "continued" cards every time another
settler speaks. A crew's cards hang off one rail under a strip of its ants on a
shared trail ("3 settlers · 2 at work · 1 done"), and their loops are staggered
so they never move in lockstep.

Departures from the design, where it showed data colonies don't send:
- A done card doesn't show a duration, because the stream doesn't record when
  a settler finished.
- A failed step says "didn't work", not "didn't work · retried", because
  nothing says it was retried.

The mock's colony now sends a crew of three (Scout, Tester, Inspector) with a
failing test run, so ?mock=1 shows every part of this.
…e top of the README (#72)

Installing was explained only in the README, about 200 lines down, and the site's
homepage showed `scripts/install.sh` with no word that it has to run inside a clone.
docs/install.md is now the one place for it, mirrored to colonizer.dev/docs/install
alongside vision, architecture and protocol. It covers:

- what a machine needs, including Claude Code on Linux;
- the build-and-run commands, and the two installer options;
- the first run, and what the installer does on a Mac;
- where settings, credentials, colonies and the app live;
- how to update.

The README's "Run it" moves up to right after the introduction: the four commands
and a pointer to the guide. The introduction no longer says Linux only, since an
Apple Silicon Mac has run a colony end to end (#36).

colonizer.dev/docs/install only exists once the website re-syncs its docs after this
merges.
A version tag now publishes a GitHub release with a prebuilt app for Linux x86_64
and one for Apple Silicon, plus an installer:

    curl -fsSL https://colonizer.dev/install.sh | sh

Nothing on the machine needs Rust or Node.js to install it.

.github/workflows/release.yml
- linux-binaries: builds colonizer-agentd and rtk for both colony architectures,
  and the Linux harness, as static musl binaries. They are built inside
  rust:1-alpine with `docker run`, because GitHub's JavaScript actions don't run
  in Alpine containers. A static harness starts on any glibc; one built on the
  Ubuntu 24.04 runner would not start on Debian 12 or Ubuntu 22.04.
- bundle: runs scripts/install.sh --bundle on ubuntu-24.04 and macos-15, then a
  smoke test. The app starts from the unpacked archive, finds its assets there,
  reports agentd, the web UI and the claude-code module, and serves the UI.
- release: runs for tags only. The tag must match crates/colonizer's version. It
  writes SHA256SUMS and publishes the archives with install.sh.

A release contains no Anthropic code. The Claude Agent SDK is "all rights reserved",
and its optional platform packages are Claude Code itself (224 MB).
- The bundle installs the module with --omit=optional. The runner already passes
  pathToClaudeCodeExecutable, so those packages were never used.
- scripts/record-fetch-at-install.mjs takes the SDK out and writes its lockfile
  tarball URL and sha256, after checking the tarball against package-lock.json's
  integrity.
- scripts/install-release.sh fetches the SDK from the npm registry against that
  record. On a Mac it also fetches the linux-arm64 Claude Code build from
  Anthropic's stable channel, checked against the manifest. macOS's own plutil
  reads the manifest, so no Node.js is needed. The Workflow's Package step fails
  if the SDK is found in a bundle.

scripts/install-release.sh (published as install.sh)
- Refuses platforms it can't serve, including Linux without read-write /dev/kvm.
- Checks the archive against SHA256SUMS.
- Swaps ~/.local/share/colonizer/app in whole, and links ~/.local/bin/colonizer.
- Running it again updates in place. It reuses a Claude Code binary that still
  matches the manifest, and never touches settings or colonies.
- Accepts COLONIZER_VERSION, COLONIZER_APP and --pull-image.
- Everything runs inside main(), so a truncated download runs nothing.

scripts/install.sh --bundle skips the KVM check and the Claude Code fetch, and
uses binaries from $COLONIZER_PREBUILT instead of building them.
build-agentd.sh and build-rtk.sh accept COLONIZER_BUILD_HERE=1, to build inside
an existing rust:1-alpine instead of a microVM. rtk's source is then unpacked
outside the repository, because cargo refused to build it as a stray member of
the harness workspace (the local test below caught that).

Tested on an Apple Silicon Mac:
- In a rust:1-alpine microVM with COLONIZER_BUILD_HERE=1: agentd, rtk and the
  harness built as static aarch64 musl binaries. The static harness started in a
  Debian (node:24-bookworm) microVM and served /api/status.
- `install.sh --bundle` with those guest binaries: a 129 MB app, a 44 MB archive,
  and no Agent SDK inside.
- `curl … | sh` against the archive served locally, into an empty HOME, took 41 s.
  It fetched the SDK (sha256 checked) and Claude Code 2.1.267 (manifest checked).
- The installed app reported its assets in the new location: agentd, web,
  claude-code, the guest Claude binary, and msb 0.6.18 from the relocated vendor
  directory. It served the UI.
- Running the installer again reused Claude Code, left no app.new or app.old,
  and kept the data directories.
- The installed colonizer-agentd, rtk and claude-guest each ran `--version`
  inside node:24-bookworm.
- The runner's 58 tests passed against an SDK restored from the npm tarball
  without optional packages.

On GitHub (this pull request's run): both linux-binaries jobs built in rust:1-alpine, and both
bundles passed the smoke test from their unpacked archives (97 MB for linux-x86_64, with headscale and
tailscale; 43 MB for darwin-arm64).

Not tested yet:
- Installing the linux-x86_64 release on a Linux machine.
- Launching a colony from an installed release.
- colonizer.dev/install.sh, which the website adds once a release exists. Until
  then, install from the release URL directly.
docs/install.md opens with "Install a release":
- `curl -fsSL https://colonizer.dev/install.sh | sh`;
- what the installer checks and where it installs;
- the two things it fetches from Anthropic's own channels, since a release
  carries no Anthropic code;
- updating by running it again;
- how to install a particular version, or pull the colony image up front.

The build steps are now "Build from source". "What you need" separates what a
release needs (git, gh, curl, tar) from what a source build adds (Node.js 20+,
Rust 1.88+). "Where things live" and "Updating" cover both ways. The README's
"Run it" leads with the one command, with the source build under it.

v0.1.0 was installed from the published release into an empty HOME on an Apple
Silicon Mac (52 s): it fetched the SDK and Claude Code 2.1.267, started, and
reported agentd, the web UI, the claude-code module, the guest Claude binary
and msb 0.6.18. The Linux release is smoke-tested in CI but hasn't been installed
on a Linux machine yet. colonizer.dev/install.sh is added by the website.
…, and publish them on release (#76)

Reserves the names on crates.io and puts the source there. The crates are not an
install: `cargo install` gives the mothership binary without microsandbox, the
in-VM daemon, the agent module or the web UI. Each crate's README says so and
points to `curl -fsSL https://colonizer.dev/install.sh | sh`. The site and docs
don't mention cargo.

- The harness package is renamed `colonizer-harness`, because `colonizer` on
  crates.io belongs to an unrelated project. Its binary is still `colonizer`, so
  the command, target/release/colonizer and the release archives don't change.
  scripts/install.sh and release.yml build it with `-p colonizer-harness`.
- The Headroom bundle pins move from vendor/vendor.lock into
  crates/colonizer/headroom.lock, because `include_str!` reached outside the crate
  and `cargo publish` packages only the crate. fetch-vendor.sh loses its skip for
  kind `bundle`, which no longer occurs. docs/protocol.md and
  headroom-bundle.yml name the new file.
- Both crates are at 0.1.1 and have a crates.io README, a homepage and a readme
  field.
- release.yml: the release job checks the tag against both crate versions. A new
  crates job, tags only and after the release, publishes whichever of
  colonizer-agentd and colonizer-harness crates.io doesn't have yet. It uses
  Trusted Publishing (rust-lang/crates-io-auth-action v1.0.5, pinned, with
  id-token: write), so no token is stored. crates.io won't create a crate that
  way ("Trusted Publishing tokens do not support creating new crates. Publish the
  crate manually, first"), so the first version is published by hand, and later
  tags publish on their own.

Checked:
- `cargo test --workspace` passes.
- `cargo publish --dry-run --locked` packages and verifies both crates. The
  harness package includes headroom.lock and README.md.
- actionlint passes on release.yml.
…m to 0.1.2 (#77)

The crates.io pages now open like the repository: a banner, a badge row, then
what the crate is.

- assets/banners/banner.html draws both banners in colonizer.dev's colours and
  type: a mono eyebrow, a headline, a lead, a foot line and a diagram leading to
  the command.
  - colonizer-harness: "The mothership.", with the colony graph from the site,
    ending in `colonizer`.
  - colonizer-agentd: "Inside every colony.", showing a colony microVM with agentd
    between Claude Code, the event log and the terminal, and the private mesh up
    to the mothership.
- scripts/render-crate-banners.sh renders them at 2560×800 with headless Chrome
  into assets/banners/<crate>.png. The READMEs link them by their
  github.com/ghraw URL on main, because crates.io renders a README
  outside the repository.
- crates/colonizer/README.md and crates/colonizer-agentd/README.md:
  - banner and badges: crates.io version, latest release, docs, MIT;
  - what each crate is, with the install command, and why the crate alone isn't
    an install;
  - for the harness, what Colonizer does and the two crates;
  - for agentd, what it does and its four endpoints.
  The claims come from the README and docs/protocol.md §3.
- Both crates are at 0.1.2: crates.io shows a README only with a new version.
  Tagging v0.1.2 publishes them through Trusted Publishing.

`cargo publish --dry-run` packages and verifies both crates.
…laiming no runtime downloads (#78)

docs/vision.md defined a settler as "the agent module working inside a colony".
Since #70 and #71, the colony chat also calls the subagents that module sends out
settlers, named for their role (Scout, Builder, Inspector…), and colonizer.dev is
getting a page about colonies and settlers. The vocabulary now covers both: the
agent module is a colony's first settler, and the subagents it sends out are
settlers too, each with a role.

Principle 4 said "No runtime downloads". That hasn't been true for a while:
microsandbox pulls a colony image the first time a colony needs it, and
switching Headroom on downloads its bundle. docs/install.md already says so.
The principle now keeps what holds (your hardware, its own dependencies, no
cloud account) and names the two downloads after install.
…itches it on (#79)

colonizer.dev/live will show a dot for every area, about 25 km across, where a
mothership is online, lit while its colonies work. This adds both ends.

The receiver, services/telemetry, is a Cloudflare Worker with a D1 table,
deployed at telemetry.colonizer.dev:
- POST /v1/heartbeat takes {install_id, version, platform, colonies} and ignores
  any other field. The location is Cloudflare's own estimate for the request,
  snapped to a 25 km grid cell; only the cell is stored. The install id is kept
  as a SHA-256 hash. The IP address is used, in memory, for rate limiting only.
  {install_id, online: false} deletes the row.
- GET /v1/presence is the public view: totals and counts per cell, never per
  mothership. Online means a heartbeat in the last 12 minutes.
- Rows older than an hour are pruned during heartbeats. A cron trigger would
  need a workers.dev subdomain on the account, and this needs nothing.
- The privacy-relevant logic is in src/presence.js with node tests.

The mothership, crates/colonizer/src/telemetry.rs:
- Off until the user answers. The answer and a random v4 install id live in
  <config>/telemetry.json. The id is created on switching on and forgotten on
  switching off, so two periods on the map can't be linked.
- While on, a heartbeat every 5 minutes (the service may name 60 s to 1 h), and
  within a minute of the colony count changing. Switching off and shutting down
  send online: false, so the dot goes at once.
- DO_NOT_TRACK or COLONIZER_TELEMETRY=off keep it off whatever Settings says;
  COLONIZER_TELEMETRY_URL points it elsewhere.
- GET/PUT /api/telemetry return the status and the exact next heartbeat.

The web UI asks once, after GitHub and Claude are connected, with "Show on the
map" / "No thanks" / "What is sent". Settings has a Live map section with the
switch and the heartbeat as JSON.

docs/telemetry.md says what is sent, what is kept and for how long, what is
public, and how to keep it off; the README's trust model, configuration and
development sections and docs/protocol.md point to it.

The crates are bumped to 0.1.3, the first release with the live map.
Colonizer session 0ee041a4

Co-Authored-By: Colonizer <noreply@colonizer.dev>
@ParallelEntrepreneur

Copy link
Copy Markdown
Collaborator Author

Superseded by #142.

This branch's only new commit was a whole-tree cargo fmt plus the gate, on a base that predated eight commits since squashed onto main — which is why it showed 71 changed files and would not merge. The reflow is machine-generated, so #142 redoes it against main instead of rebasing ~4,000 conflicting lines of pure whitespace.

One deliberate difference: #142 adds a rustfmt.toml pinning width 130 rather than taking rustfmt's 100-column default. The tree was written wider (99% of its lines already fit 129 columns), and at 100 the reflow is nearly twice the churn. The CI half of this branch is already on main as #138.

@ParallelEntrepreneur
ParallelEntrepreneur deleted the colonizer/session-0ee041a4 branch September 18, 2026 10:48
ParallelEntrepreneur added a commit that referenced this pull request Sep 23, 2026
#138 left the fmt gate out on purpose: reformatting the whole tree is a
dedicated change, not a drive-by. This is that change, redone against main
rather than rebasing #139's version, whose base predated eight merged
commits.

`rustfmt.toml` sets width 130 instead of taking rustfmt's 100-column
default. The tree was written wider — 99% of its lines already fit in 129
columns — and at 100 the one-time reflow is nearly twice the churn, for
lines nobody asked to be broken. Everything else is rustfmt's default.

Nothing but whitespace changed: `cargo test --workspace` is 240 passed
before and after, and clippy stays clean. CI runs `cargo fmt --all --check`
after the tests and the lint, so a formatting nit never hides a real
failure, and the rust job's toolchain now installs rustfmt and clippy
rather than relying on what the runner image happens to ship.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant