kibble

Dogfood your docs.

Verify your documentation in CI. Runs install instructions and examples in a clean container, then checks documented commands and flags against the installed tool.

VERIFIED · FAIL · GAP · DRIFT · BLOCKED · SKIP

myrepo
  ! brew             0s  GAP  no formula, cask, or alias named "acme/tap/nope" in the c…
  – doc-coverage     0s  SKIP  no command list in the help output to check against
  ✗ example         22s  FAIL  b1 "mytool --nosuchflag" exited 2: flag provided but not d…
      myrepo/README.md:17
      $ mytool --nosuchflag
        exited 2: flag provided but not defined: -nosuchflag
  ✓ flag-check       0s  2 cited flags ok, 0 subcommands cited
  ✓ go-install      22s  v1.4.0

FAILED  a documented line ran and did not work
2 passed  1 failed  1 gap  1 other  5 checks in 45s

kibble does not check whether your documentation exists. It checks whether the interface your documentation promises still exists. It reads your README, everything in docs/, and the instructional documents beside them, runs what they tell a reader to run in a clean Linux container from zero, and puts the flags and subcommands they cite to the binary it just installed. A page still teaching a flag the tool dropped is caught as drift, even though nothing failed to run.

What comes back is a verdict, not a pass. kibble separates documentation that is wrong from documentation it could not check, so a green run means something specific rather than nothing went visibly bang. That distinction is pinned in both directions: a corpus of real repositories proves correct documentation passes, and deliberate one-line corruptions of those same documents prove rot gets caught. A verifier that only ever agrees with you is worse than no verifier, so kibble is tested against its own willingness to say yes.

What rots, and who reads it

The stranger is not always a person now. Coding agents install tools by doing what the README says, and they fail differently than people do. Someone who follows a broken instruction knows they followed it correctly, reads the error, and works around the document. An agent cannot tell a stale command from its own mistake, so it retries, invents variants, and reports success it did not have. A line that has been wrong for six months gets run all day by something that will never complain about it. Stale documentation used to cost a reader ten minutes. Now it is instructions a machine executes without ever telling you they were wrong.

Your README tells people to run go install ..., then some setup, then a quickstart. Every one of those rots the moment the code moves, and you are the last to know. kibble reads those steps from the README and from everything instructional beside it: a docs tree, a getting-started, an install guide, a tutorial, a cookbook, an FAQ. Those rot faster than the front page, because they are where instructions go once it fills up and nobody reads them on the way past. Documents that describe the project rather than instruct a reader are left alone, since running a contributing guide's test commands proves nothing about the docs a user follows. A piped shell installer and a system package install are recorded but not run, and reported as skips with a reason, so an install kibble declines to execute is still visible rather than silently missed.

Commands are not the only thing that rots. kibble takes the flags and subcommands your documents cite and puts them to the binary it just installed, so a page still teaching a flag the tool dropped is caught as drift rather than passing because nobody happened to run that line. It also reads the binary's own help and reports the documented surface against the real one.

That container is one environment, not every environment. kibble answers whether your documentation works from zero in a reproducible Linux container, which is a narrower claim than whether it works for every reader: a macOS path, an ARM host, musl, a corporate proxy, and a private registry are all outside what it sees. The narrow claim is the one worth making, because it is the one kibble can actually settle.

Install

go install github.com/dcadolph/kibble@latest

Requires Docker on the host. A drop-in replacement that speaks docker's command line, such as Podman, works through KIBBLE_DOCKER=podman.

Usage

Point it at one or more repository directories, or none at all:

kibble
kibble ./myrepo
kibble ./repo-a ./repo-b

With no path it checks the directory you are standing in, so cd into a project and run it. There are no prompts. kibble's home is a CI job, and a tool that stops to ask a question there either hangs on a closed pipe or needs a flag to defeat it.

A run that finds something says where it is:

myrepo
  ! brew             0s  GAP  no formula, cask, or alias named "acme/tap/nope" in the c…
  – doc-coverage     0s  SKIP  no command list in the help output to check against
  ✗ example         22s  FAIL  b1 "mytool --nosuchflag" exited 2: flag provided but not d…
      myrepo/README.md:17
      $ mytool --nosuchflag
        exited 2: flag provided but not defined: -nosuchflag
  ✓ flag-check       0s  2 cited flags ok, 0 subcommands cited
  ✓ go-install      22s  v1.4.0

FAILED  a documented line ran and did not work
2 passed  1 failed  1 gap  1 other  5 checks in 45s

The file and line under a failure are the point. A verdict a reader has to go hunting for in a log costs them the search, so the broken line, where it is written, and what the tool said are printed together. In CI the same three become an annotation on that README line.

Use it in CI

Add a workflow that fails a pull request when a documented install breaks:

name: docs
on: pull_request
jobs:
  kibble:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dcadolph/kibble@v0.0.5
        with:
          repo: .
          # args: -strict   # fail on timeouts, smoke failures, drift, and gaps too

The runner already has Docker. A failed install or example is annotated on the exact README line that broke, so it shows up inline in the pull request the way a failing test does; doc drift becomes a warning annotation, and the job summary gets the full results table. kibble runs on its own README this way on every commit.

Pin an exact release tag so a run stays reproducible a year from now: the action downloads the released binary for that version and verifies its checksum before running it, so a CI log can always say which kibble produced its verdict.

Flags

Flag	Default	What
-image	golang:1.26	Fallback image when no toolchain is detected.
-timeout	240s	Per-step build timeout.
-workers	3	Max concurrent installs.
-json	false	Emit results as JSON to stdout.
-version	false	Print the version and exit.
-strict	false	Also fail on timeouts, smoke failures, drift, and gaps.
-examples	true	Replay each document's example blocks in the container.
-plan	false	Print the example plans as JSON and exit.
-suggest	false	Propose a .kibble.yml using a model and exit.
-mcp	false	Serve the Model Context Protocol over stdio.
-brew-install	false	Run documented brew installs for real instead of checking the formula exists.

One dash or two, either works: -strict and --strict name the same flag.

What the verdicts mean

kibble runs installs from zero, smoke-tests what lands, replays quickstarts in one session, and checks cited flags and subcommands against the binary's own help. Every result is one verdict, and the boundaries between them are the product. Only one verdict claims the documented step works:

• VERIFIED built or installed and the binary answered a smoke test. It is the only green verdict.
• BUILT, RAN, EXISTS, and CROSS-ARCH all mean a step ran or a package exists but nothing proved it works: BUILT compiled but the smoke test failed, RAN finished but produced no binary to test, EXISTS confirms a package is in its index without installing it, and CROSS-ARCH installed a binary for another architecture this host cannot run. None of them is a pass, and none is an accusation.
• FAIL ran and did not work, or named a package that does not exist.
• SKIP means kibble chose not to run the line, before running it, on evidence it holds: a placeholder to fill in, a block scoped to another platform, a command the shell reported missing. A machine-readable reason says which.
• BLOCKED means kibble ran the line and could not establish whether it works. The output resembled a condition kibble excuses, and resembling is all it did: a 403 is a missing account or a wrong argument, a refused connection is an absent service or a misnamed port, an exit 1 with no output is a search finding nothing or a command dying quietly. A skip would claim the document is fine. BLOCKED claims nothing.
• GAP means the document is incomplete: a file, directory, or setting nothing creates.
• DRIFT means the docs cite a flag or subcommand the binary no longer has.
• TIMEOUT and ERROR keep slow networks and kibble's own trouble out of your verdict.

Every verdict also carries a coarse bucket (works, unverified, broken, doc-drift, not-attempted, inconclusive) so a consumer can condense the fine verdicts without hard-coding them. By default only FAIL and ERROR fail a run; -strict also fails everything outside the works and not-attempted buckets, so an unverified step, a drift, or a gap all count when you demand proof. The full reasoning, including why a check that cries wolf is worse than no check, is in docs/DESIGN.md.

Why believe any of it

A verifier is worth exactly what its false positives and false negatives are worth, so both are pinned and enforced weekly against real repositories.

corpus/repos.tsv holds ripgrep, fd, glow, nodemon, httpie, and rich-cli, each at a fixed commit with the verdict counts a person read the documentation and verified by hand. A scheduled run fails when any count moves, because a moved count means kibble's judgment changed and the repository did not.

corpus/mutations.tsv runs the other direction. Each row corrupts exactly one documented line of a pinned repository, a crate name typo, a brew formula that no longer resolves, and requires kibble to reach a named verdict on the damage in a finding that quotes the mutated token. A count alone would not do, since a coincidence can produce a count.

ripgrep   5,0,1  as expected      ripgrep mutation:  FAIL=1 naming ripgreppp
fd        7,0,0  as expected      rich-cli mutation: GAP=2  naming richhh
rich-cli  3,0,1  as expected

Correct documentation passes and one edit of rot flips it. Both halves are the claim; a tool that only holds one of them is either a rubber stamp or a nuisance. The same property is pinned below the integration layer by a unit suite that mutates a healthy document one edit at a time and requires the untouched copy to come back clean first. Reasoning in docs/DESIGN.md.

What kibble does not check

The container is Linux, x86 or ARM depending on the runner, with a network and no credentials. Documentation that only breaks on macOS or Windows, only under musl, only behind a proxy or a corporate CA, or only with a private registry configured, is documentation kibble cannot speak to. A block whose own prose scopes it to another operating system is skipped and said so, rather than run and blamed. Nothing here is a verdict about your reader's laptop; it is a verdict about a clean machine.

Security

kibble executes commands it read out of documentation, which means a README is untrusted input: anyone who can change the docs can change what runs. Every command runs in a fresh container with nothing mounted from the host, most Linux capabilities dropped, and no-new-privileges set. It is not an unprivileged container: the session is root inside it, because installing the packages a document depends on needs to be. The network stays open too, because verifying an install is fetching it. Treat a kibble run the way you treat a build script, and read docs/SECURITY.md before pointing it at a repository you do not trust.

Configuration

Most repositories need none. When a heuristic cannot settle a call, a .kibble.yml at the repository root does: fixtures, environment, substitutions, background services with readiness probes, and per-line run or skip rules, so the run stays reproducible and the engine stays the thing that decides pass or fail. -suggest has a model draft the file for you to review; -mcp serves the same engine to an agent. All of it is in docs/CONFIG.md.

Roadmap

• JUnit XML output for CI systems that are not GitHub.

Why "kibble"

Eating your own dog food means using what you ship the way a stranger would. Nobody does it for documentation. Your machine already has everything installed, and you stopped reading your own docs the day the workflow became muscle memory. kibble is the bowl: it feeds your docs back to a fresh machine and tells you whether they still go down.

License

MIT. See LICENSE.