M1 The toolkit

[tamnd]

Roughly fourteen weeks. The milestone with no reader visible output, and therefore the one most likely to get cut short when a deadline looms.

It must not be. There are 103 lessons and 60 blueprints coming, which is far past the point where hand work scales. Every later milestone borrows its speed from this one, and a weak toolkit here shows up as three parts that ship late.

Exit criterion. All nine visual primitives exist in both widget form and animation form, driven from the same kxray.models objects. bpc generates sections 2, 5 and 7 for five blueprints. kxdiff demonstrates all five strictness levels on real traces. The whole CI matrix is green.

kxray

• kxray.btf: types, fields, offsets, sizes, enum values, function prototypes, and the type tags for __user, __rcu, __percpu and __iomem. All nineteen kinds read, holes found, bitfields in both encodings, ops tables picked out of a struct of function pointers. __iomem is the one that cannot be delivered as written: it is a sparse annotation the compiler drops, so BTF never records it, and asking for it is refused with the reason instead of answered with an empty list. The other three are queryable through Btf.annotated. One caveat that stays open: everything here has only ever been run against the handwritten fixture, because there is no built kernel yet. The first real /sys/kernel/btf/vmlinux read is what will find whatever this got wrong.
• kxray.trace: function_graph, the flat function tracer, and trace_pipe events read through each event's own format file rather than a hard coded layout. All three are in, across Ask BTF which pointers the kernel is not allowed to follow #55, Read the flat function tracer, and the flags column it prints #56 and Read trace events through each event's own format file #57. function_graph gives the shape and the durations, the flat tracer gives the flags column and so the execution context, and events give named fields read through the layout the kernel publishes for them. Flags.context maps the column onto five of the six contexts in kxray/vocabulary.py and refuses to guess at atomic, because a held spinlock and a bare preempt_disable() raise the same counter and the column cannot tell them apart. The event reader loads /sys/kernel/tracing/events/<group>/<event>/format, so prev_state being four bytes on this 32-bit box and eight on a 64-bit one is read correctly on both.
• kxray.proc: the /proc and /sys files the lessons actually use, each parser recording the ABI stability class of the file it read. In with Read /proc, and record what the kernel promises about each file #58. Five readers for four file shapes and eight real Tier 0 captures, and every read carries a Promise from kxray/proc/stability.py citing the file in the kernel tree that makes the claim. The number that made the ledger worth building: on 7.2.2 there are 685 files under Documentation/ABI and six of them describe a path in /proc, and not one of those six is a file anybody reads.
• kxray.source: symbol to file and line, MAINTAINERS lookup, Kconfig symbols, the syscall tables, and citations with their context hash (Read the kernel's own source files #59)
• kxray.models: the shared intermediate representation, so a trace renders as a tape, an animation and a blueprint table without three parsers (One arrangement of a trace, instead of three #60). Two pieces. The 1424 line module is a six file package now, split by subject, with a re-exporting __init__ so every caller was left alone. And the reason for the split: kxshapes/scene.py is the one conversion from a trace to a picture, where there used to be three. The tape widget built its own cells and wrote its own hover text, the blueprint compiler printed a tree and drew nothing, and a storyboard listed the shapes it used in a hand written field with no capture behind it. All three take a Scene now, and the storyboard field is checked against a scene built from the file it names, which is what stops a beat claiming CPU lanes over a uniprocessor capture.
• kxray.replay: stepped playback of a recorded Tier 1 session, with the first recording, Linux 7.2.2 built for User Mode Linux and booted in the pinned container (kxray.replay: step through a recorded Tier 1 session #61)
• kxray.corpus: loading, plus the normalisation that makes any of this diffable (kxray.corpus: load a pinned capture, and take the run out of it #62)

kxbox

• Three boot profiles: teaching, lockdep and memcheck (kxbox: three boot profiles, and the kasan one cannot exist #63. Not kasan. arch/x86/Kconfig selects HAVE_ARCH_KASAN only if X86_64 and Tier 0 is 32 bit, so the symbol does not exist here and Kconfig drops it without a word. config/lockdep.config had been asking for it since it was written and no kernel this project built ever had it. The third profile is KFENCE, page poisoning and the slab debugger instead, kxbox/profiles.py is the registry, and boot() now refuses a name nothing builds.)
• The bridge API: boot, sh, read, trace, insmod (kxbox: the bridge API on both backends, checked by signature #65. All five existed, two of them did not work the same way on both backends, and nothing was checking. insmod on a recording looked the recipe up under a name no recipe has ever used, so that path had never worked at all, and max_depth, documented as the fix for a guest that appears to hang, was on the live backend only and could not be reached from a lesson. What the PR really adds is a test comparing the two backends signature by signature, plus one asserting Box forwards every argument, because an argument one backend has and the other does not fails on the reader's machine rather than on the author's. write is left off a Box on purpose and there is a test saying so.)
• The rootfs recipe, with busybox, strace and our own tools (rootfs: strace in the image, built from source and pinned by checksum #66. strace was the missing third. Built from the release tarball rather than downloaded, because nobody publishes a static i686 one, which cost about a megabyte compressed and thirty seven minutes of qemu translating x86 one instruction at a time. Both are measured and written down in RESULTS.md, and the binary is cached so the second cost happens once. What it buys is the one question a function graph tape cannot answer: what the program asked for, with the arguments, the flags and the errno on the way back. On writebyte it shows one write between the two writes to tracing_on, which is the claim all three of our programs are built on and which nothing was checking.)
• The banner, showing version, architecture, profile, preemption model and which backend is live (banner: every line says where its fact came from #67. All five, and the part worth having is that every line says where its fact came from. The release and the preemption model are read off /proc/version on both backends, never filled in from pin.toml, and when the running kernel disagrees with the pin the banner says so. The architecture is copied out of the config fragments and the line says per its fragments, because nothing in the guest names one and printing three facts in one voice would let a reader believe a config file had been checked against something. Two hardcoded claims went away in the process. Verified against a real v86 boot in headless Chrome, which is the only place the live wording exists, and the page now prints it so a person can look at it.)

kxwidgets and kxmanim

• The nine primitives, once each, shared between widgets and animations. They live in kxshapes as data, kxwidgets/shapes.py draws them as HTML, kxmanim draws them as video, and a test asserts the widget side covers all nine. SyscallTape and OpsExplorer were rewired to go through the shapes rather than keep their own copies, so there is one implementation of a trace cell and one of a socket.
• The four signature artifacts: the Syscall Tape, the Layer Descent, the Lock Timeline and the Structure Map (SyscallTape, Descent, LockTimeline, and StructMap with ObjectGraph for the pointer side. The Lock Timeline needed a new capture, in corpora/traces/tier1/contended-lock.txt, because there was no evidence of a contended lock anywhere in the repository. It will not call a wait contention unless it has been told the clock behind the trace was real.)
• Tape diff mode, since one trace is data and two traces are a transformation (TapeDiff. The verdict is kxdiff's. What the widget adds is the ring on functions that are only on one side, the faded boxes for frames the policy took out of the comparison, and shared_scale for when the question is whether something got faster.)

Checkers

• bpc, generating sections 2, 5 and 7, failing the build on any diff. The tool is done and there are two layers against a hand edit: the seal hash, and a dry run regeneration that catches an edit which was resealed afterwards. The exit criterion asks for five blueprints and there are four, page-fault, write-path, lock-ordering and trace-ring-buffer, so the remaining work on this line is one more blueprint rather than any compiler. Each of the last two made the tool better anyway. The third: section 5 could only draw a function_graph capture, and every other kind of artefact rendered as one line reading Tracer unknown, so a blueprint resting on a lock report and two /proc readings had nothing observable in it. It now dispatches on the same routing table tools/baseline accounts lines through, and draws a before and after pair of one file as a single table with the change in it. The fourth: a stats file full of counters that answer no question on their own, so the generator now does the addition and the division and prints what was written, what share of it was thrown away, and how much history the buffer was holding.
• kxdiff, with exact, sequence, set, counters and distribution modes. The four name based levels are a chain and a test checks that on the real corpus. distribution is deliberately outside it, because it is about the numbers every other level throws away.
• claimledger, enforcing evidence, the config and architecture declaration, and the cap of two unobservable claims per lesson. The declaration half was the part that was missing. A lesson says which kernel, architecture, profile and tier it is about, an artefact says which machine it came off, and nothing compared the two. Now the profile has to be one pin.toml builds, the version is read through that profile so C-longterm is held to the fallback tree, and evidence that disagrees has to carry a why_not_pinned reason. It found four claims, including one whose evidence kind meant nothing was checking it at all.
• tools/coverage, so a lesson citing a subsystem with no owner fails CI
• tools/lintnb, enforcing the seven point notebook contract
• The unparsed line baseline, so parser drift becomes a build failure instead of something a reader finds. Every line of every committed artefact is now read, skipped or unparsed, the three have to add up to the length of the file, and corpora/BASELINE.toml holds the numbers. The bucket that matters turned out to be skipped, not unparsed: four parsers had a bare continue in the middle of a loop, and a line one of those walks past looks exactly like a line that was never there.

[tamnd]

kxdiff, in #45

First piece of M1 done. It is the one named in the exit criterion, it did not exist, and it is the natural next step from what #44 learned about comparing traces.

The idea it is built on. Comparing two traces is two questions and the old code answered both in one function. What are we comparing, meaning what to leave out first, is now kxdiff/policy.py. How strictly to compare what is left is kxdiff/levels.py. Keeping them apart matters because the comparison in #44 collected seven concessions on the way to passing, and every one of those fixes landed in the same place as the rules about what a trace means. After a while nobody could tell which lines were the check and which were the excuses.

All five levels, which is what the criterion asks for. exact, sequence, counters and set are a chain, and a test checks that on the committed traces rather than trusting the argument. distribution is outside the chain on purpose. It is about the numbers, which every other level throws away, so it can agree where sequence disagrees and the other way round.

python -m kxdiff --demo shows exactly that happening on two real recipes:

write-1byte against two-writes
  exact         differs  9 difference(s)
  sequence      differs  3 difference(s)
  counters      differs  9 difference(s)
  set           differs  6 difference(s)
  distribution  same     0 difference(s)

two-writes writes to a pipe as well as to a file, so it reaches functions write-1byte never does and every name based level says so. The time still lands in the same places in the same proportions, so distribution says they match. That is the opposite answer and it is the right one for the question it is asking, which is the whole reason it is not just a looser set.

Shares and not durations. distribution compares what fraction of the traced time each function spent in itself. It has to, because the emulator and a real machine disagree about the length of a microsecond by a factor of four. Absolute times do not survive that and proportions do.

What this does to kxbox/bothways.py. It is now three lines of intent and a call. It says which comparison it is running and kxdiff says what that comparison means. Both of those lines are pinned by a test, because the way a check like this dies is somebody loosening a constant to make a red build go green.

Still true in a browser. Wheel rebuilt with kxdiff in it and the emulator against recording comparison re-run on a real page. write-1byte, two-writes and page-fault all still agree, over 50, 62 and 41 calls.

Where M1 is

One box of twenty two. That reads worse than it is, because several of the kxray and kxbox boxes have real work behind them from M0 and are waiting on the last piece rather than on the first. I have not ticked those, since a box ticked early is worse than a box left alone.

Next is bpc generating sections 2, 5 and 7, which is the other named part of the exit criterion and the one that decides whether 60 blueprints is a plausible number.

M0 in #1 still has five boxes and all five need a person rather than a machine: four reviewed-by lines, three beginner testers on three devices, and two browser rows that need hardware I do not have.

[tamnd]

PR #46 is merged. The write path blueprint is in, and the bpc box is ticked.

Ticking it needs a word of explanation, because the box and the exit criterion are not the same claim. The box says bpc generates sections 2, 5 and 7 and fails the build on any diff, and that has been true for a while. The exit criterion says it does that for five blueprints, and until today there was one. So the honest reading was that the compiler was finished and the thing it compiles was not, and the way to find out which of those two was the real problem was to write a second blueprint and see what broke.

What the second one found out

The compiler held up. Sections 2, 5 and 7 of write-path came out of the pinned kernel's BTF and the two write captures with no changes to bpc at all, including a struct file_operations table with thirty two slots in it that nobody had asked it to render before.

What did not hold up was three things I had written from memory, all caught by reading the source with the tree unpacked:

• the cache flush before the copy happens before the copy, and there is a second unconditional one after it
• the fatal signal check is at the top of each turn of the loop, before anything is locked
• write_end returning zero does not end the write, it halves the chunk size and tries again, and only gives up with EFAULT when the source buffer cannot be faulted in either

None of those are subtle once you look. All three were wrong in the first draft. That is the argument for confirming citations against a real tree rather than treating them as decoration, and it is worth writing down while it is still fresh.

The citation that could not be written

One had to go. The line that caps the write size is if (count > MAX_RW_COUNT), and that exact line appears three times in fs/read_write.c: once in vfs_read, once in vfs_write and once in do_sendfile. refcheck rejects an anchor that matches more than once, on the grounds that the first match wins and nobody finds out when the wrong one was picked, and there is no longer anchor available because the surrounding lines are identical too.

So the cap is cited by the function that applies it and by the value itself, and the citations file carries a comment saying why the obvious citation is absent. An anchor that resolves confidently to the read path would have been worse than no anchor, and this is the first time the rule has cost something rather than caught something.

refcheck now reports 73 citations across the two blueprints with none waiting, which is the first time the whole directory has been in that state.

Where this leaves M1

The remaining work on the bpc line is three more blueprints, not compiler work. On the evidence of this one that is writing rather than building, which is the answer I wanted, because writing scales with time and a compiler that needs a change per blueprint does not.

Next is the nine visual primitives, which is the other half of the exit criterion and the half that decides whether 103 lessons is a plausible number. kxwidgets and kxmanim are supposed to share them, driven from the same kxray.models objects, and until that is true every lesson costs a widget and an animation written separately.

[tamnd]

The nine shapes now live somewhere both renderers can reach them

Merged in #47. This is the first half of the visual primitives box, and it is a move rather than new drawing, so it is worth saying what the move was for.

The nine shapes were in kxmanim/primitives.py. They are a frame card, a layer band, an object box, a pointer thread, an ops plug, a trace cell, a CPU lane, a context badge and a memory slot, held as data rather than as drawing, each one working out its own rows and labels and alt text.

Being inside kxmanim was fine while animations were the only thing drawing them. Reading kxwidgets showed that is not the situation. There are four widgets there and not one of them touches the shapes. SyscallTape, StructMap, OpsExplorer and PredictionGate each derive their own layout from kxray.models and kxray.layout independently.

So the arithmetic was about to exist twice, once per renderer, and that is the thing the exit criterion is actually protecting against. Two renderers doing their own arithmetic can disagree, and the disagreement is silent. Both pictures render, both look reasonable, and one of them has a field at the wrong offset or draws an empty function pointer slot as filled. Nothing fails. A reader who trusts the widget and a reader who trusts the video learn different things, and neither of them has any way to tell which one got the true version.

What is in this pull request:

• kxmanim/primitives.py becomes kxshapes/__init__.py, code unchanged
• kxmanim stops re-exporting the shapes, on purpose, because a shape reached through the animation package is a shape a widget author will not go looking for
• the tests split the same way the code did, so tests/test_kxshapes.py holds the shape rules and tests/test_kxmanim.py keeps the budget, the still, the captions and the adapter
• LAYOUT.md and CONTRIBUTING.md say where the shapes are and tell a widget author to reach for one before writing layout arithmetic

The test asserting there are exactly nine moved file and still passes, which is the check that the set stayed closed across the move.

What is still missing, honestly

No widget draws a shape yet. Counting what the existing widgets cover in effect rather than by name: OpsExplorer is an ops plug, SyscallTape is trace cells and CPU lanes, and StructMap is part of an object box. That leaves the frame card, the layer band, the pointer thread, the context badge, and proper object box and memory slot renderers with no widget form at all.

So the box stays unticked. It says all nine in both widget form and animation form, and this pull request gets the definition of the nine into a place where a widget can use it, which is a precondition rather than the thing itself.

Next is the widget renderers, one per shape, driven from kxshapes. Each has to draw itself as text as well as as HTML, because the text is what a screen reader gets and what a test asserts on, and each goes on the preview page in kxwidgets/__main__.py in the same pull request, since a visual thing nobody looks at goes wrong quietly. That is the pull request that ticks the box.

[tamnd]

All nine shapes now draw in both mediums

Merged in #48, and the box for the nine primitives is ticked.

The last comment ended with the shapes moved into kxshapes and no widget drawing any of them. This is the other half. kxwidgets/shapes.py is one HTML renderer per shape, and each one decides nothing: colours out of kxray/vocabulary.py, geometry out of kxshapes, and what is left is tags and inline styling.

The first test in tests/test_widget_shapes.py asserts the renderer set is all nine. That is the exit criterion written as an assertion rather than as a note in an issue, so it stays true.

The five that had no widget at all

Descent is the picture the whole book is organised around, and it took the longest to get right for one reason. A write goes userspace, syscall entry, VFS, filesystem, page cache, and returns. It never reaches the block layer, the driver or the disk. The first version drew only the bands the call touched, which looked cleaner and threw away the lesson, so the untouched bands are drawn in grey and the reader can see what was skipped. Frame cards sit under the band they ran in, coloured by subsystem and badged by context, so which layer and which context are the same picture.

ObjectGraph draws structs as boxes of named fields with the pointers between them. StructMap beside it draws one struct as its bytes, and that is a different question: padding, holes, cache lines. This one answers how the kernel gets from a file descriptor to bytes on a disk. Line style is why it exists. Solid means a reference is held, dashed means one is not, dotted means RCU and the answer is stale the moment you leave the read side. A picture that draws all three the same way has dropped the only part that decides whether the code is correct.

MemoryScale puts a struct file, a page and a huge page on one scale, which is a ratio of about nine thousand to one. There is a test that checks the small one stays above a third of the width where a linear scale would put it at a thousandth, and every row prints the real byte count, because a log scale that does not admit to being one is a wrong picture rather than a simplified one.

ContextKey is the legend for the six contexts and what each forbids. Rule 7 says concurrency is never deferred, and the badges are how that shows up everywhere else. A reader meeting one on a frame card had nowhere to look it up.

The fifth, the pointer thread, is inside ObjectGraph rather than a widget of its own, because a pointer with no objects at either end is not a picture of anything.

The part that makes it worth doing

Two existing widgets were rewired rather than left alone.

SyscallTape builds kxshapes.TraceCell now and hands them to the cell renderer. Its own box drawing is deleted. It also grew by_cpu, which draws one lane per CPU instead of one block per outermost call, because the trace file interleaves every CPU into a single stream and a reader following the indentation down the page can be following two call stacks at once with nothing saying so.

OpsExplorer grew compact, which draws the ops plug: a row of sockets, hollow where nobody has looked. A hollow socket is not a null pointer and the drawing says so, because what sits in a function pointer is a fact about a running kernel and BTF only knows the socket exists.

Both have a test asserting the widget output contains exactly what the shape renderer produced. If either starts working its own geometry out again, that test fails. That is the actual guarantee behind the criterion, and it is stronger than the criterion asked for.

Where M1 stands

• kxdiff, done, all five levels
• bpc, tool done, 2 of the 5 blueprints the criterion wants
• the nine primitives, done, both mediums
• the four signature artifacts, next
• tape diff mode, claimledger hardening, tools/coverage, tools/lintnb, the unparsed line baseline, and the kxray module rows still open

Next is the four signature artifacts. Three of the four are now mostly assembly rather than invention: the Syscall Tape is SyscallTape, the Layer Descent is Descent, and the Structure Map is StructMap and ObjectGraph between them. The Lock Timeline is the one with real work left, because it needs two CPU lanes with a contended lock between them, and the pinned kernel is uniprocessor. So the honest version of that artifact has to say where its second CPU came from, and that is a question worth answering in the open rather than drawing around.

[tamnd]

#49 is merged and the four signature artifacts are done.

Three of the four were assembly. The Syscall Tape is SyscallTape, the Layer Descent is Descent, and the Structure Map is StructMap and ObjectGraph between them, depending on whether the question is about bytes or about pointers. All three were mostly built already once the nine shapes moved into kxshapes.

The Lock Timeline was not assembly, and the reason is worth writing down.

There was no evidence of contention anywhere in this repository

The last comment said the problem was where the second CPU comes from, since the pinned kernel is uniprocessor. That turned out to be only half of it. There was already a Tier 1 capture with six CPUs in it, multi-cpu-write.txt, so the second CPU was not the hard part. Running the widget on it found twelve locks and not one of them contended, which is correct and is also a picture of nothing happening. Six writers on six different files take six different inode locks, so there was nobody to wait for.

So a new capture, corpora/traces/tier1/contended-lock.txt. It is the same recipe with one thing changed: four writers on one file rather than six writers on six files. One inode means one lock, and everybody after the first one queues.

task	cpu	waited	held
dd-2791	1	0.25 us	29626.70 us
dd-2792	2	10435.87 us	4298.67 us
dd-2790	3	14778.45 us	6541.71 us
contain-1214	3	0.333 us	6.75 us
dd-2793	0	0.25 us	5039.04 us

contain-1214 is a bystander that happened to be writing to a file nobody else wanted, caught in the same capture a few hundred lines away. Waiting and not waiting, four orders of magnitude apart, on one machine, in one file.

The best line in it is not a number. dd-2790 opens down_write() { on CPU 3, and the very next line in the file is a task switch banner away from it. Waiting for a rwsem means going to sleep, and here you can watch a process ask for a lock, not get it, and lose the CPU inside the call. That is Rule 7 paying for itself: concurrency was not deferred, and what came back was better than the diagram would have been.

The bug this found, which is the useful part

The first version of contended was a property with a threshold. Running it on the Tier 0 capture reported one lock somebody waited for, off a 4.25 us down_write, on a uniprocessor emulator where there is nothing to queue behind. It was the emulator being slow.

That is exactly the class of claim this project is not allowed to make, and there was nothing in the output that looked wrong. So contended now takes timings_are_real and has no default. It comes out of the capture's .meta.toml, which has said true or false for every trace in here since the first one. Told nothing, the widget draws every wait and refuses to use the word contention, and says so where the verdict would have been.

That pattern is in LAYOUT.md now, because it generalises. A widget that quietly guesses on the reader's behalf is worse than one with a blank column, because the guess looks exactly like a measurement.

What the widget still cannot tell you

function_graph records that down_write was called and does not record what it was called on. The two captures are evidence that the widget is reading the right thing, since they differ only in whether the writers share a file, but the widget itself has no idea which lock any row is about and says so in the footnote rather than printing a plausible name. Getting further needs a tracepoint that carries the address, or lockdep, and that is a different capture and probably a different lesson.

Next

Back to the toolkit rows in this milestone. The tape diff mode is next, then claimledger hardening and the unparsed line baseline. The bpc exit criterion needs five blueprints and there are two, so page-fault and write-path need three more alongside them.

[tamnd]

#50 is merged and tape diff mode is done.

The comparison itself was already there. kxdiff has answered the two questions that make one repeatable for a while now, which are what is being compared and how strictly, so TapeDiff takes a policy and a level and shows the answer rather than working out a second opinion. What was missing was the picture, and the picture turned out to be two things.

A ring on the functions that are only on one side. Not a change of fill, because on a trace cell the fill already means the duration marker and the border already means whether the width is a measurement, and a colour with two jobs is a colour a reader cannot trust. So trace_cell grew marked and faded and the widget does not draw its own boxes. The marking is about names and never about times: whether a box got wider is a fact about a clock, and on Tier 0 the clock is v86.

Faded boxes for the frames the policy took out. A pass over ninety frames of a hundred and fifty and a pass over all hundred and fifty are different results, and the faint boxes are the difference between reporting the first and showing the second.

The bug the new capture caught, again

Deciding which boxes to fade by name would have looked right. same-operation drops frames belonging to a task other than the one whose trace it is, and in tier1/contended-lock.txt eight function names appear both inside the traced task and inside somebody else's, including vfs_write. Fading every vfs_write fades frames that were compared. Keeping every vfs_write bright shows frames that were not. So it works by frame identity and there is a test asserting the eight.

That capture went in last week for the Lock Timeline and has now caught a second bug in a different widget, which is a decent argument for taking a messy real trace over a tidy fixture.

The picture that came out of it

The preview page has write-1byte against two-writes, which is the obvious one. The better one is the two vfs_write frames inside two-writes, drawn against each other on a shared scale. Same system call, same trace, same process, one going to a file on tmpfs and one going to a pipe.

vfs_write  549.333 us            vfs_write  121.875 us
  shmem_file_write_iter            anon_pipe_write
    down_write                       mutex_lock
    generic_write_checks             __alloc_pages_noprof
    file_remove_privs                __copy_user_ll
    file_update_time                 mutex_unlock
    generic_perform_write            __wake_up_sync_key
    up_write                         kill_fasync

One level below vfs_write they have nothing in common. That is what it means for the VFS to be an interface, shown rather than asserted, and the pipe write is 22 percent as wide.

About the shared scale

Worth writing down because it will come up again. By default each tape is a hundred percent of its own total, so a box the same width on both sides is the same share of its own trace and not the same number of microseconds. Sharing the scale needs both totals, and most real captures do not have one, because they end with an unclosed frame at the edge of the ring buffer. When it cannot be given it is refused and the footnote says so, rather than being approximated from the roots that do have durations. A shared scale that silently is not one is the worst outcome available here.

Next

claimledger hardening, then the unparsed line baseline, then tools/coverage and tools/lintnb. The bpc exit criterion still wants five blueprints against the two that exist.

[tamnd]

claimledger is done, in #51.

The evidence rule and the cap of two unobservable claims have been enforced since M0. The declaration half of that line never was, and it turned out to be the half worth having.

Every lesson has said which kernel, which architecture, which profile and which tier it is written against since the first meta.toml was written. Every artefact under corpora/ has said which machine it came off since the first capture. Nobody ever compared the two files. A lesson pinned to 7.2.2 on i386 could cite a trace taken off 6.8 on aarch64 and the checker would print "clean", which it did, for months.

The declaration has to be real. All four fields present, the profile one that kxbox/kernel/pin.toml actually builds, and the kernel version the one that profile builds. The version is read through the profile rather than off the top of the pin, which matters for C-longterm: it is on the fallback tree, so a lesson written against it is held to 6.18.48 while everything else is held to 7.2.2. The other side of that is the bump. When 7.2.3 lands, every lesson still claiming 7.2.2 fails, which is the failure you want.

Evidence that does not match has to say why. The obvious rule is that evidence has to match, and it is wrong. Some of the mismatches here are correct and permanent. v86 is a uniprocessor with no real clock, so a Tier 0 lesson that needs a second CPU or needs to measure a duration has to reach for Tier 1. Lockdep is not in profile A, so a lesson about lockdep has to cite captures taken under D-lockdep. A rule that forbids those is a rule people delete.

So the rule is that a mismatch has to carry a why_not_pinned sentence. All four claims it caught already had that explanation beside them as a TOML comment. Comments are for people. Moving the same sentences into a field a checker reads is the difference between a convention and a rule. The stale case is caught too: a reason for a difference that is no longer there is a wrong explanation, and a reader believes those.

The four it found were the two lockdep captures in C09 and the two Tier 1 artifacts in Z02. The last of those, Z02-06, is the one that makes the case. Its evidence kind is experiment, which is not a file kind, so nothing in the ledger had ever looked at that file for any reason. The comparison runs on any evidence value that turns out to be a real file with metadata beside it, which is how a claim nobody was checking got checked.

One small thing that will save an argument later. Architecture names go through a family table, because uname -m says i686, the build says i386 and everybody says x86 for the same thing. Anything not in the table is compared as written, so an unknown name fails to match rather than matching something it should not, and there is a test for that direction rather than only for the happy one.

Eighteen new tests. Full gate green.

Next is the unparsed line baseline, so parser drift becomes a build failure rather than something a reader finds, then tools/coverage and tools/lintnb. After that the kxray module rows, which are the bulk of what is left in M1.

[tamnd]

The unparsed line baseline is done, in #52. That finishes four of the six checker lines on this milestone.

I went in expecting to write down a number that was already being computed and to wire it into CI. What I found first was worth more than the tool.

Four parsers had a bare continue in the middle of a loop: kallsyms.parse, both lockdep readers, and the per CPU stats reader inside tracefs. Every one of them is right to be tolerant, because these files come off kernels nobody here has seen and a parser that dies on line four hundred thousand helps nobody. The problem is that a line one of those walks past looks exactly like a line that was never there. Nothing raises, nothing is logged, and the lesson shows the reader less than it did last week.

So the accounting is three buckets rather than one count. read turned into something, skipped was never data, unparsed was meant to be data and was not understood. The three have to equal the length of the file, and the tool checks that before it checks anything else, because a reader that classifies some lines and forgets others can report zero unparsed lines forever while understanding less and less of the file.

The bucket that matters is not the one the milestone line names. A rise in unparsed is loud and the trace tests already caught it. The quiet failure is a line sliding from read to skipped. I checked that it actually catches that, by widening the /proc/lockdep header rule until it swallowed one data row:

corpora/proc/handwritten/lockdep.txt: read was 8, is now 7
corpora/proc/handwritten/lockdep.txt: skipped was 1, is now 2

Same total lines, still zero unparsed, no exception, and the build fails with the file name and both numbers.

Each counting function is built on the same _read_one the parser itself uses, so parse and account cannot disagree about what a data line is. A second implementation of that rule would drift from the first, and then the baseline would be measuring the wrong parser.

Where it landed: 17 artefacts, 1736 lines, 1408 read, 159 skipped, 0 unparsed, all in corpora/BASELINE.toml.

Two of those skipped lines are the ones I would point at. corpora/proc/tier0/ring-overrun.txt has oldest event ts and now ts in it, which are clock readings rather than counters, and that reader collects counters. The old code treated "a line I do not collect" and "a line I could not read" identically, because both hit the same continue. They are different things and the file now says which is which. That is also why the rule is that the number is written down rather than that the number is zero. A tool insisting on zero would have made somebody either delete a true line from a real capture or teach the reader to pretend it was not there.

One more rule fell out of it. An artefact that no reader claims is an error rather than an omission, so adding a capture to corpora/ without saying what opens it fails the build. A committed file nothing has ever read is its own kind of lie.

Next is tools/coverage, so a lesson citing a subsystem with no owner fails CI, and tools/lintnb for the seven point notebook contract. That closes the checker block. After that the seven kxray module rows and the four kxbox rows, which are the bulk of what is left here, and three more blueprints to reach the five that bpc wants.

[tamnd]

tools/coverage is in, from #53, so the coverage box is ticked.

This one is a declaration with a checker behind it. The kernel is around forty million lines and drivers/ is roughly two thirds of that, so nothing covers all of it. Every book about the kernel is a book about a chosen slice, and the only real question is whether the author says which slice up front or lets the reader work it out over six months. coverage.toml says it.

The rule the milestone asked for is the one that does the work. A document citing a file that no subsystem owns fails the build. I broke it on purpose twice before writing the tests, once with a path nothing owns and once with a path that is owned by an entry which has never heard of the citing document, and got:

Z02: cites accel/drm/thing.c, and no subsystem in coverage.toml owns it. Add an entry, or stop citing it.
coverage.toml [networking]: Z02 cites it and is not named on the entry

The second one matters as much as the first. Nobody ever sits down and decides to start covering the network stack. Somebody cites net/core/dev.c in a lesson about something else, and a year later there are eleven half taught subsystems and no finished ones. Now that costs a ledger edit that a reviewer sees.

It is checked in both directions, because a ledger checked one way stops describing the repository within about a month, and then the headline rule is passing over a file that says nothing. An entry naming a lesson that stopped citing it fails, and so does a lesson citing a subsystem whose entry does not name it.

The statuses have to be earned rather than typed. taught needs a lesson and a blueprint marked complete, partial needs something written, mentioned owes no blueprint and may not name one, and out-of-scope needs a reason in a sentence with no citation into it. Nothing is taught today, because both blueprints are still partial, and the ledger says that rather than rounding up. There is a test asserting nothing is taught, so the day something earns it, somebody has to say so out loud in the diff.

Where the tree stands: 30 subsystems, 0 taught, 7 partial, 19 mentioned, 4 out of scope, covering the 23 kernel paths the three lessons and two blueprints cite between them. The seven partial ones are vfs, mm, writeback, locking, tracing, procfs and arch-x86.

Two prefix footguns are build failures rather than comments. Two entries claiming the same prefix have no answer under longest match, and a directory written without its trailing slash owns the wrong half of the tree, because arch/x86 matches arch/x86_64/mm/fault.c.

45 tests, including one per cited path and one per top-level kernel directory, so the sweeps grow as the ledger does. just coverage, just coverage-show and just coverage-cited, wired into just check and into the standard library only CI job next to the claim ledger.

Next is tools/lintnb and the seven point notebook contract, which is the last checker on this list. After that the checker block is closed and what is left in M1 is the module rows: the seven kxray modules, the four kxbox rows, and three more blueprints to reach the exit criterion of five.