Read the kernel's own source files

[tamnd]

What this is

kxray.source, the last unbuilt row of the toolkit in M1. Everything else in kxray reads what a running kernel printed. This reads what the kernel ships as text: MAINTAINERS, the syscall tables, Kconfig files, and the source itself.

Six modules, all measured against real files rather than written from memory.

The tree handle

tree.py answers one question before anything else asks a second one: how much kernel is here. There are three honest answers on a machine somebody is sitting at.

The full tree is the 1.6 GB ./kxbox/kernel/tree.sh unpacks, checksum verified. The partial tree is five files committed under corpora/source/pinned/, so the parsers work in CI and in a notebook that has downloaded nothing. Nothing at all is a fresh checkout, and the right answer there is the command that fixes it.

Tree.complete is the field to read first. On the corpus a lookup that misses says the file is not in the corpus, never that it is not in the kernel. Those are different sentences and only one of them is true.

MAINTAINERS is 29847 lines, so the corpus has MAINTAINERS.excerpt. The suffix is the mechanism rather than a label. read("MAINTAINERS") finds it and sets partial on what it hands back, and parse_file carries that onto the Maintainers, so being a slice travels with the content instead of living in a comment somebody has to remember.

The MAINTAINERS glob is not fnmatch

This is the one worth reading twice.

FILESYSTEMS (VFS and infrastructure) carries F: fs/*.

fnmatch.fnmatch("fs/proc/base.c", "fs/*")   True
covers("fs/*", "fs/proc/base.c")            False

The kernel's own header block says a single star stops at a slash and a double star crosses one. fs/proc/base.c belongs to PROC FILESYSTEM, and the VFS section said so by writing one star instead of a trailing slash. A tool built on fnmatch mails every patch under fs/ to the VFS maintainers and looks right while it does it.

Two more from the same file, both confirmed against scripts/get_maintainer.pl rather than assumed.

K: patterns are applied with perl's /x, at lines 575 and 622 of the script. So AUDIT SUBSYSTEM's \baudit_[a-z_0-9]\+\b has a literal plus sign in it and does not match audit_log_start. It matches audit_l+x. That is a fact about the file rather than a bug here, and it is a test so nobody helpfully fixes it.

THE REST carries F: * and F: */, so it matches every file in Linux. Any lookup that takes the first hit and stops reports Linus for the whole kernel. lookup returns everything in file order and specific is the same list with the catch-all removed.

PROC FILESYSTEM has a status of Maintained, two mailing lists and no M: line at all, so contacts falls through to the lists. Coming back empty would read as nobody looks after this, and the file does not say that.

Write is 4 here and 1 there

name           i386    x86_64            same number
write          4       1                 no
read           3       0                 no
open           5       2                 no
exit           1       60                no
rt_sigaction   174     13 under 64, 512 under x32   no
break          17      not in this table  no

A syscall number with no architecture attached is not an identifier. The pinned box is 32 bit, so every number a lesson prints comes out of syscall_32.tbl, and a reader on their own laptop comparing against the numbers they know finds different ones. Both tables are committed so that can be a table rather than a claim.

number_of returns None when a name appears under two abis rather than picking the first. Nineteen rows in the 32 bit table have a name and no entry point, which is a number reserved forever so a binary from 1994 gets ENOSYS rather than somebody else's system call. Two rows carry a literal - in the compat column holding the place open so that noreturn lands in the right one.

Why a .config has symbols nobody chose

CONFIG_PREEMPT=y gives a build with CONFIG_PREEMPT_BUILD=y and CONFIG_PREEMPTION=y in it as well. kernel/Kconfig.preempt is the whole explanation and it is 194 lines.

CONFIG_PREEMPT_BUILD has no prompt, so nobody chose it by hand, and in
corpora/source/pinned/kernel/Kconfig.preempt it is selected by CONFIG_PREEMPT,
CONFIG_PREEMPT_LAZY, CONFIG_PREEMPT_DYNAMIC

Six of the fifteen symbols in that file have a type and no prompt, which means they never appear in menuconfig and cannot be set by hand at all. In a .config they look exactly like something a person picked.

The parser had two unparsed lines at first, the default lines that belong to the choice block rather than to any symbol. That grew a Choice dataclass rather than the number being written down as acceptable. Fourth time in this project a parser was fixed instead of a baseline.

Three names for one call

the syscall table   sys_write
the source          SYSCALL_DEFINE3(write, ...)   fs/read_write.c:747
the running kernel  __ia32_sys_write              /proc/kallsyms

No two of them equal, with a macro standing between each pair. Grepping fs/read_write.c for sys_write is worse than finding nothing: it finds lines 728 and 750, both ksys_write, which is a different function that the real entry point calls. The reader lands three lines from what they wanted, on something close enough to be believed.

That last paragraph is a correction. The first draft of the module docstring said the string does not appear in the file at all, which was written from memory and is false. Running it found ksys_write, and the true version is a better fact than the one that was wrong.

unwrap strips the wrappers, which are a list rather than a rule because they are architecture dependent. The same call is __ia32_sys_write on the pinned box and __x64_sys_write on the reader's laptop.

The context hash

A citation here names a file and a piece of text to find in it, never a line number, and that already survived files moving. What it did not survive is the anchor holding still while the code under it changes.

An anchor is usually a function signature. The signature is the most stable line in a function and the body underneath is the part people edit. So a citation supporting a sentence about what a function does goes stale with the anchor sitting exactly where it always was, the checker stays green, and the lesson is wrong.

So a confirmed citation now records a hash of the seven lines around its anchor. Proved end to end by poking one line inside a window in the unpacked tree:

write-path-R38  fs/proc/base.c:3002  a162f8981f43  the lines around it have changed,
recorded 14b7bac656cb and found a162f8981f43

blueprints/write-path.refs.toml#write-path-R38: 'static int do_io_accounting(struct
task_struct *task' is still there and the lines around it have changed, so go and read it

"Still there and changed" is a different sentence from "gone" and needs a different answer from a reader, so it is a separate finding.

The hash is over whitespace normalised lines, which is a deliberate choice about what counts as a change. Reindenting a block and tab to space churn do not fire. Renaming a variable, adding a branch or changing an argument do. Hashing the bytes gives a checker that cries every release and gets switched off, which is worse than not having one.

All 73 citations in the repository picked one up from a single --confirm run against the pinned tree, and no line number moved. tests/test_refcheck.py now asserts every confirmed citation carries one.

What is committed

100K of corpus, all out of the tarball rather than off a running kernel, so it lives in corpora/source/ rather than beside the traces.

corpora/source/pinned/
├── MAINTAINERS.excerpt                        the format block and thirteen sections
├── arch/x86/entry/syscalls/syscall_32.tbl     the numbers the pinned box answers to
├── arch/x86/entry/syscalls/syscall_64.tbl     the numbers the reader's laptop answers to
├── fs/read_write.c                            one real source file, whole
└── kernel/Kconfig.preempt                     why a .config has symbols nobody chose

Plus corpora/proc/tier0/kallsyms-write.txt, eight lines off the running box, which is where the third name comes from. It is marked not-abi, because the closing section of Documentation/ABI/README names kernel symbols as something that must not under any circumstances be considered stable.

Baseline goes from 32 artefacts and 1982 lines to 38 and 5224. Every reader accounts for every line and nothing is unparsed.

corpora/source/pinned/MAINTAINERS.excerpt      maintainers      296   216    80   0
corpora/source/pinned/.../syscall_32.tbl       syscall-table    479   461    18   0
corpora/source/pinned/.../syscall_64.tbl       syscall-table    443   421    22   0
corpora/source/pinned/fs/read_write.c          none            1822     -     -   -
corpora/source/pinned/kernel/Kconfig.preempt   kconfig-source   194    69   125   0

read_write.c has no reader on purpose. symbols opens it with a name to look for, so there is no whole file count to take, and saying so beats inventing one.

Where the models live

Per module rather than in kxray/models.py, which is 1424 lines already. The /proc types belong there because widgets, the baseline and each other all share them. A syscall table row is only ever a syscall table row, the same way kxray/btf/format.py keeps its own. The reasoning is written down in kxray/source/__init__.py so the next person does not have to guess.

Checks

43 new tests in tests/test_source.py, 6 more in tests/test_refcheck.py, whole suite green. Every gate green: ruff, pytest, node, diagrams, nbbuild, sitebuild, kxbox, vendor, baseline, claimledger, coverage, bpc, kconfig, refcheck, lintnb, kxmanim, lintprose.

Closes the kxray.source row in #2.