Read /proc, and record what the kernel promises about each file

[tamnd]

Closes the kxray.proc row on #2.

Five readers for four file shapes, because /proc has fewer shapes in it than it has files. keyed reads Key: value, which covers meminfo and a process status file. percpu reads a label and one count per CPU, which covers interrupts and softirqs. maps reads one record per line with positional columns. pidstat reads the single line file. version reads one line of free text with two useful things in it.

Eight new captures under corpora/proc/tier0/, all off the pinned box.

The part I would push back on if somebody else sent it

Every one of these readers hands back an object carrying a Promise saying what the kernel tree says about the file it read. That could easily be decoration. Here is why it is not.

The kernel keeps its own record in Documentation/ABI, four directories, one per level, defined in Documentation/ABI/README. On 7.2.2 that tree has 685 files in it. Six of them describe a path in /proc:

obsolete/procfs-i8k        ->  /proc/i8k
testing/procfs-diskstats   ->  /proc/diskstats
testing/procfs-smaps_rollup->  /proc/pid/smaps_rollup
testing/procfs-attr-exec   ->  /proc/*/attr/exec
testing/procfs-attr-current->  /proc/*/attr/current
testing/procfs-attr-prev   ->  /proc/*/attr/prev

Not meminfo. Not interrupts. Not /proc/<pid>/stat, which is the file behind every process monitor ever written. Not maps. Not one file in this corpus.

That is undocumented, and it is deliberately not spelled unstable. Those files have had the same shape for many years and will keep it, because changing one breaks userspace. What is missing is anybody having written down which part of the shape you may lean on. So the readers read them and say what they are leaning on:

/proc/self/stat is undocumented: no file under Documentation/ABI describes this path,
which is true of nearly all of /proc

Two paths are worse than undocumented and this project reads both. The closing section of that README names, as things that "should not under any circumstances be considered stable", Kconfig, calling out /proc/config.gz by name, and kernel symbols, saying not to rely on "the presence, absence, location, or type of any kernel symbol". The second is /proc/kallsyms, which kxray.kallsyms has been reading since M0. Counting ops tables by name on a machine in front of you is a fine thing to do. The same code inside a deployed tool is not, and now something says so.

The capture I am happiest with

/proc/<pid>/stat is one line of space separated values, so the obvious way to read it is line.split(), and that is wrong. The kernel prints the command name in brackets and does not escape it. Here is a real line off the pinned box, from a process whose executable is named od) d ma:

37 (od) d ma) R 1 0 0 0 -1 4194304 37 0 0 0 0 1 0 0 20 0 1 0 265 ...

line.split() gives 37, (od), d, ma), R, and everything after the name has slid two places along. What the reader gets:

/proc/self/stat: undocumented
pid:     37
comm:    'od) d ma'
state:   R
fields:  50 named, 0 beyond what proc.rst lists
naive:   a whitespace split would call the state 'd'

d is not a state any process is ever in. Nothing raises, every number is still a number, and a monitor would carry on reporting it.

Getting that capture took a detour. busybox dispatches on its own argv[0] and refuses to run under a name that is not an applet, so a copy of /bin/sleep called od) d ma exits instantly with "applet not found". A shell script works, because the kernel takes comm from the script's own filename. self-stat.txt is the same file for a process called cat, kept next to it, because the reason this bug survives everywhere is that the wrong parse is right almost always.

The other trap, which is invisible

start     end       perms  size     pages  what
--------  --------  -----  -------  -----  -------------
08048000  08149000  r-xp   1052672  257    /bin/busybox
08149000  0814c000  rw-p   12288    3      /bin/busybox
b7f8f000  b7f9f000  rw-p   65536    16     anonymous
b7f9f000  b7fa3000  r--p   16384    4      [vvar]
b7fa3000  b7fa5000  r--p   8192     2      [vvar_vclock]
b7fa5000  b7fa7000  r-xp   8192     2      [vdso]
bfa7b000  bfa9c000  rw-p   135168   33     [stack]

busybox appears twice because a program's text and its data are one file mapped two ways. Line three has no name, which is the anonymous memory a first write has to go and find a page for. And line three ends in a space, because the kernel pads every line out to a fixed column before printing the path it does not have. So that line has five whitespace separated fields and the others have six, and line.split()[5] works on every maps file until it meets one. The regex has the path optional. Do not let an editor strip trailing whitespace from that capture.

One number ties two captures together: vsize in self-stat.txt is 1298432 and the seven mappings in self-maps.txt add up to 1298432, because vsize is that sum. Two files, two readers, one fact, and a test that fails if either drifts.

Smaller things worth knowing

The unit in meminfo says kB and means KiB. MemTotal is 102308 kB on a box given 100 MiB, and times 1024 that is a shade under 100 MiB while times 1000 it is nowhere near. The reader keeps the kernel's spelling, because changing it here would put this project at odds with every tool on the machine, and does the multiplication by 1024 in bytes_of.

A value is not always a number. Uid: is four values on one line, State: is a letter and a word in brackets, Groups: is empty and the kernel prints the key anyway. So the model holds a tuple of words and offers a number only when there is exactly one word and it is one.

The column count in interrupts and softirqs comes off the header and nowhere else. Not os.cpu_count(), not /proc/cpuinfo. These are per possible CPU, which is not the number online and not the number in the machine, and only the header has that already resolved. Reading a row before the header has been seen is unparsed rather than read with an empty list of counts, because a row with no counts is the shape that sails through and means nothing.

The rows under the numbered interrupts are per architecture and per config. Two here, NMI and TLB. An x86-64 desktop prints around fifteen. There is no list of them anywhere in the code.

interrupts and softirqs are worth reading together. One counts the hardware asking and the other counts what answering it deferred, and corpora/traces/tier0/flat-interrupt.txt from #56 is that same gap happening once with timestamps on it. The two vectors that have ever fired on this box are TIMER and RCU, which is exactly what that trace shows being raised and serviced.

STAT_FIELDS comes from Table 1-4 of Documentation/filesystems/proc.rst. That table is headed "as of 2.6.30-rc7" and it still describes 7.2.2 with all 52 fields in the right order, for a file with no ABI entry at all. That is what not breaking userspace looks like from outside.

Checks

104 new tests in tests/test_proc.py. corpora/BASELINE.toml is at 32 artefacts, 1982 lines, 0 unparsed. Whole suite green, 15 checkers clean, 63 node tests passing.

One test is aimed at the future rather than at the code: test_nothing_in_this_corpus_is_documented_anywhere asserts that all eight paths classify as undocumented. If a kernel release ever writes an ABI entry for one of them, that test fails, and the failure is good news.