kxray.corpus: load a pinned capture, and take the run out of it

[tamnd]

The M1 row that reads kxray.corpus: loading, plus the normalisation that makes any of this diffable.

Two halves of one job, in two modules, because they get used at different moments. Load first, normalise second, compare third.

Loading

The routing table has moved out of tools/baseline.py and into kxray/corpus/index.py. That is the change that made the package worth having. Until now the only thing that knew which reader opens which artefact was the baseline tool, and a lesson that wanted to open one capture had to work it out again by hand, which meant two answers to the same question that could drift apart without anybody noticing.

An artefact is asked for by id, which is its path under corpora/ with the suffix dropped. It comes back with its metadata already loaded and the name of the reader that opens it.

>>> from kxray import corpus
>>> one = corpus.get("traces/tier0/write-1byte")
>>> one.kind, one.kernel, one.tier
('function_graph', '7.2.2', 0)
>>> corpus.read(one).found
57

traces/tier0/write-1byte rather than something shorter because that is the name every lesson, blueprint and test in this repository already uses. A shorter id would have bought nothing and cost a migration, plus a period where two names for the same file were both correct.

read() hands the /proc readers the kernel path out of the artefact's own metadata, which is the part everybody opening one of those by hand kept getting wrong. What a file is called on disk does not decide how it is read or what it is worth: self-maps.txt is /proc/self/maps, and only the second of those two names reaches the stability ledger.

$ python3 -m kxray.corpus list
id                                           reader           tier  lines  found
btf/handwritten/tiny                         btf                 -      0     44
events/tier0/sched_process_exec              event-format        0     13      7
events/tier0/sched_switch                    event-format        0     17     11
experiments/tier1/tracer-cost                none                1     22      0
oops/tier0/lockdep-ab-ba                     lockdep-splat       0     82      1
proc/tier0/self-maps                         proc-maps           0     22     22
traces/tier0/write-1byte                     function_graph      0     89     57
...

Routing on the file name looks like the sort of thing that breaks the first time somebody names a file badly. It is held up by the thing that depends on it: an artefact nothing claims raises, the baseline fails, CI goes red. A file cannot land in corpora/ and be read by nothing.

Normalising

The process happened to be pid 83138 this time. The trace happened to start three seconds after boot. The mutex happened to live at c78250b8. Every one of those changes on the next run, so a comparison of two captures of the same work reports several hundred differences and none of them is the one anybody wanted to know about.

--- before
 5)    dd-83138    | + 10.875 us   |  }
 5)    dd-83138    |               |  vfs_write() {
 5)    dd-83138    |               |    rw_verify_area() {
 5)    dd-83138    |   0.250 us    |      security_file_permission();

--- after
 lane#3)    dd-pid#3    | DUR us   |  }
 lane#3)    dd-pid#3    |               |  vfs_write() {
 lane#3)    dd-pid#3    |               |    rw_verify_area() {
 lane#3)    dd-pid#3    |   DUR us    |      security_file_permission();

The legend is half the output and is not optional, because a normalised trace that cannot say which process pid#3 was is a trace nobody can take back to the machine it came off.

pid#3   was  83138
lane#3  was  5

Six rules, in the order the specification puts them, and the order is load bearing twice. Pointers run before offsets, because __mutex_lock+0x8c/0x230 has hex in it that is not an address. Identity runs before cpu, because a task column holds a pid and a cpu column holds a cpu and telling them apart is done by where they are and not by what they look like.

The two properties, checked over the whole corpus

Both tests are parametrised over every artefact in corpora/ rather than over a fixture. A rule is a regular expression and a regular expression is wrong in ways nobody predicts, so running all of them over all of the files is the only check here that catches a rule doing something to a file its author never opened.

It never changes the number of lines. Normalising is a substitution and not an edit. A rule that dropped a line or wrapped one would make the result impossible to read next to the original, and reading the two next to each other is how anybody checks a rule at all.

Running it twice is the same as running it once. A rule that matches its own output keeps renaming things every pass, and a file that changes every time you look at it is not a baseline.

What it deliberately does not do

It does not reorder anything by default. Rule seven in the specification sorts concurrent events by a stable key. That is right for an event trace where every line stands alone, and wrong for a function_graph trace where the order of the lines is the shape of the call tree. So order exists, it is off unless asked for by name, and sorting a call graph is a mistake this will not make on your behalf.

It does not guess at what might vary. Every pattern was written against a line that is in corpora/, with one exception, which is the counter rule, and that exception says so in its own comment rather than being hidden.

The one heuristic, and which way it fails

Eight hex digits and eight decimal digits are indistinguishable when none of the digits happens to be a letter, and the kernel prints plenty of eight digit decimals that are not addresses. utime out of /proc/self/stat is one.

So a bare run has to contain a hex letter to count as an address. Written 0x it counts either way, and a range like the address pairs in /proc/self/maps counts either way, which is how 08048000-08149000 still comes out as ptr#1-ptr#2.

What that gives up is the occasional real address whose digits are all decimal and which is neither in a range nor written 0x. That is the direction to fail in. A number that was not renamed is sitting there in the output where anybody can see it. A size that was renamed to ptr#4 is a difference nobody will ever see.

The command line

python3 -m kxray.corpus list                      every artefact and what reads it
python3 -m kxray.corpus show traces/tier0/write-1byte        one artefact, in full
python3 -m kxray.corpus normalize traces/tier0/write-1byte   the run taken out of it

normalize is for the moment before writing a rule: run it, read the output next to the original, and see whether the substitution did what it claimed. That reading is the whole check on a rule, so it is worth being one command. Nothing here writes anything.

What moved and what did not

tools/baseline.py keeps only the comparison against corpora/BASELINE.toml, which is the part of it that was ever about the baseline rather than about the corpus. Its numbers are unchanged: 39 artefacts, 7837 lines, 0 unparsed.

The full gate is green. 1449 passed, 7 skipped, up from 1394.