madspace: write N LHE files from combine_to_lhe, and have MadSpin use it

[oliviermattelaer]

For the madspace author. This is an optimisation, not a fix. MadSpin already
gets the outcome this asks for, in Python, with no compiled dependency, and that
fallback stays in the tree and stays exercised. Please judge it on the numbers
below, which do not obviously carry it.

The C++ change

EventGenerator::combine_to_lhe gains an overload taking a list of paths
instead of one, and deals the combined events round-robin into them as it
formats them:

void combine_to_lhe(const std::vector<std::string>& file_names,
                    LHECompleter& lhe_completer, const LHEMeta& meta = {});

The fan-out is a new LHEMultiFileWriter in lhe_output.hpp: one
LHEFileWriter per path, all constructed from the same LHEMeta, so every
output file is a complete LHE file — its own <header>, its own <init>
block with the process cross sections. reserve() claims a run of stream
positions on the main thread before a chunk is submitted, file_index() maps
each position to a file; the worker formats into one string per file and hands
them over when it is done, in any order. The chunking, the thread pool and the
completion-order writeback are unchanged.

The single-path entry point now delegates to the list one with a one-element
list, so there is one implementation rather than two that can drift. Measured
below: that costs nothing.

Both overloads are exposed through pybind (the list caster rejects str, so
they cannot collide), along with LHEMultiFileWriter itself so the fan-out is
unit-testable without standing up a whole EventGenerator.

Round-robin, not contiguous

read_and_combine hands out chunks of up to 1000 events. A contiguous split of
a 100k-event pool over 18 files would therefore be balanced only to within a
whole chunk — 1000 events on a ~5500-event slice — and a decay pool that runs
dry triggers a refill. Dealing per event is balanced to within one whatever the
chunking. Confirmed on the real benchmark: the 18 slices of one channel came out
16903/16903/…/16902, and every one of them carried
1.4599439254e+00 in its own <init>.

Ordering within a file is not guaranteed — but it already was not, since jobs
completed out of order into a single writer, so nothing downstream loses a
property it had.

Why MadSpin wants it

MadSpin's parallel unweighting gives each of N workers its own slice of a decay
pool. Handed one file instead, a worker has to stride it, and _StridedEvents
skips by calling next() on the other workers' events — so each of 18 workers
parses the whole pool to read 1/18 of it. The per-worker files also have to end
up at particular paths, because on a pool refill those paths are what the
waiting workers open.

What the alternative already in this branch does

c173250a5 (in the base branch) has MadSpin write madspace's single
events.lhe and then deal it out into the per-worker files itself, with
_split_pool_round_robin. Same layout, same banners, same everything — one
extra full write and full read of the pool.

That fallback is still here and still reached: madspace is not upgraded in
lockstep with MadSpin (it can be an older binary wheel), so madevent.py checks
whether this build has the overload, warns if not, writes the single file, and
MadSpin splits it exactly as before. There is a unit test that drives that path
with a launcher stub pretending to be the old madspace.

The path problem, and why it is a count and not paths

madspace takes paths — that is the right primitive; it imposes no naming or
directory policy. But the mg7 run_card takes a count
(nb_output_files, mg7's counterpart of madevent's nb_unweight_output),
because the run_card is a user-facing card rendered from a template and a list
of absolute paths is not something a user sets. mg7 writes
<run_path>/events_<i>.lhe, and since mg7 names its own Events/run_NN
directory, MadSpin then moves them to
Events/<run_name>/unweighted_events_<i>.lhe.

That move is N renames inside one filesystem. It does not erase the gain — it is
not measurable at all. It also buys something back: the canonical directory
never contains a half-written pool, which is the same reason
_materialise_refill_pool builds in a .part directory. Writing straight into
it, as the Python split did, did not have that.

The measurement

ttbar_full (p p > t t~, both tops fully decayed), spinmode PA, seed 42,
nb_core = 18, 18-core Apple silicon, warm back-to-back, three passes each with
the decay directories rebuilt every time. Means of total:

config	10k	100k
madevent (original baseline)	38.91 s	90.14 s
mg7, MadSpin-side Python split (this branch's base)	21.30 s	37.61 s
mg7, madspace multi-file (this PR)	21.05 s	36.32 s

Per-pass, so you can see the spread:

• 10k — madevent 35.64 / 40.51 / 40.57; python split 21.84 / 20.51 / 21.54;
  native 21.37 / 21.03 / 20.74
• 100k — madevent 89.06 / 90.00 / 91.37; python split 38.45 / 36.70 / 37.69;
  native 35.24 / 37.33 / 36.38

The phase this deletes, measured directly in the python-split runs:

	decay_mg7_split	as % of total
10k	0.34 s	1.6%
100k	1.11 s	3.0%

And the same question asked in isolation, away from MadSpin — t > b u d~,
300k events, four repetitions, madspace's own combine wall time out of
info.json:

	wall
combine_to_lhe before this PR, 1 file	0.284 s
combine_to_lhe after this PR, 1 file	0.273 s
combine_to_lhe after this PR, 18 files	0.285 s
MadSpin _split_pool_round_robin, the same pool → 18 files	0.50 s

So: the single-file path does not regress, the fan-out costs about
0.01 s, and it replaces a 0.50 s Python pass. On the benchmark that is
1.11 s off a 37.6 s run.

Verdict

Honestly: the wall-clock case is weak. 1.1 s on a 37 s run is 3%, and the
run-to-run spread of the benchmark is ±1.7 s, so the saving does not show up
reliably in total — the two mg7 configurations overlap, and one 100k pass had
the python split come out ahead. The measurement that does resolve it is the
phase timing, and the isolated combine bench. If a compiled dependency has to be
paid for in wall clock, this does not pay for it.

The stronger argument is not time but peak disk. The Python route
materialises the pool as one file and as the slices before deleting the
source: 276 MB per channel at 100k, two channels, so ~550 MB of transient extra
disk, and a refill adds a generation rather than replacing one. Writing the
slices directly never has both.

So: merge it if you think the primitive is worth having in madspace on its own
terms — it is a small, self-contained generalisation of an entry point that
already existed, and it makes the single-file case a special case of the general
one rather than a separate code path. Do not merge it expecting the benchmark to
move.

Also here: a measurement fix

decay_event_generation — the largest phase of a run — was missing from the
report of every run where two or more particles decay, i.e. every benchmark
worth running, p p > t t~ included. _generate_decays forks one process per
particle and the phase accounting is in-process state, so the child charged its
work to a dict the parent never saw. The child already marshals its results back
through a small JSON file; the timings now go with them. That is what makes the
decay_mg7_split row above exist at all — without it only total was usable,
and total cannot resolve this change.

Two things a reader has to know about those numbers: the child clears the
accounting it inherited through the fork (it has already been charged in the
parent), and what comes back is a sum over particles of concurrent work, so it
is work done, not wall time, and may exceed the wall time of the fork that
contained it. Hence decay_event_generation reading 114% of total in a madevent
run.

Tests

• madspace/tests/test_lhe_multifile.py — 11 new tests on LHEMultiFileWriter:
  every file's own <init> and cross sections, the header on each, a file that
  got no events still being valid LHE, exact round-robin, balance to within one,
  nothing lost or duplicated, the chunked reserve/file_index/write_string
  path agreeing with the one-at-a-time path, one file behaving as the ordinary
  single-file case, and an empty file list rejected.
  Full madspace suite: 1494 pass (4 pre-existing failures need scipy, and
  test_flow/test_mlp need torch; neither is installed here).
• MadSpin unit suite: 570 → 577, all green. The new ones cover the run_card
  handshake, mg7 writing the slices itself (nothing is ever materialised as one
  file), the old-madspace fallback, and the fork phase timings.
• End to end on a real generated process: t > b u d~, 20k events,
  nb_output_files = 5 → five files of exactly 4000 events, each with
  4.8954031474e-01 in its own <init>, all readable by MadSpin's
  lhe_parser.EventFile.

One thing found on the way, not fixed here

One cmake --build of madspace produced a libmadspace_cpu.dylib of 2.3 MB
instead of 6.1 MB. It loaded fine and then threw
std::runtime_error: empty tensor during survey, reproducibly, on a process
that a good build handled without complaint. Wiping build/ and reconfiguring
with the same arguments produced a library byte-for-byte the size of a
known-good one and the error went away; two further from-scratch builds since
have both been fine. So: seen once, not characterised. Flagging it only in case
it is familiar — a silently short library that fails at run time rather than at
link time is an unpleasant failure mode to hit twice.

[oliviermattelaer]

@theoheimel
This was an attempt to make madspin faster... clearly this is irrelevant for speed at the end of the day.
So for me I would just drop this. Except if you see some intererest in your side (in can also be interesting for Pythia8 / ... ) But likely not worthed ...