MLM: choose the clustering history per diagram (beam.clustering_history)

[oliviermattelaer]

Summary

Adds beam.clustering_history to the mg7 MLM back-clustering. It chooses which diagrams a clustering history may follow:

value	behaviour
all_diagrams (default)	unchanged: smallest-measure history over the union of all diagrams, each line's flavour taken from the first diagram that has it
diagram	per event, pick one diagram with probability `
madevent	setclscales' rule: keep the all_diagrams history unless it calls a different number of legs merging jets than the picked diagram does, and fall back to the diagram then

It also ports madevent's ptclus rule for the LHE pt_clust_i under scale_scheme = "madevent", as a separate commit. Only legs that iqjets calls merging jets take a clustering scale; every other leg is written at sqrt(s).

Depends on beam.clustering_measure (87c4967, already on the base branch). All validation below uses clustering_measure = "madevent".

Implementation

• MLMClustering (mlm_clustering.cpp): under a per-diagram history, compiles one state machine per diagram from that diagram's own lines and appends it after the union machine. diagram_start_states() maps a diagram index to its root offset (0 = none, use the union).

• Kernel (kernels/mlm.hpp): the walk takes a per-event start_state and returns its merging-jet count. The new history_mode input implements the madevent rule by walking the diagram, then the union, and re-walking the diagram if the counts differ.

• Integrand: under a per-diagram history the union clustering only feeds the pdf prior for the flavour sampling. Then:

  1. The matrix element is evaluated at the union scales, which gives the diagram weights and the channel weights.
  2. One diagram is picked (a choice between histories, not an importance sample, so it puts no factor on the weight), and the event is re-clustered along it.
  3. The pdfs, merging cut, alpha_s and pdf reweighting are all taken from the new history, and the matrix element is evaluated again at the new scales.

  The second evaluation keeps mixed coupling orders (gluon fusion + VBF) exact; rescaling by alpha_s ratios would not. The shared weight factors moved into Integrand::mlm_weights.

• Run card: beam.clustering_history in RunCardMG7 and run_card.toml, wired in launch.py.

• Restriction: a single matrix element per integrand (throws otherwise).

Validation

Parton-level cross sections, madspace / madevent

Same cards on both sides (ickkw=1, NNPDF23_lo_as_0130_qed, maxjetflavor 4,
dynamical scales, jet_radius 1.0), clustering_measure = "madevent",
scale_scheme = "madevent". 100k events (W 150k, tt~ 50k, VBF 30k).

process (xqcut)	all_diagrams	diagram	madevent
HEFT p p > h +0-3j, h > ta+ ta- (15)	1.067	0.993	1.013
2j	1.108	0.992	1.020
3j	1.158	0.975	1.028
2j qq>qq (the VBF-like channel)	1.291	1.030	1.028
HEFT with QED=0 (15)	1.051	0.989	-
VBF h j j QCD=0 + h j j j QCD=1 (20)	1.011	0.999	1.004
W p p > e+ ve +0-3j (15)	1.029	0.988	0.999
2j / 3j	1.086 / 1.140	0.961 / 0.934	0.998 / 1.003
tt~ p p > t t~ +0-3j (20)	0.996	0.926	0.986
2j / 3j	0.977 / 1.041	0.894 / 0.890	0.964 / 1.029

tt~ in diagram mode is 7% low by design: inside a single diagram an FSR gluon
off a top cannot be taken into the beam instead, so its scales and pdf ladder
change. madevent only reclusters along the channel when the merging-jet count
differs, which is what the madevent mode does.

Showered, MLM-matched rates (MG5aMC_PY8_interface), HEFT, xqcut 15

Ratio to madevent of the merged cross section and of the per-multiplicity rates.

qcut		all_diagrams	diagram	madevent
20	total	1.063	0.994	1.015
	2j	1.183	1.008	1.042
45	total	0.985	1.000	1.001
	2j	0.867	1.025	1.008
	3j	1.140	0.980	1.015
75	total	0.941	1.003	0.996
	2j	0.547	1.018	0.998

The 0.547 at qcut 75 is the effect this branch is about: madspace's VBF-like jets
were matched and vetoed because they had been clustered as gluon-fusion jets.

DJR spectra (kT, R=1) around the matching transition at qcut 75, ratio to
madevent: all_diagrams 0.82-0.90, diagram 0.98-1.00, madevent 0.99-1.01.

pt_clust (second commit)

Fraction of the p p > h j j j (QCD=1) weight where only one of the two tagging
quarks is non-merging (pt_clust = 13000), i.e. a tagging quark the shower is
allowed to veto on:

	madevent	all_diagrams	diagram
before the ptclus commit	0.000	0.201	0.364
after	0.000	0.000	0.000

A residual difference stays in the classes where the two quarks come from a
W/Z decay: madevent puts 3.3% of the weight in "gluon merging, one decay quark
merging", madspace 0.1-0.4%.

W+jets, fraction of events with one non-merging leg, after the commit:
madevent 0.305 (2j) / 0.405 (3j), all_diagrams 0.014 / 0.042 (unchanged by the
commit - none of these legs sits at a jet vertex the union history calls
non-merging), diagram 0.318 / 0.441, madevent-history 0.300 / 0.402.

HEFT 3j non-merging-leg distribution after the commit (0/1/2/3 legs):
madevent 0.752/0.122/0.121/0.004, diagram 0.742/0.124/0.130/0.004
(0.784/0.122/0.090/0.004 before).

The ported samples are bit-identical to the pre-port ones in every event record
apart from the <scales> line, for all three history modes (VBF and HEFT), so
the commit moves no cross section: VBF stays at 1.011 / 0.999 / 1.004
(all_diagrams / diagram / madevent). The fraction of outgoing partons written
at sqrt(s) over the whole VBF sample goes from 0.849 (all_diagrams) and 0.840
(diagram) to 0.868 / 0.875 / 0.875, against madevent's 0.871.

Showered, MLM-matched rates, pure VBF, xqcut 20

Ratio to madevent, before and after the ptclus commit (30k events, so the 3j
rates carry 1.6-2.4% statistical errors).

qcut		all_diagrams before / after	diagram before / after
20	total	0.964 / 1.002	0.944 / 0.965
	3j	0.920 / 1.022	0.905 / 0.964
60	total	0.980 / 1.002	0.968 / 0.996
	3j	0.970 / 1.146	0.840 / 1.070

The showered 3j deficit reported for pure VBF was the pt_clust bookkeeping, not
the history selection: it is gone in all_diagrams too.

Showered, MLM-matched rates, W + 0-3j, xqcut 15

Ratio to madevent, with the ptclus commit in (150k events).

qcut		all_diagrams	diagram	madevent
20	total	1.025	1.009	1.012
	2j	1.114	1.007	1.005
	3j	1.073	0.928	1.011
45	total	0.987	1.000	1.002
	1j	0.891	1.024	1.022
	2j	0.904	0.951	0.974
	3j	0.916	0.877	0.934

The 3j rates carry 2-4% statistical errors.

Cost

Generation-phase CPU, per-diagram modes vs all_diagrams: HEFT 1.6x, W 1.25x,
VBF 2.2x (two matrix-element evaluations per event, plus a second clustering).

Test plan

• madspace/tests/test_mlm.py: 120 passed. The new tests cover: per-diagram machines extend the union machine; the union walk is unchanged; the walk along a diagram reports that diagram; a one-diagram process clusters the same both ways; the madevent history is always the union or the diagram; each diagram uses its own line flavours; ptclus for a non-merging radiating quark.
• madspace Python suite (excluding test_pdf and the torch-dependent tests)
• all_diagrams runs are bit-identical (event records) to samples generated before this change with the same seed and cards (VBF, HEFT, HEFT QED=0)
• MG5 suite (./tests/test_manager.py): 1549 tests, 3 pre-existing reds
  (test_output_standalone_accepts_loop_induced,
  test_output_standalone_cpp_refuses_loop_induced, MadLoop
  testIO_UnitProcOutputIOTests), none of them touching the run card or banner

Rebased onto claude/madspace-madevent-scales at 8a1b37c (which carries the
main merge and the updated feat-mlm); all four commits applied cleanly, and
both suites above were re-run on the merged base.

The generation runs were made with these four commits on 87c4967, i.e. before
the rebase, which changed no clustering, scale or integrand code. Re-checked on
the merged base (62897e4), 30k events of the VBF process:

• all_diagrams on this branch and the base branch alone, same build, give
  identical event records and the same integral (0.19522 pb). The only
  difference anywhere is pt_clust on jet legs (1257 of 30000 events), from the
  ptclus commit; mur, muf, muf1 and muf2 are untouched.
• diagram gives 0.19283 pb against 0.19288 before the rebase, and
  all_diagrams 0.19522 against 0.19518 - both within 2.5e-4, far inside the
  0.17% statistical error, so the ratios to madevent above are unchanged.

(The base branch could not run MLM at all until 62897e4: the main merge
renamed the madspace observable m_inv to pair_mass while apply_auto_ptj_mjj
still asked for jet-m_inv, so every xqcut > 0 run raised TypeError. Diagnosed
here, fixed by the branch owner.)

Test plan

• madspace/tests/test_mlm.py: 120 passed. The new tests cover: per-diagram machines extend the union machine; the union walk is unchanged; the walk along a diagram reports that diagram; a one-diagram process clusters the same both ways; the madevent history is always the union or the diagram; each diagram uses its own line flavours; ptclus for a non-merging radiating quark.
• madspace Python suite (excluding test_pdf and the torch-dependent tests)
• all_diagrams runs are bit-identical (event records) to samples generated before this change with the same seed and cards (VBF, HEFT, HEFT QED=0)
• MG5 suite (./tests/test_manager.py): 1549 tests, 3 pre-existing reds
  (test_output_standalone_accepts_loop_induced,
  test_output_standalone_cpp_refuses_loop_induced, MadLoop
  testIO_UnitProcOutputIOTests), none of them touching the run card or banner

Rebased onto claude/madspace-madevent-scales at 8a1b37c (which carries the
main merge and the updated feat-mlm); all four commits applied cleanly, and
both suites above were re-run on the merged base.

The generation runs were made with these four commits on 87c4967, i.e. before
that rebase. They cannot currently be repeated on the merged base: any mg7 run
with phasespace.xqcut > 0 fails there, before any of this code is reached.
The main merge renamed the madspace observable m_inv to pair_mass, while
apply_auto_ptj_mjj in mg7/launch.py still does

for key in ("jet-pt", "jet-m_inv"):
    cuts.setdefault(key, {})["min"] = xqcut

so ms.Observable(all_pids, observable='m_inv', ...) raises TypeError for every
MLM run - setdefault creates that cut even for a card that never had one, so
fresh process directories are affected too. Confirmed to be pre-existing and
untouched by this PR: 8a1b37c checked out on its own, with its own madspace
built from source, fails identically on the same process directory (reported to
the branch owner). The rebase changed no clustering, scale or
integrand code, so the numbers above stand; I will re-run the VBF cross-check on
the merged base once that is fixed.

🤖 Generated with Claude Code