Skip to content

mg7: npy events by default, npy->LHE converter - #195

Merged
theoheimel merged 15 commits into
mainfrom
claude/madspace-default-npy-output-495881
Sep 23, 2026
Merged

mg7: npy events by default, npy->LHE converter#195
theoheimel merged 15 commits into
mainfrom
claude/madspace-default-npy-output-495881

Conversation

@oliviermattelaer

@oliviermattelaer oliviermattelaer commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Includes #188 and #190 (merged in; merge those first or close them in favour of this one). Merge decisions:

  • the automatic switch to LHE is kept (this branch's version), not Bugfixes of final and intermediate madspace output #188's error;

  • Bugfixes of final and intermediate madspace output #188's header.lhe replaces the lhe_meta.json this PR first added: npy_to_lhe copies it verbatim as the LHE header;

  • mg7: default histograms, plots with bands, and user defaults for the launch switches #190's plots, HwU output, default histograms and input/default_switch.txt are taken as they are.

  • Default output_format is now lhe_npy (was lhe): the complete LHE record (pdg, status, mothers, colour, helicity) in npy form; compact_npy stays available.

  • Automatic switch to lhe before generation, only when something will read the LHE file:

    • a post-processing driver mg7 actually runs (reweight, MadSpin, MA5, Pythia8, Delphes, Rivet);
    • [postprocessing] time_of_flight >= 0;
    • legacy [postprocessing] systematics with native [systematics] disabled.

    analysis = ExRoot (the default wherever ExRootAnalysis is installed) no longer forces LHE, since mg7 has no ExRoot driver. The run_card is read as raw TOML and only its output_format line is edited, so scan:[...] cards survive (a full RunCardMG7 parse crashed on them).

  • npy→LHE converter (madgraph/iolibs/template_files/mg7/npy_to_lhe.py) for when LHE is needed after the run:

    • compact_npy runs also save lhe_completer.json; the header comes from Bugfixes of final and intermediate madspace output #188's header.lhe;
    • the converter rebuilds events.lhe.gz from these: colour, helicity, resonances/mothers, <rwgt> and <initrwgt>;
    • entry points: bin/npy_to_lhe [RUN|dir|events.npy] [-o OUT] [--seed N] [--no-gzip] (written at output time), the npy_to_lhe run command, and automatically when a post-processing tool runs on a run that only has npy.

  • Gridpacks default to LHE: grid_run_card.toml always gets output_format = "lhe" whatever the run used (bin/generate_events --output_format still selects npy). test_gridpack_mg7 checks the card and that events.lhe.gz is produced (passes locally).

  • Complete gridpack LHE headers: save_gridpack stores the cards, beams and PDF in data/lhe_meta.json. The gridpack adds its own cross section and seed, for both the lhe output and header.lhe (before: only <initrwgt>, <init> all zeros). A compact_npy gridpack run also gets lhe_completer.json, so it can be converted.

  • Histogram switches in [run]:

    • weighted_histograms (default off): fills the histograms with the weighted events during integration;
    • postprocessing_histograms (default on): fills them with the final events and their variation weights, which the plots and the HwU file use.

    The scan-summary means now come from the post-processing histograms when present, and no longer count the under/overflow bins as in-range bins.

  • selected_tools() is now shared by the post-processing runner and the LHE check. The tool list is checked before looking for or converting an event file.

Validation

  • p p > e+ e- j, 5000 events, same seed: a compact_npy run (converted) and an lhe_npy run (converted) were each compared with a direct lhe run. They match event by event on momenta, pdg, status, mothers, colour, helicity, weight, scale and systematics weights. The header differs only in the recorded output_format. Conversion takes about 1 s for 5k events.
  • New unit tests: TestForceLHEOutput (default, no-op, tool, time_of_flight, legacy systematics, scan card, missing line) and TestNpyToLHE (lhe_npy round trip, default run pick, pre-existing run without header). test_mg7_launch, test_banner and the MadSpin unit tests pass.
  • test_mg7_reproducibility now sets output_format = "lhe" because it hashes the LHE file. The acceptance suite was not run.

Limits

  • The <mgrwt> block ([systematics] write_inputs) is not rebuilt by the converter.
  • compact_npy conversion re-draws the completer's random choices, so it is statistically equivalent to a direct LHE run, not guaranteed byte-identical (it was identical in the test above).
  • Runs made before this change lack header.lhe and cannot be converted; the converter says so.
  • Gridpacks made before this PR have no data/lhe_meta.json and keep the old minimal header.

🤖 Generated with Claude Code

theoheimel and others added 10 commits September 20, 2026 16:28
@theoheimel
bugfixes of final and intermediate madspace output
547f5c3
@theoheimel
write LHE header info in numpy output mode
3a9f5d8
@oliviermattelaer @claude
user defaults for the launch switches, in input/default_switch.txt
4dd595e 
Restore the automatic switch to output_format = 'lhe' when a post-processing
tool is selected on an mg7 run: the npy formats carry no event record those
tools can read, but asking for a shower in the launch question should be
enough -- the user does not have to know that it constrains output_format.

Add input/default_switch.txt (materialised from the shipped, fully commented
input/.default_switch.txt), the switch counterpart of
input/default_run_card_lo.dat: "analysis = MadAnalysis5" there is the file
version of typing "analysis=MadAnalysis5" at every launch. It is read by
ControlSwitch, so the same file serves the LO madevent, NLO aMC@NLO and mg7
questions; each ignores the keys it does not have. Entries go through the
usual validation and consistency resolution, so an unavailable program is
reported and skipped rather than silently accepted, and an NLO
"fixed_order = ON" still drags the shower default off.

Along the way:
 - the run_card default files were read from MG5DIR/internal/, a directory
   that does not exist: input/default_run_card_{lo,nlo}.dat and
   input/default_run_card_mg7.toml were therefore never applied.
 - only default_run_card_lo.dat was checked before materialising all three,
   so an installation predating the mg7 one never got it.
 - AskRun.get_allowed_reweight returned None on its first call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: default histograms for the process, written at output time
81e9a13 
[histograms] was an empty section nobody filled in. The observables it takes
are named after the final state, so they can only be written once the process
is known: RunCardMG7.create_default_for_process now fills it with the pt and
eta of every final-state particle, the invariant mass of every pair of them,
the partonic sqrt(s) and the distribution of the event weight.

The particles are named after the [multiparticles] groups, with the "_1",
"_2", ... suffix picking the hardest, second hardest, ... of a group:
`p p > t t~ j` gets t_1-pt, t_2-pt, jet-pt, the three etas, the three pair
masses and sqrt_s. A particle is histogrammed under a predefined group (jet,
lepton, ...) when the run time cannot tell its members apart -- MadGraph
merges the flavours of a group into a single subprocess and only the merged
representative reaches the observables, so `p p > l+ l-` is "lepton", not "e"
and "mu" (the latter would silently select nothing). A particle that matches
none of that, like the top, gets a [multiparticles] group of its own. The
multiplicity is the smallest over the subprocesses, since asking for the n-th
hardest particle of a group is an error in a subprocess that has fewer.

The ranges are order-of-magnitude guesses from the collider energy (or the
mass of the decaying particle for a 1 -> N width), rounded to 1/2/5 x 10^k.
On p p > t t~ j at 13.6 TeV every histogram holds the cross section with
under 1.2% in the overflow bin.

"weight" is the one [histograms] key that is not an observable of the
momenta. madspace gained EventHistogramSpec::from_weight for it: the event
weight divided by the reference weight (the cross section, i.e. the mean
weight), so the axis does not depend on the process -- a fully unweighted
sample is a spike at 1 and a partially unweighted one shows its spread.

In the launch question, "set histograms OFF" removes them all and
"set histograms default" puts the generated ones back from
Cards/run_card_default.toml.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: write the event histograms as MADatLO.HwU
cc1f65c 
The histograms only existed as JSON inside Events/<run>/info.json, which
nothing else in MadGraph reads. HwU is the format the rest of the code speaks
-- an aMC@NLO run writes MADatNLO.HwU and madgraph/various/histograms.py
plots and compares those -- so an mg7 run now writes the same thing next to
its events, and an LO distribution can be overlaid on an NLO one with no
conversion step in between.

Everything info.json carries goes into the file: the nominal distribution, one
column per systematics variation, and the scale/PDF bands. The variation
columns are labelled "muR=1.00 muF=2.00" / "PDF= 331901" (and
"dyn=3 muR=..." for a dynamical scale), which is what makes histograms.py read
them as an envelope and a band rather than as anonymous curves -- the envelope
it rebuilds from them is bin-by-bin the one madspace computed. The group a
weight belongs to decides its label, not the variation's own fields: the
central member of a PDF set carries mur = muf = 1 and would otherwise be
written as a scale variation, hiding it from the band.

Two conventions of the format are applied on the way out, both of them where a
converter goes wrong: HwU bins hold dsigma/dx, so every column is divided by
the bin width (sum(central * width) is then the cross section), and HwU has no
under/overflow bins, so the first and last entries of the madspace arrays are
dropped, as an aMC@NLO analysis does.

The conversion is a pure function of the two dicts madspace writes, in
hwu_output.py, which imports nothing. The tests read the result back with
madgraph.various.histograms rather than checking the text against itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: write_hwu switch, off by default
89f3325 
MADatLO.HwU is a second copy, in another format, of numbers Events/<run>/
info.json already carries, so it is now opt-in rather than written by every
run: [run] write_hwu in the run_card, false by default, or "set write_hwu
True" in the launch question.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: draw the histograms of a run, bands included, into Events/<run>/…
4159fab 
…plots

A run had no picture of anything: info.json held the distributions and
MADatLO.HwU could be fed to histograms.py by hand, but nothing was ever drawn.
Every run now writes one PDF per observable into Events/<run>/plots and says
where they went ("11 plot(s) stored in Events/run_01/plots").

The drawing reads the event-sample histograms straight from madspace rather
than going through the HwU file, which buys the bands: the scale envelope and
the PDF uncertainty are already computed per bin there, while histograms.py
has to rebuild the PDF band from the member weights and needs the LHAPDF
python module to do it -- not installed everywhere, and broken on python 3.14.
Each plot is the differential cross section with its statistical errors, the
bands behind it, and a second panel with the bands relative to the nominal.
The y axis goes logarithmic when the spectrum falls by more than a factor 50,
which is what tells a pt spectrum from an eta one.

The conventions are the ones the other two consumers of these numbers already
use, so the three agree: dsigma/dx on the y axis, and no under/overflow bins
(HwU has none, and MadBoard's browser plots slice them off too).

[run] make_plots, true by default, turns it off. matplotlib is imported
lazily and only inside plots.py: a run without it says so once and carries on,
every number being in info.json either way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: run through the interpreter that wrote the output, and let it be…
b2dfef8 
… quiet

Two things a run got wrong on its own stdout.

bin/generate_events trusted "#! /usr/bin/env python3". A run needs the
packages of the environment MadGraph7 was started from -- matplotlib for the
plots, the LHAPDF bindings, madspace -- and the python3 the PATH resolves to
often has none of them: on this setup the plots were silently skipped for
exactly that reason. The launcher now pins sys.executable as its shebang and,
because a shebang is only a default (bypassed by "python3 bin/generate_events",
ignored by the kernel when the path is too long), re-executes itself through
that interpreter whenever it finds it is running somewhere else. Both paths
draw the plots now. The path is deliberately not resolved through its
symlinks: in a virtual environment sys.executable points at the environment,
and the base interpreter it links to does not see its site-packages -- taking
the realpath handed the run a python without matplotlib. A vanished
interpreter is ignored rather than fatal, and the gridpack, which is the
portable artefact, keeps its "/usr/bin/env python3".

matplotlib narrates every font it embeds in a PDF at DEBUG level, which was a
dozen lines per plot -- 28549 of them for one run here -- as soon as anything
let DEBUG records through the root logger. Its loggers are now pinned to
WARNING, before the import rather than after: matplotlib logs its own start-up
while it is being imported, and a logger silenced afterwards is silenced four
lines too late. An explicit level above WARNING is left alone.

The launcher text moved into mg7_launcher_source() so that the re-exec can be
tested by running it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
test_systematics_mg7: replace the [histograms] section, do not append…
a6dbfb1 
… to it

The test pasted its one sqrt_s histogram right after the [histograms] header,
which was fine while that section was always empty. The output now writes
process-specific default histograms there, sqrt_s among them, so the edited
card defined sqrt_s.min twice: a TOML error ("Cannot overwrite a value") on
which generate_events died at once. It died with exit code 0, so the helper's
return-code check never printed the log and the test reported "no mg7 run
directory" instead.

The test also relies on its histogram being the only one (it reads
event_histograms[0] and expects 10 bins), so the section's content is now
replaced as a whole, which keeps that true whatever the defaults generate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: write npy events by default, convert to LHE on demand
7f41edb 
output_format now defaults to compact_npy. The run_card is switched to lhe
before the generation only when something reads the LHE file: a
post-processing driver mg7 actually runs (reweight, MadSpin, MA5, Pythia8,
Delphes, Rivet -- not ExRoot, which has no mg7 driver), a time_of_flight
threshold, or legacy systematics.py without madspace's native weights. The
card is read as raw TOML and only its output_format line is edited, so a
run_card scan survives.

npy runs now keep lhe_meta.json (LHE header/<init>) and, for compact_npy,
lhe_completer.json. The new npy_to_lhe module rebuilds events.lhe.gz from
them (colour, helicity, resonances, rwgt/initrwgt), via bin/npy_to_lhe, the
npy_to_lhe run command, or automatically when a post-processing tool runs on
a run that only has npy events. Validated event by event against a direct
lhe run with the same seed (p p > e+ e- j, 5000 events).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@theoheimel

theoheimel commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

I'd propose lhe_npy instead of compact_npy because it has the full LHE info

@oliviermattelaer @claude
mg7: default output_format to lhe_npy instead of compact_npy
047804d 
lhe_npy keeps the complete LHE record (pdg ids, status, mothers, colour,
helicity) in the npy file, so the events are usable without the LHE
completer and convert to LHE by plain reformatting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@theoheimel

theoheimel commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Also #188 introduces a header.lhe serving the same purpose as the lhe_meta.json added here, so we should decide on one of these options

oliviermattelaer and others added 4 commits September 23, 2026 15:45
@oliviermattelaer @claude
mg7 gridpack: default its output to LHE
06382a3 
The gridpack's grid_run_card.toml copied the run's output_format, so with
the new npy default a gridpack wrote npy events. A gridpack feeds a
shower/detector chain, so its card now always says lhe (--output_format
still picks an npy output). test_gridpack_mg7 checks the card and that
events.lhe.gz is produced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@oliviermattelaer @claude
Merge #188 (fix-output-formats) into the npy-default branch
d1cdffb 
Take #188's header.lhe (LHE <header>/<init> written next to npy events),
ExRoot-off-by-default and prompt cleanup of the per-channel npy files.

Conflicts:
- keep the automatic switch to LHE (force_lhe_output_if_needed) instead of
  #188's check_lhe_output_required error: with npy the default, raising
  would stop every run that turns a shower on;
- drop lhe_meta.json: npy_to_lhe now copies header.lhe verbatim (it carries
  the cards, beams, cross section, MG7Seed and <initrwgt>) and takes the
  rwgt columns from the npy dtype; lhe_completer.json is kept for
  compact_npy (save_lhe_completer);
- output_format default stays lhe_npy, with #188's comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@oliviermattelaer @claude
Merge #190 (mg7-histograms-and-switch-defaults) into the npy-default …
ba4962b 
…branch

Takes #190's default [histograms], plots/HwU output and input/default_switch.txt.
Both branches restore the automatic switch to LHE; keep this branch's
force_lhe_output_if_needed (raw-TOML edit that survives scan cards, driven by
selected_tools() and the [postprocessing] steps), with #190's rationale in
its docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@oliviermattelaer @claude
mg7: complete gridpack LHE headers; switches for the two histogram kinds
da9605d 
Gridpack header: save_gridpack stores the run-independent LHE header
(proc/param/run cards, beams, PDF, weight mode) in data/lhe_meta.json; the
gridpack builds its LHEMeta from it with the cross section and seed of its
own run, for the lhe output and for header.lhe (which had only <initrwgt>
and an all-zero <init>). A compact_npy gridpack run also gets
lhe_completer.json, so npy_to_lhe can convert it. Gridpacks made before
this fall back to the old minimal header.

Histograms: [run] weighted_histograms (default false) fills the [histograms]
with the weighted events during integration; postprocessing_histograms
(default true) fills them with the final events and their variation
weights, which is what the plots and the HwU file use. The scan-summary
means now come from the post-processing histograms when present, and no
longer count the under/overflow bins as in-range bins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 23, 2026
Closed
Closed
@theoheimel
theoheimel merged commit bbdd779 into main Sep 23, 2026
176 checks passed
@theoheimel
theoheimel deleted the claude/madspace-default-npy-output-495881 branch September 23, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@oliviermattelaer @theoheimel