Build all SubProcesses in parallel from a single make jobserver

[oliviermattelaer]

Running make -j 18 in SubProcesses/ now spreads 18 concurrent compilations over all the P* directories at once — instead of building one directory at a time with 18 jobs, or giving each directory 18/N.

This is the make-level counterpart of #61, which did the same thing with a ThreadPoolExecutor in mg7/madevent.py.

Mechanism

The parallelism comes from GNU make's jobserver. SubProcesses/makefile becomes a plain recursive dispatcher that fans out with +$(MAKE) -C <dir>, so its -j slots are shared with every sub-make: a directory that runs out of work immediately hands its slots to the others.

File layout

SubProcesses/makefile used to be the per-process build rules (symlinked into each P*). That name is now the dispatcher, so the files are rearranged:

file	role
SubProcesses/makefile	the dispatcher (new template madmatrix_subprocesses.mk)
SubProcesses/madmatrix.mk	the build rules, linked as each P*/makefile
SubProcesses/madmatrix_standalone.mk	ditto for standalone_mg7 (it includes madmatrix.mk)

The dispatcher owns no build logic of its own. It even builds the common library through a representative P* directory, so the BACKEND/cppauto resolution, the compiler flags and the src/, lib/ paths stay defined in madmatrix.mk only.

The race that had to be fixed first

Every P* makefile recursed into src/ to build the shared common library. With N directories running at once they would all have done that simultaneously, on the same objects. The dispatcher builds it once, up front, and passes MADMATRIX_COMMONLIB_EXTERNAL=1 to say it owns that step; a P* directory built on its own is unaffected.

The bldall backend list is collapsed into a single BLDBACKENDS variable, so multi-backend builds can be driven from the dispatcher without duplicating the platform logic.

Python side

mg7 compiled one subprocess per MadgraphSubprocess. It now resolves cppauto once and makes a single make -jN call on SubProcesses/ (N = cpu_thread_pool_size, -1 meaning the CPU count). One make invocation replaces the per-subprocess loop.

Incidental

find_output_type told standalone_mg7 apart by the presence of SubProcesses/madmatrix.mk, which the regular mg7 export now also has; the discriminator moves to madmatrix_standalone.mk.

Testing

p p > e+ e- + p p > e+ e- j (4 subprocesses), both mg7 and standalone_mg7:

• scaling on an 18-core machine: -j1 12.3s → -j4 4.0s → -j18 2.1s; the common library is compiled exactly once
• clean, cleanall, bldall (2 backends, one common lib each), make P2_gQ_epemQ, BACKEND=… USEBUILDDIR=1, plain make, incremental no-op, and a plain make inside a single P* directory
• a deliberate compile error reports make: *** [build@P2_gQ_epemQ] Error 2, so the failing directory is still named
• check_sa.exe runs; full bin/generate_events completes in 5.2s including the build
• acceptance tests test_standalone_mg7_goodhel_filter, test_standalone_mg7_vs_cpp, test_output_mg7_directory pass (test_group_subprocess_mg7 skips)

Note: the systematics computation failed message at the end of generate_events is pre-existing — reproduced on a stashed baseline.

🤖 Generated with Claude Code

[Qubitol]

Nice!

I think the only thing that is missing would be a progress indicator of the status of the compilation, if we are still interested in adding it (see #6), of course.

[oliviermattelaer]

Ok I added some log for the compilation of the SubProcesses (with information where more detailled log are kept).

Let's merge?

[Qubitol]

Good for me!

[oliviermattelaer]

Thanks !!!