Backend separation 2 - #166
Conversation
depending on backend and pre-stored in separate folders template files need to be prunned and make it compiled
delete Bridge. Resolving the not nested #ifdefs. next the .inc in template_files
ProcessData.h ColorMatrixData.h process specic data
Relocate sigmaKin, calculate_jamps, computeDependentCouplings, getGoodHel/
setGoodHel, and color_sum_cpu/color_sum_gpu out of the per-process
CPPProcess.cc into backend-owned SigmaKin.{h,cc} and color_sum.cc under
backend/{cpu,simd,gpu}/.
The storage they operate on (cHel/cFlavors/cIPD/
cIPC/cIPF_*/bsmIndepParam/good-helicity cache) moves with them as generic,
backend-owned declarations sized via nIPD/nIPC/ncomb/etc. - constexpr
constants pulled from each process's own ProcessData.h/ProcessTables.h at
compile time, the same pattern already used by color_sum.cc (ncolor from
ColorMatrixData.h).
CPPProcess's constructor/initProc populates this
storage through new setters (setHelicitiesAndFlavors, setIndependentParams,
etc.) instead of writing it directly.
+ more resolution of #ifdefs paths to anhilatethem
Split for review: this slice lands the storage/setter plumbing only
(SigmaKin.h declarations, storage + setters + computeDependentCouplings +
getChannelId in SigmaKin.cc, CPPProcess.cc/.h wiring); calculate_jamps,
getGoodHel and sigmaKin's bodies follow in the next commits.
…tax validation
backend/{cpu,simd}/mgOnGpuFptypes.h dropped the non-GPU fpsignbit overload during
the commit-2 backend split resolution (theirs=empty was a hunk-boundary
artifact, not intentional, same class of bug documented earlier in this
port); HelAmps_<model>.h calls it directly, so every generated process
failed to compile. Restored, matching main's flat file.
backend/{cpu,gpu,simd}/SigmaKin.h declared fptype_amp_sv-typed parameters
(GPU's sigmaKin_getGoodHel) and will need cxtype_amp_sv/fptype_colour_sv
for calculate_jamps next, none of which mgOnGpuCxtypes.h alone provides;
added the mgOnGpuVectors.h include. Both caught by real clang -fsyntax-only
against a freshly generated process, not just #if/#endif balance.
Relocate sigmaKin, calculate_jamps, computeDependentCouplings, getGoodHel/
setGoodHel, and color_sum_cpu/color_sum_gpu out of the per-process
CPPProcess.cc into backend-owned SigmaKin.{h,cc} and color_sum.cc under
backend/{cpu,simd,gpu}/.
The storage they operate on (cHel/cFlavors/cIPD/
cIPC/cIPF_*/bsmIndepParam/good-helicity cache) moves with them as generic,
backend-owned declarations sized via nIPD/nIPC/ncomb/etc. - constexpr
constants pulled from each process's own ProcessData.h/ProcessTables.h at
compile time, the same pattern already used by color_sum.cc (ncolor from
ColorMatrixData.h).
CPPProcess's constructor/initProc populates this
storage through new setters (setHelicitiesAndFlavors, setIndependentParams,
etc.) instead of writing it directly.
+ more resolution of #ifdefs paths to anhilatethem
Split for review, slice 2 of N: calculate_jamps' body (prologue, EvaluateDiagrams.inc
#include, epilogue) now lives in SigmaKin.cc for all three backends, rebuilt from
main's current mixed-precision logic rather than the branch's pre-mixed-precision
version; validated by real generation + clang -fsyntax-only against the actual
diagram code for cpu/simd (gpu can't be syntax-checked on this machine - no CUDA
toolchain - so it's built by the same transformation but unverified by compilation).
sigmaKin/getGoodHel bodies still follow in the next commit.
Relocate sigmaKin, calculate_jamps, computeDependentCouplings, getGoodHel/
setGoodHel, and color_sum_cpu/color_sum_gpu out of the per-process
CPPProcess.cc into backend-owned SigmaKin.{h,cc} and color_sum.cc under
backend/{cpu,simd,gpu}/.
The storage they operate on (cHel/cFlavors/cIPD/
cIPC/cIPF_*/bsmIndepParam/good-helicity cache) moves with them as generic,
backend-owned declarations sized via nIPD/nIPC/ncomb/etc. - constexpr
constants pulled from each process's own ProcessData.h/ProcessTables.h at
compile time, the same pattern already used by color_sum.cc (ncolor from
ColorMatrixData.h).
CPPProcess's constructor/initProc populates this
storage through new setters (setHelicitiesAndFlavors, setIndependentParams,
etc.) instead of writing it directly.
+ more resolution of #ifdefs paths to anhilatethem
Split for review, slice 3 of N: sigmaKin_getGoodHel/setGoodHel now live in
SigmaKin.cc for all three backends, rebuilt from main's current mixed-precision
logic (including the two-neppV-page merge for FPTYPE_DOUBLE+FPTYPE2_FLOAT mode);
validated by real generation + clang -fsyntax-only for cpu/simd, which on this
Apple Silicon host auto-enables MGONGPU_CPPSIMD, so the mixed-precision branches
were actually exercised, not just the scalar fallback (gpu still unverified by
compilation - no CUDA toolchain here). sigmaKin's own body (BLAS helicity
batching, channel/color selection) is the last remaining piece.
Relocate sigmaKin, calculate_jamps, computeDependentCouplings, getGoodHel/
setGoodHel, and color_sum_cpu/color_sum_gpu out of the per-process
CPPProcess.cc into backend-owned SigmaKin.{h,cc} and color_sum.cc under
backend/{cpu,simd,gpu}/.
The storage they operate on (cHel/cFlavors/cIPD/
cIPC/cIPF_*/bsmIndepParam/good-helicity cache) moves with them as generic,
backend-owned declarations sized via nIPD/nIPC/ncomb/etc. - constexpr
constants pulled from each process's own ProcessData.h/ProcessTables.h at
compile time, the same pattern already used by color_sum.cc (ncolor from
ColorMatrixData.h).
CPPProcess's constructor/initProc populates this
storage through new setters (setHelicitiesAndFlavors, setIndependentParams,
etc.) instead of writing it directly.
+ more resolution of #ifdefs paths to anhilatethem
Split for review, slice 4 of 4 (final): sigmaKin's own body now lives in
SigmaKin.cc for all three backends - the BLAS helicity-batching loop (gated
by MGONGPU_CPP_HAS_BLAS + the per-process ColorMatrixData::shouldUseBlas
constexpr from commit "Put the process specific constants in the files"),
broken_symmetry_factor, and channel/color selection, rebuilt from main's
current process_sigmaKin_function.inc rather than the branch's pre-BLAS
version. Also fixes a real bug this surfaced: coloramps.h is process-specific
(like ColorMatrixData.h), so the raw unsubstituted copy commit 1 triplicated
into backend/{cpu,gpu,simd}/ was shadowing the real one via same-directory
quote-include; edit_coloramps() now writes a real substituted override into
each process's own backend/<variant>/coloramps.h, same pattern as the
color_sum.cc split-orders override.
Validated by real generation + clang -fsyntax-only against the actual
generated process for cpu/simd, in all four combinations of
BLAS/no-BLAS x cpu/simd - all clean. gpu is built by the identical
transformation but still unverified by compilation (no CUDA toolchain on
this host); the residual errors when syntax-checking it are the same
namespace-lookup artifacts that an already-unmodified, pre-existing gpu
file (MatrixElementKernels.cc) exhibits under the same ad-hoc clang
invocation, confirming they predate this change rather than being
introduced by it.
This completes "Backend algos from CPPProcess to /backend" (commit 4 of 26
in the original backend_separation history).
Include right backend and copy right files.
Also supersedes this port's own earlier workaround for the same bug (a
per-process coloramps.h override written directly into backend/{cpu,gpu,simd}/
during "Backend algos from CPPProcess to /backend"): that fix worked, but this
is the branch's real, canonical one - INCFLAGS/vpath make the Makefile find
the P1-level substituted coloramps.h itself once the shadowing stray copy in
backend/ is gone, no per-backend duplication needed. Reverted that workaround
accordingly. Validated by real generation + clang -fsyntax-only against the
actual generated process, passing -I. -Ibackend/cpu exactly as the new
INCFLAGS does.
Also removes _link_backend_dirs_in_P entirely (superseded by this commit's own Makefile change - INCFLAGS/vpath now reach the single top-level backend/<variant>/ directly via ../../backend/$(BACKENDDIR), so per-process symlinking is unnecessary) and, as a consequence, this port's split-orders override mechanism from "Backend algos from CPPProcess to /backend" (which wrote into that now-nonexistent per-process backend/ copy). Rather than leave it silently writing to a location nothing will ever compile - which would have made a split-order process silently use the wrong, non-split color sum - edit_colorsum() now raises a clear error for that case instead, pending a real fix (folding color_sum_splitorders.cc into the shared color_sum.cc behind a compile-time flag, matching ColorMatrixData::shouldUseBlas). Validated by real generation + clang -fsyntax-only for cpu/simd using the exact relative include path the real Makefile now uses (-I. -I../../backend/<variant>, run from the P* directory) - both clean.
cleaning of the EvaluateDiagrams.inc Also fixes a real, latent bug this commit's Makefile hunk exposed: main independently renamed the C++ backend identifiers since this branch's fork (cppnone/cppsse4/cppavx2/cpp512y/cpp512z/cppauto -> scalar/simd_128/simd_256/ avx512y/simd_512/auto), but BACKENDDIR's resolution (added in "Makefile and stray file deletion") still matched on the old name 'cppnone' - meaning every scalar (non-SIMD) build would have silently compiled against backend/simd/ instead of backend/cpu/. Fixed to match 'scalar'; confirmed via `make -p BACKEND=scalar | grep BACKENDDIR` (now cpu) and BACKEND=simd_128 (still simd). Validated by real generation + clang -fsyntax-only for cpu/simd (both BLAS/no-BLAS) using the real Makefile-relative include path.
only in standalone_mg7 copy needed files - e.g. check_sa.cc
delete them from /backend and delete the death code files
Retires epoch_process_id.h and processConfig.h entirely (their content -
MG_EPOCH_PROCESS_ID/MG_EPOCH_REFERENCE_FILE_NAME macros and ndiagrams -
folds into ProcessData.h, already the single source of truth for everything
else process-specific); deletes the fbridge/fsampler Fortran-bridge files and
the whole gtest/RandomNumberKernel/rambo family from backend/{cpu,gpu,simd}/,
none of which the standalone_mg7 output mode needs (they were wrongly
triplicated in "Outputting into 3 diff folders" along with coloramps.h and
testxxx.cc, same class of bug already found and fixed in "Redundant template
file delete" and the fpsignbit fix).
Also caught along the way: this whole session's validation had been
exercising `output standalone_mg7 <path>` - not a registered format name, so
MadGraph silently falls back to the plain 'mg7' exporter (no check_sa.cc,
no RamboSamplingKernels, etc.), meaning check_sa.cc itself - the actual
check_sa.exe test driver this branch's numerical validation methodology
relies on - had never actually been compiled in this port until now. Switched
to the real `output standalone <path>` format for this and future validation.
Validated by real generation with the correct 'standalone' format + clang
-fsyntax-only for cpu/simd (BLAS and no-BLAS) against both SigmaKin.cc and
check_sa.cc, using the real Makefile-relative include path
(BACKENDDIR correctly resolves to cpu for BACKEND=scalar).
more prune for standalone_mg7 output
Merges ColorMatrixData.h and coloramps.h into one process-specific file
(ColorData.h): both were per-process substituted headers the backend-owned
color_sum_cpu/color_sum_gpu and channel/color selection code needed, so
having them separate was pure duplication once the backend split settled
down. The ColorMatrixData namespace (and this port's shouldUseBlas addition)
carries over unchanged. Also moves nvtx.h/rambo.h out of the base
SubProcesses/to_link_in_P lists into the 'standalone' exporter's
_standalone_extra_files, since only the standalone driver needs them.
Caught and fixed a real, previously undetected bug while re-validating this:
color_sum.cc's mixed-precision "MEs_next" scatter (restored during "Backend
algos from CPPProcess to /backend" after the raw patch silently dropped it)
had its guard weakened to #if FPTYPE_DOUBLE+FPTYPE2_FLOAT, missing the
MGONGPU_CPPSIMD check that its own deltaMEs_next declaration still requires -
copied uncritically from SIMD's already-simplified version, where CPPSIMD is
always true so the omission was harmless. For CPU, FPTYPE_DOUBLE+FPTYPE2_FLOAT
without CPPSIMD is a real, valid build (the "v" mixed-precision mode without
vectorization), and referenced an undeclared variable. Found because this is
the first time color_sum.cc was syntax-checked as its own translation unit
rather than only transitively through files that call it via color_sum.h -
now doing that for every .cc file in backend/{cpu,simd}/ going forward.
Validated by real generation (correct 'standalone' format) + clang
-fsyntax-only for every .cc file in backend/{cpu,simd}/, across all four
combinations of {mixed, plain} precision x {BLAS, no-BLAS} - all clean.
delete the Gpu* files from cpu and simd. More prune of death code and #ifdefs Re-derived against main's mixed-precision rewrite: GpuAbstraction.h/GpuRuntime.h were confirmed dead for cpu/simd (never MGONGPUCPP_GPUIMPL there) and dropped along with their now-unused includes, while flat files shared with the gpu backend (check_sa.cc, CommonRandomNumberKernel.cc, RamboSamplingKernels.cc) got the include guarded behind #ifdef MGONGPUCPP_GPUIMPL instead of removed outright; validated by regenerating g g > t t~ and building/running both the scalar and simd_128 backends end to end, with identical matrix-element output.
Re-derived against main's mixed-precision src/ layout: main's own src/ never held these files, so stopped writing the stale, pre-backend-split flat mgOnGpu*.h/constexpr_math.h copies into src/ (they were shadowing the correct per-backend backend/<variant>/ versions via -I. and were a latent correctness risk for anything backend-dependent); TREX still lacks the /backend split so its own exporter keeps copying them and calling edit_mgonGPU() as before. Validated by regenerating g g > t t~ and building/running both the scalar and simd_128 backends end to end, with identical matrix-element output and Parameters.cc now resolving these headers via the new -I../backend/$(BACKENDDIR) path.
fpeEnable(), getCompiler() move in constexpr_math.h and mgOnGpuConfig.h tHel/tFlavors/flavorPDGs in ProcessData getCompiler had zero calls -> dropped mass/tIPD/tIPC setup (non-hardcoded branch) is now a generated pointer-to-member table read by generic gatherFptype/ gatherCxtype helpers in Parameters.h, instead of one assignment per parameter. The MGONGPU_HARDCODE_PARAM branch same TODO Re-derived against main's mixed-precision Parameters.h/ProcessData.h: the pointer-to-member gatherFptype/gatherCxtype helpers, tHel/tFlavors move, and fpeEnable()/getCompiler() relocation applied cleanly since none of main's own additions touched this code path. Validated by regenerating g g > t t~ and building/running both the scalar and simd_128 backends end to end (identical matrix-element output to before this commit), and confirming CPPProcess.cc still compiles standalone under MGONGPU_HARDCODE_PARAM (the get_hardcoded_initProc_lines path is unaffected by this refactor, as the commit message's own TODO notes). Found and confirmed pre-existing (predates this commit, not caused by it): building with HRDCOD=1 fails in SigmaKin.cc because ProcessTables::nMF (from FLV_COUPLING::max_flavor) is not a constant expression under MGONGPU_HARDCODE_PARAM; filed separately rather than fixed here since it's unrelated to this commit's scope.
unified namespace madgraph instead of mg5amcGpu/mg5amcCpu.
Fix for some functions depending on vector lane size
Re-derived as a mechanical namespace rename (mg5amcCpu/mg5amcGpu ->
madgraph) across all backend/{cpu,simd,gpu} and shared flat template
files, collapsing the #ifdef MGONGPUCPP_GPUIMPL / namespace mg5amcGpu
#else / namespace mg5amcCpu #endif conditional blocks into a single
namespace madgraph now that GPU no longer shares content with cpu/simd
via that mechanism. Applied the real bug fix (fpvmerge/fpvsplit0/
fpvsplit1 in mgOnGpuVectors.h only handled MGONGPU_CPPSIMD==2, silently
producing wrong mixed-precision denom64 results on wider AVX2/AVX512
vector widths) to both backend/cpu and backend/simd copies, matching
upstream doing so even though the cpu copy is dead code there (CPPSIMD
is never defined for the cpu backend). Validated by regenerating
g g > t t~ and building/running scalar and simd_128 (including
FPTYPE=v, which exercises MGONGPU_SIMD_DENOM64 and the fixed
functions directly) end to end with correct, non-garbage matrix-element
output, plus a full standalone-syntax sweep across the usual
precision/BLAS combinations for both backends.
to get rid off more ifdefs by having noop instance for cpu and simd Re-derived directly against the current (post-namespace-unification) color_sum.h/process_function_definitions.inc: added the no-op inline definition to both backend/cpu and backend/simd color_sum.h, dropped the now-unnecessary #ifdef MGONGPUCPP_GPUIMPL guard around both createNormalizedColorMatrix() call sites in initProc, and fixed a leftover "madgraph::/madgraph::" comment artifact in ProcessTables.h from the prior namespace-unification commit's mechanical rename. Validated by regenerating g g > t t~ and building/running both the scalar and simd_128 backends end to end, with identical matrix-element output to before this commit.
Re-derived directly against the current ColorData.h/GpuAbstraction.h: dropped the compile-time hostChannel2iconfig duplicate array (which doubled channel2iconfig's storage on GPU builds) from ColorData.h, added gpuMemcpyFromSymbol to GpuAbstraction.h (CUDA and HIP), and moved the host-accessible copy into a lazily-populated getHostChannel2iconfig() helper local to runTest.cc, its only consumer. Validated the cpu-side (#ifndef MGONGPUCPP_GPUIMPL) branch by compiling and running a minimal standalone test against the real generated ColorData.h, and confirmed the rest of the backend (SigmaKin.cc, color_sum.cc, etc., none of which reference hostChannel2iconfig) still compiles and runs correctly by regenerating g g > t t~ and building/running the scalar backend end to end. runTest.cc itself could not be built directly (it needs gtest/gtest.h, not vendored in this environment, and isn't wired into the standalone exporter's file list either before or after this commit), so the GPU branch of the new helper was reviewed by hand against the original diff rather than compiled.
Re-derived directly against the current cpu backend files: the cpu backend now unconditionally #undef MGONGPU_CPPSIMD instead of running the AVX512/AVX2/SSE4.2/ARM-NEON auto-detection logic that only ever made sense for the simd backend, and the fpvmerge/fpvsplit0/fpvsplit1 mixed-precision functions (plus the fptype2_v operator<<) - dead code for cpu once CPPSIMD can never be defined there - were deleted outright rather than kept as unreachable #if-guarded code, per the file's own "Keeps living only on the SIMD /backend" note. Validated by regenerating g g > t t~, confirming MGONGPU_CPPSIMD really is undefined for the cpu backend even under -DMGONGPU_NOARMNEON, and building/running both the scalar and simd_128 backends end to end with unchanged matrix-element output.
Conditionaly included for the simd
Needed stuff for the cpu/gpu in mgOnGpuCxtypes
Re-derived against main's mixed-precision content: deleted
backend/{cpu,gpu}/mgOnGpuVectors.h entirely and moved everything still
needed into the corresponding mgOnGpuCxtypes.h, updating every
#include "mgOnGpuVectors.h" in cpu/gpu backend files to
"mgOnGpuCxtypes.h" and switching the three flat/shared templates
(cpp_hel_amps_h.inc, cpp_model_parameters_h.inc, process_h.inc) to the
__has_include("mgOnGpuVectors.h") pattern so they still pick it up for
simd but fall back to mgOnGpuCxtypes.h for cpu/gpu. Since main's
mixed-precision rewrite added a much larger set of _amp/_momenta/
_denom/_colour scalar-or-vector typedefs and helpers
(fpamp_of_mom/fpamp_scalar/cxpow/generic template fpternary,cxternary,
cxabs2) to this branch's mgOnGpuVectors.h than upstream's own version
ever had, moved that fuller content instead of upstream's smaller
diff, and also updated backend/cpu/SigmaKin.h and
backend/{cpu,gpu}/color_sum.cc (not touched by the upstream commit,
since upstream's own versions of those files never included
mgOnGpuVectors.h directly) to keep everything consistent. Validated by
regenerating g g > t t~, confirming CPPProcess.h/Parameters.h/
HelAmps_sm.h each resolved the __has_include fallback correctly for
cpu, and building/running both the scalar and simd_128 (FPTYPE=v, to
exercise the mixed-precision path) backends end to end with correct,
unchanged matrix-element output, plus a full standalone-syntax sweep
across the usual precision/BLAS combinations for both backends.
Re-derived directly against the current simd/mgOnGpuVectors.h: removed the remaining dead commented-out blocks (the "DANGEROUS!" using-decl leftover, the commented-out bool_v operator<<, print(), and fpvmake() blocks) and simplified the two "defined MGONGPU_CPPSIMD and defined MGONGPU_FPTYPE_DOUBLE and defined MGONGPU_FPTYPE2_FLOAT" outer guards to just the FPTYPE conditions, since this file is now simd-only and MGONGPU_CPPSIMD is therefore always defined here - the inner MGONGPU_CPPSIMD==2/4/8 width-dispatch branches (from the earlier vector-lane-size bugfix) are untouched. Validated by regenerating g g > t t~ and building/running both the scalar and simd_128 (FPTYPE=v, exercising the mixed-precision path through the simplified guards) backends end to end, with unchanged matrix-element output.
Re-derived as a mechanical namespace rename (madgraph -> madmatrix)
across all backend/{cpu,simd,gpu} and shared flat template files,
including the "NB: the madgraph namespace..." comment introduced by
the earlier unification commit, replaced verbatim with "One namespace.
Split ber backend." to match. Also renamed the same identifier in
color_sum_splitorders.cc, my own earlier addition for the split-orders
exception path (not part of upstream's branch, so not in its own file
list for this commit, but needing the same rename for consistency).
Validated by regenerating g g > t t~, confirming no leftover
madgraph::/namespace madgraph reference remains in the generated
output, and building/running both the scalar and simd_128 (FPTYPE=v)
backends end to end with unchanged matrix-element output.
Re-derived directly against the current madmatrix.mk/output.py: moved
the 9 rambo/random-number template files into src/rambo/, added the
matching vpath and -I$(SRC)/rambo to madmatrix.mk, updated
madmatrix_standalone.mk's comment, and switched output.py to copy them
once into src/rambo/ via a new from_template['src/rambo'] entry instead
of symlinking them into every P* directory via _standalone_extra_files.
The historical upstream commit introduced a stray unindented comment
continuation line in output.py at this point ("symlinked into every
P1_*/." left over from an edit, with no leading #) that made the file
fail to even parse; the very next upstream commit ("Comment bugfix")
deletes exactly that line. Wrote the comment correctly here instead of
reproducing the transient breakage, since committing a
syntactically-invalid Python module would leave this branch's own
history briefly unbuildable, unlike upstream's linear history where
both commits land together before anyone builds off it; the next
commit in this re-derivation is expected to be a no-op given this.
Validated by regenerating g g > t t~, confirming src/rambo/ is
populated with all 9 files and no longer symlinked per-process, and
building/running both the scalar and simd_128 backends end to end with
unchanged matrix-element output.
when making ccpnone -> cppsse4 -> cppnone rule for check_sa checked that cppnone buildir content is older than the check_sa and did not relink, silently reusing the sse4 lib. now small prerequisity storing last compiled backend solution Re-derived directly against the current madmatrix_standalone.mk: added the CHECK_SA_BACKEND_MARKER file (rewritten only when BACKEND actually changes, via an always-evaluated FORCE-driven rule) as a prerequisite of check_sa.exe, and clean_standalone now also removes it. Validated the actual bug by building scalar -> simd_128 -> scalar with USEBUILDDIR=1 (no clean between switches, reproducing the described ccpnone/cppsse4 alternation): check_sa.exe correctly relinked at every switch (confirmed via the linker command appearing each time and the marker file tracking the last-built backend), and check_sa.exe's matrix-element output matched the scalar baseline after switching back from simd_128. Also confirmed a normal single-backend clean build still works and clean_standalone removes the new marker file alongside check_sa.exe.
Re-derived against the current backend/simd/SigmaKin.cc and check_sa.cc: both of this commit's real fixes (the static_cast<fptype> around broken_symmetry_factor()/helcolDenominators[0] in the MEs_sv broken-symmetry normalization, and using massesD instead of the truncated-to-fptype masses vector in classic_rambo::get_momenta(), which always needs double precision regardless of FPTYPE) were already present in this port from earlier validation work in this same session, so only the explicit #include "mgOnGpuConfig.h" (redundant here since SigmaKin.h already pulls it in transitively, but added for parity with upstream) was still needed. Validated by regenerating g g > t t~ and building/running both the scalar and simd_128 backends under FPTYPE=f specifically (the precision mode this commit's fixes target, not otherwise build-tested with a real check_sa.exe run earlier in this session), getting identical, correct matrix-element output between the two backends, plus confirming FPTYPE=d is unaffected and the full standalone-syntax sweep stays clean.
… initProc CPPProcess::initProc's generated cipfassign block (tIPF_partner1/2/value, built from FLV_COUPLING via nMF*nIPF) references ProcessTables::nMF unqualified, but process_h.inc never included ProcessTables.h and process_function_definitions.inc never had `using namespace ProcessTables;` - only SigmaKin.cc had picked this up (fixed earlier in this port's history), leaving CPPProcess.cc broken for any process with at least one merged-flavor coupling (nIPF > 0). Never caught earlier in this session because g g > t t~ has nIPF == 0, so cipfassign is empty and this code path is dead there. Found via a user-reported build failure for u u > h u u g g (BACKEND=auto -> simd_128): 9x "use of undeclared identifier 'nMF'" in CPPProcess.cc. Fixed by adding #include "ProcessTables.h" to process_h.inc (alongside the existing ProcessData.h) and `using namespace ProcessTables;` to process_function_definitions.inc (alongside the existing `using namespace ProcessData;`). Validated by regenerating u u > h u u g g and building/running both scalar and simd_128 end to end: identical, non-garbage MeanMatrixElemValue between backends. Also re-validated g g > t t~ (nIPF == 0, exercises the unaffected code path) to confirm no regression, plus the standard standalone-syntax sweep for both backends.
… setup The backend split moved `constexpr int nMF` out of process_function_definitions.inc into madmatrix::ProcessTables, which CPPProcess.cc did not include, while the generated initProc still sizes tIPF_partner1/partner2/value as nMF * nIPF. Every process with merged flavor couplings (nIPF > 0, e.g. MSSM_SLHA2 u u~ > n1 n1) failed with "use of undeclared identifier 'nMF'". Include ProcessTables.h and pull nMF in next to `using namespace ProcessData`, where nIPF already comes from, so the generated text is unchanged. u u~ > n1 n1 now builds on scalar and simd_128 and reproduces the pre-split MeanMatrixElemValue (0.000570885 +- 6.9555e-06). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The backend split hard-coded the color choice in calculate_jamps as `jamp2_sv[ncolor * iParity + icol] += cxabs2( jamp_sv[icol] )` for icol < ncolor. Before, it went through the per-process jampflow_lines / jamp_flow_col holes of process_matrix.inc, which set_color_flow_lines_cpp still computes but which nothing emitted any more. For a process whose color sum runs on the (n-2)! DDM basis the two differ: g g > g g g has ncolor = 6 but picks among ncolor_flow = 24 trace flows, rebuilt from the DDM jamps through the Kleiss-Kuijf relations. The split filled 6 slots with the wrong amplitudes and left 18 at zero, so the selected color (the LHE color flow) was wrong while the matrix element stayed bit-identical -- which is why an ME-only comparison could not see it. Same 1024 events, simd_128: 24 distinct flows selected on the merge base, 12 on the split, one of them 568 times. Write those lines to a generated ColorFlows.inc, #included by backend/<variant>/SigmaKin.cc right after EvaluateDiagrams.inc in the same scope, and have it expose jampflow_sv; the color choice loops over ncolor_flow again (cpu, simd and gpu). Checked through UMAMI on identical momenta and random numbers: g g > g g g (scalar, simd_128 mixed and FPTYPE=d), g g > t t~ and e+ e- > mu+ mu- reproduce the merge base's per-event color and helicity sequences and matrix elements exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… path Four regressions against the merge base in backend/gpu, all restored to the merge base's own lines: * DeviceAccessJamp2 lost kernelAccessIcolConst, which the color choice in sigmaKin still calls (compile error). * MatrixElementKernels.cc lost #include "CPPProcess.h", while the device kernel still sizes its color-choice buffer from CPPProcess::ncolor_flow (compile error). * calculate_jamps wrote its jamps with nhel = 1. allJamps is the [2][ncolor][nGoodHel][nevt] super-buffer, where nhel is the color stride, and color_sum_kernel reads it back with nGoodHel: for ncolor > 1 and more than one good helicity writer and reader address different slots, and helicities overwrite each other. Write with dcNGoodHel again (it is 1 during the good-helicity scan, nGoodHel in sigmaKin). * The color-choice accumulation into colAllJamp2s lost its atomicAdd. That buffer has no helicity dimension and the good helicities run concurrently (one stream each, or blockIdx.y in the async path), so a plain += races. No CUDA/HIP toolchain was available: the device semantics (the last two items) are untested here. What was checked is a host -fsyntax-only of every gpu translation unit against minimal CUDA/cuBLAS/thrust stubs, run differentially: the merge base's GPU code is clean under the same stubs, the split had the two compile errors above, and with this commit it is clean again for g g > t t~ and g g > g g g. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r sum
The split made any process whose '^2' constraint leaves more than one
amplitude order abort at generation, on the reasoning that the shared
backend/{cpu,simd}/color_sum.cc cannot carry a process-specific variant.
But those backend sources are compiled once per P* directory, against
that directory's generated ProcessData.h / ColorData.h, so a compile-time
switch is enough -- the same pattern as ColorMatrixData::shouldUseBlas.
* ProcessData.h always defines nampso, njampso = ncolor * nampso and
nsqampso (1, ncolor, 1 without split orders); CPPProcess's
split_order_constants now alias them.
* ColorData.h carries the sqSoIndex / chosenSqso pairing tables (the
trivial 1x1 ones when unsplit).
* backend/{cpu,simd}/color_sum.cc gain color_sum_cpu_splitorders, the
pair loop of the former color_sum_splitorders.cc template (ALL ordered
pairs, never a triangle), selected by `if constexpr( nampso > 1 )`, so
the unsplit path's source is untouched. That template is removed.
* The calculate_jamps / sigmaKin jamp buffers are sized njampso again,
where the merge base used the jamp_ncolor hole; the BLAS buffers keep
ncolor behind a static_assert (cpp_blas_wanted is already false when
split).
* backend/gpu/SigmaKin.cc static_asserts nampso == 1, the replacement
for the template's #error under MGONGPUCPP_GPUIMPL.
Validation against the merge base on identical momenta and random
numbers (1024 events through UMAMI): unsplit processes (g g > t t~,
g g > g g g, e+ e- > mu+ mu-, MSSM u u~ > n1 n1, u u~ > u u~ QED^2==0)
stay bit-identical; with FPTYPE=d, u u~ > u u~ QED^2<=4 and
u u~ > d d~ g QED^2==2 are bit-identical and u u~ > u u~ QED^2==2 agrees to
9e-16, same per-event color and helicity choices, on simd_128 and
scalar. At the RAMBO 1000 GeV point the interference is
-5.5828746494657286e-02 against the recorded -5.5828746494657258e-02.
test_standalone_split_orders_interference passes again. The host (clang) stub
syntax check of the GPU build refuses a split process with the new
static_assert message and stays clean for unsplit ones.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TREX_ProcessExporter.generate_subprocess_directory called
self._link_backend_dirs_in_P(matrix_element), which is defined nowhere,
so `output standalone_trex` raised AttributeError on the first
subprocess. TREX deliberately stays on the old layout ("for TREX for now
no /backend"), and one TREX P* directory builds both the C++ and the
CUDA library, so there is no single backend variant to link anyway.
Note this is not the first thing that stops TREX: on main already,
edit_rwgt_header looks up PLUGIN_export_cpp on model_handling (trex.py
defines it in its own namespace), and librex.so does not compile with
Apple clang (Rex.h:1609 builds std::bad_any_cast from a string). Both
are pre-existing and untouched here. Checked by driving
`output standalone_trex` for g g > t t~ through a PYTHONPATH plugin copy
with the first of those patched out locally: before this commit it
stops with this AttributeError, after it the output is written without
a traceback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…_include
CPPProcess.h, Parameters.h and HelAmps_<model>.h decided "am I the simd
backend?" with #if __has_include("mgOnGpuVectors.h"). That asks whether
some file of that name is reachable on the include path, and -I$(SRC) is
on it: a stray src/mgOnGpuVectors.h -- TREX copies one there, and an
older output can leave one behind -- silently moves a scalar or GPU
build onto the SIMD path. Reproduced by dropping the simd header into
src/ of a g g > t t~ output: BACKEND=scalar then fails with
"redefinition of 'neppV'".
backend/simd/mgOnGpuConfig.h, and the legacy single-layout
mgOnGpuConfig.h TREX still uses (which always ships the header), now
define MGONGPU_HAS_VECTORS_H, and the three shared headers test that.
The stray-header case builds and matches again; g g > t t~ (scalar and
simd_128) and MSSM u u~ > n1 n1 are bit-identical to the merge base, and
the GPU stub syntax check stays clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ards Two of the mixed-precision guards in backend/simd/color_sum.cc (deltaMEs_next, and the store into the second neppV page) had dropped `defined MGONGPU_CPPSIMD`, while nParityCS in the same file and nParity in SigmaKin.cc keep it. The three have to agree: without the SIMD term color_sum_cpu would write MEs at ievt0 + neppV for a page the caller neither zeroed nor owns. Not reachable today -- mgOnGpuVectors.h does not even compile without MGONGPU_CPPSIMD (`const int neppV = MGONGPU_CPPSIMD`) -- so this only restores the merge base's guards. color_sum_cpu is now textually the same in backend/cpu and backend/simd. simd_128 output is bit-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e dirs
The backend/{cpu,simd,gpu} and src/rambo from_template lists were built
from `sorted(os.listdir(...))`, i.e. whatever happened to be in those
directories when madmatrix.output was imported. Reproduced: a
backend/cpu/.DS_Store (which macOS Finder creates routinely) was copied
into every generated output.
template_sources() keeps only regular, non-hidden .h/.cc files, which is
everything these directories are meant to hold: the copied file sets are
unchanged (30/31/32/9), while .DS_Store and an editor backup umami.cc~
placed in the template dirs no longer reach the output.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sigmaKin and everything it calls moved to backend/<variant>/SigmaKin.cc, but get_sigmaKin_lines still read process_sigmaKin_function.inc and %-substituted it for every subprocess, and the result was thrown away: process_function_definitions.inc no longer has a hole for it. The split nevertheless kept editing that template in step with the three SigmaKin.cc copies, which made it a fourth copy of the same code that no build compiles, and the obvious place for the next fix to land in three copies out of four. get_sigmaKin_lines now returns nothing (it keeps the base class's multichannel guard); process_sigmaKin_function.inc and color_sum_blas_loop.inc, reachable only through it, are removed, with their class attributes. The generated sources are byte-identical for g g > t t~ and MSSM u u~ > n1 n1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_all_sigmaKin_lines no longer calls get_matrix_single_process, and the
per-process color_sum.cc it used to write is gone, which leaves without
any caller:
* OneProcessExporterMadMatrix.get_matrix_single_process, and
jamp_ncolor(), which only it (and the sigmaKin rendering dropped in the
previous commit) used -- the jamp buffers are sized by
ProcessData::njampso now;
* process_matrix.inc (single_process_template), whose color-choice and
jamp-copy lines live in backend/<variant>/SigmaKin.cc and
ColorFlows.inc now;
* color_sum_blas.inc (blas_color_sum_template), whose body was copied
into backend/{cpu,simd}/color_sum.cc and would only have drifted from
it.
Comments that named them are updated. Generated sources are
byte-identical for g g > t t~ and MSSM u u~ > n1 n1, and
test_standalone_split_orders_interference passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ckends The SIMD helpers of MemoryAccessVectors.h exist only in backend/simd. backend/cpu/ and backend/gpu/ each kept a MemoryAccessVectors.h holding nothing but #include "mgOnGpuConfig.h" / "mgOnGpuCxtypes.h" (and, for cpu, an empty namespace), copied into every output only so that five MemoryAccess*.h per backend could include it. Those five now include mgOnGpuCxtypes.h directly -- exactly what the stub provided beyond the mgOnGpuConfig.h they already include -- and the two stubs are removed. Scalar builds are bit-identical (g g > t t~, MSSM u u~ > n1 n1, u u~ > u u~ QED^2==2) and the GPU stub syntax check stays clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ackend/common
Nine files were byte-identical in backend/cpu, backend/simd and
backend/gpu -- constexpr_math.h, CrossSectionKernels.h, EventStatistics.h
and the MemoryAccess{Amplitudes,Denominators,Numerators,RandomNumbers,
Wavefunctions,Weights}.h -- i.e. three copies of ~1300 lines kept in step
by hand. They move to backend/common/, which output.py copies like the
variants and madmatrix.mk / madmatrix_src.mk search after
backend/<variant>/ (and list as prerequisites, so touching one rebuilds).
No file name is in both backend/common and a variant directory, on
purpose: a quoted #include resolves in the including file's own
directory first, so a common header would pick up a common "host"
version of a file even in a build whose variant overrides it. For the
same reason the six files shared only by cpu and simd (CrossSectionKernels.cc,
MatrixElementKernels.{cc,h}, MemoryAccessHelpers.h, MemoryBuffers.h,
mgOnGpuFptypes.h) stay where they are.
Outputs are bit-identical (g g > t t~ scalar and simd_128, g g > g g g,
MSSM u u~ > n1 n1, u u~ > u u~ QED^2==2), the GPU stub syntax check is
clean, and test_standalone_split_orders_interference passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* output.py: the backend/ copy is no longer "step 1, not yet wired into the build"; and the backend sources, while read from the single top-level backend/<variant>/ and backend/common/ dirs, are compiled once per P* directory -- necessarily, since they include that directory's generated ProcessData.h / ColorData.h / ProcessTables.h. The old wording suggested one shared compilation, which is the misreading that made split orders look impossible to support. * model_handling.py: calculate_jamps lives in backend/*/SigmaKin.cc, not a CalculateJamps.cc that never existed, and EvaluateDiagrams.inc is #included by SigmaKin.cc, not written for CPPProcess.cc; a duplicated comment line on edit_colordata is dropped. * process_function_definitions.inc: SigmaKin.cc #includes ColorFlows.inc too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The backend split moved fpeEnable(), the runtime SIGFPE-trap switch with its <cfenv> dependency and one-shot static latch, out of the generated CPPProcess.cc into constexpr_math.h -- a header otherwise about compile-time math, which every translation unit that wants constexpr sqrt/pow includes, and where nobody looking for where FPE traps get enabled would think to look. It now lives in backend/common/fpe.h, included by CPPProcess.cc, its only caller (under MGONGPUCPP_DEBUG); constexpr_math.h loses <cfenv>. MatrixElementKernels.cc's own fetestexcept use already includes <cfenv> itself. CPPProcess.cc compiles with -DMGONGPUCPP_DEBUG on cpu and simd, the default build is bit-identical, and the GPU stub check stays clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving helcolDenominators from process_function_definitions.inc into ProcessData.h dropped its "assume nprocesses == 1 (#272 and #343)" comment, the only in-tree record that the size-1 array is an assumption: den_factors is joined over every matrix element of the P* directory, and ProcessData.h computes nproc > 1 two lines above. Put the caveat back and point at MadGraph7 issue #168, which tracks sizing it properly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The setters the backend split added to backend/{cpu,simd}/SigmaKin.cc,
and backend/gpu/MemoryBuffers.h, call memcpy without including
<cstring>. With clang/libc++ (macOS) it arrives transitively, so every
local build passed; with GCC/libstdc++ it does not, and every madmatrix
build on the Linux CI runners stopped at
backend/cpu/SigmaKin.cc:71:5: error: 'memcpy' was not declared in this scope
That single error is behind 33 of the 34 failing checks on #166 and
#179: every mg7 run (no info.json, events, gridpack, MadSpin, Pythia8,
Rivet, reweight or systematics output), the standalone builds (no
matrix element) and the check command's missing "MG7 SA" column.
Reproduced and checked locally with g++-16: before, the CI error and
unittest_31's failure reproduce; after, both CPU backends build, and
test_check_language_function_epem_aa, test_standalone_cpp,
test_standalone_split_orders_interference and test_e_e_collision_mg7
pass with CXX=g++-16 (the mg7 build demonstrably compiling with it,
40.23(42) pb against the 40.3 pb reference).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
testIO_FDgauge_madmatrix records the generated src/HelAmps_sm.h, which the backend split changed in two intended ways: the single namespace madmatrix replaces the mg5amcGpu/mg5amcCpu pair, and mgOnGpuVectors.h is included only by the simd backend (on MGONGPU_HAS_VECTORS_H, falling back to mgOnGpuCxtypes.h). Those are the only two differences; regenerated with `-U testIO_FDgauge_madmatrix -F 10`, and the test passes. No other golden under IOTestsComparison records madmatrix output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
In the FD gauge HelAmps_<model>.h carries define_gauge_dir and calculate_propagator_factor, pasted from helas_fd.h. Unlike the helas routines around them they are plain functions, not templates, and were marked INLINE -- which is empty unless HELINL=1 -- so with the default build they are ordinary external definitions in a header. That was harmless while only CPPProcess.cc included HelAmps. With the backend split CPPProcess.h includes it and SigmaKin.cc (which now makes the helas calls) and umami.cc include CPPProcess.h, so the process library failed to link: duplicate symbol 'madmatrix::define_gauge_dir(...)' in umami.o, CPPProcess.o, SigmaKin.o which is acceptancetest_uncovered_merged (test_standalone_cpp_fd_output_consistency) on #179, and was hidden behind the memcpy compile error on #166. A non-template function defined in a header must be inline whatever HELINL says, so the two are now `inline` (ALWAYS_INLINE kept on the declarations). The FD-gauge u/d-flavour q q > h q q q q~ QCD=0 process now builds and prints all 54 matrix elements on scalar and simd_128, with HELINL=1, and with g++-16; test_standalone_cpp_fd_output_consistency, test_madmatrix_fd_vs_fortran and test_madmatrix_fd_simd_lanes pass with clang and GCC; the GPU stub check is clean. The FD-gauge madmatrix golden records helas_fd.h and is regenerated for exactly these lines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Olivier was able to fix all the caveats during the weekend in the #179. Shall we merge both? @Qubitol @oliviermattelaer |
Only conflict was with the GPU bounds padding in umami.cc
The base branch took origin/main and four fixes of its own that overlap
commits already here; each conflict keeps one copy of the fix, and takes
the base branch's where both are equivalent:
* nMF for the initProc flavor-coupling setup: the base's
`using namespace ProcessTables` plus ProcessTables.h in CPPProcess.h;
this branch's `using ProcessTables::nMF` and the now redundant include
in process_cc.inc are dropped.
* <cstring> in backend/{cpu,simd}/SigmaKin.cc: the base's line (the one
in backend/gpu/MemoryBuffers.h, only here, stays).
* FD gauge inline helpers: helas_fd.h is the base's version; the FD-gauge
madmatrix golden is regenerated for it.
* GPU: the base's relocated DeviceAccessJamp2 (with kernelAccessIcolConst)
and its CPPProcess.h include in MatrixElementKernels.cc; the duplicate
class and include from here are dropped, while this branch's
split-order static_assert, dcNGoodHel stride, atomicAdd and
ColorFlows.inc color choice, which the base does not have, are kept.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review fixes for #166 (backend separation)
|
Full 5-backend (
Those digits are probably anyway numerical error induced and only discovered by different order of rounding. As CI passes, after reviews can be merged @Qubitol @oliviermattelaer |
oliviermattelaer
left a comment
There was a problem hiding this comment.
let's merge !!!
Asked Claude to redo the commits in my previous backend separation PR #76 but from the current
main, so with the current status of MadMatrix. For details refer to the previous PR.Summary
Physically separates backend-owned driver code (cpu / simd / gpu) out of the madmatrix generator's shared and per-process-generated files into
backend/{cpu,simd,gpu}/, so a given backend's code lives in its own directory instead of being selected at compile time via#ifdef MGONGPUCPP_GPUIMPL/MGONGPU_CPPSIMD.What changed
SigmaKin,color_sum,CrossSectionKernels,MatrixElementKernels,EventStatistics,umami, the 14MemoryAccess*.hfiles, andmgOnGpuConfig/Cxtypes/Fptypes/Vectors.hnow live underbackend/{cpu,simd,gpu}/, sourced by the Makefile via-I/vpathinstead of being copied per process.ColorData.h(merges colour channel information + the color matrix),ProcessData.h(merges contexpr process data e.g.nwf, extended with the helicity/flavor tables that used to be inline inCPPProcess.cc),ProcessTables.h(flavor-coupling / broken-symmetry tables) — single source of truth read by bothCPPProcessand backend-owned code.CPPProcess.cc: all funcs related toSigmaKincomputation moved to/backend/SigmakinfpeEnable()/getCompiler()moved to backend files; per-parameter imperative literal-assignment code replaced by a generated pointer-to-member table plus one genericgatherFptype/gatherCxtypeloop written once inParameters.h.Bridge.h,BridgeKernels.*,fbridge.*,fsampler.cc(zero remaining references);GpuAbstraction.h/GpuRuntime.hmade gpu-only.mg5amcCpu/mg5amcGpu→ onemadmatrixnamespace (the split is now physical, not namespace-based)#ifdefat the point of use:createNormalizedColorMatrix(): no-op inline on cpu/simd, realgpuMemcpyToSymbolon gpu.ColorData.h's device→host copy ofchannel2iconfigremoved entirely (its only consumer,runTest.cc, isn't wired into any export path — kept a local#ifndef/#elsethere since it can't get per-process%-substituted literals the wayColorData.hcan for now).backend/{cpu,gpu}/mgOnGpuVectors.hentirely — neither backend has real C++ vector types; their scalar-only content (neppV,fpternary/cxternary/maskand, the_svtypedefs,cxabs2) merged intomgOnGpuCxtypes.h. The handful of shared, non-per-backend.incfiles that need this now pick the right header via__has_include. Stripped the remaining always-true#ifdef MGONGPU_CPPSIMDguards (and the dead code blocks they wrapped) frombackend/simd/mgOnGpuVectors.h.output standalone_mg7onlyUSEBUILDDIRdefaults to 1 for cleaner buildsramboused in standalone mode for both massive and massless version as well as the random number kernels live in./src/ramboand is sourced whenP1_*is compiledNet effect
MGONGPUCPP_GPUIMPLnow survives only for poping diagnostics while carryinghip/cudadiversification and in the standalone specific files (rambo/check_sa)MGONGPU_CPPSIMDnow survives only in the /backend/simd as carry for the sims lane size and in theHelAmps_sm.hpolarisation functions for masking paths of conditional codeTest