Skip to content

ci: Backport earthflowAI solution to unbind ATS launches and run CTest in parallel - #4118

Merged
rrsettgast merged 3 commits into
developfrom
feature/fixIntegratedTests
Aug 15, 2026
Merged

rrsettgast merged 3 commits into
developfrom
feature/fixIntegratedTests

Conversation

@rrsettgast

@rrsettgast rrsettgast commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

GEOS ATS starts each integrated test through an independent Open MPI
launcher. For serial tests, each launcher applied its default binding
policy independently and selected the first available core. As a result,
many concurrent one-rank tests could all be pinned to the same core on
the streak2-32core runner despite ATS accounting for 32 available slots.

Verify that the integrated-test launcher is Open MPI and disable its
hwloc binding policy before invoking geos_ats.sh. This leaves each rank
unbound within the container's available CPU set so the Linux scheduler
can use all available cores. Enable Open MPI binding reports so the
effective placement remains visible in retained ATS logs.

Also:

  • validate that NPROC is a positive integer;
  • use NPROC for the ATS per-node and maximum process capacities;
  • authorize root execution through the Open MPI environment variables;
  • set OMP_NUM_THREADS=1 to prevent each parallel test from creating an
    additional OpenMP thread pool;
  • run CI unit tests with CTest parallel scheduling.

Add GEOS_CTEST_PARALLEL_LEVEL to control parallel CTest execution:

  • AUTO, the default, lets CTest determine the parallel worker count;
  • OFF disables parallel scheduling;
  • a nonnegative integer passes an explicit worker count to CTest.

Preserve an explicitly supplied CMAKE_CTEST_ARGUMENTS value and use the
resulting arguments for both the generated build_test target and CI test
execution.

Set the CTest PROCESSORS property from NUM_MPI_TASKS so parallel CTest
accounts for the full rank count of MPI tests instead of treating every
test as a single scheduling slot.

Parallel execution exposed tests that created and removed identically
named PVT table files from shared working directories. Give the affected
constitutive and wells test executables unique scratch filenames, and
give the BlackOil fixture a unique temporary directory. Include the
newer thermal-estimator well tests, which have the same shared-file
behavior but were not present in the original EarthFlowAI change.

This removes Open MPI's forced first-core binding for independent ATS
launchers. It intentionally does not provide exclusive core allocation;
the operating system remains responsible for scheduling unbound ranks
within the runner's available CPU set.

stats:
on rocky gcc13 (cpu), unit tests went from 397 s to 71 s
integratedTests go from 43m missing some tests, to 18m with all tests re-enabled.

@rrsettgast rrsettgast self-assigned this Aug 14, 2026
@rrsettgast rrsettgast added the ci: run integrated tests Allows to run the integrated tests in GEOS CI label Aug 14, 2026
@rrsettgast rrsettgast changed the title ci: unbind ATS launches and run CTest in parallel ci: Backport earthflowAI solution to unbind ATS launches and run CTest in parallel Aug 14, 2026
@victorapm victorapm linked an issue Aug 15, 2026 that may be closed by this pull request
@rrsettgast

Copy link
Copy Markdown
Contributor Author

@joshua-white I think this is ready to merge unless there is something else to add back in from your streak2 work.

@victorapm
victorapm self-requested a review August 15, 2026 15:37

@victorapm victorapm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the improvements!

@rrsettgast
rrsettgast merged commit ea6d769 into develop Aug 15, 2026
21 checks passed
@rrsettgast
rrsettgast deleted the feature/fixIntegratedTests branch August 15, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: run integrated tests Allows to run the integrated tests in GEOS CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Long application/integrated tests.

3 participants