The mesh cache races: '.meshes' is hardcoded in 42 places, so two concurrent UW3 processes in one directory corrupt each other

[lmoresi]

What happens

Mesh._from_plexh5 fails with petsc4py.PETSc.Error: error code 76 when two processes build the same mesh in the same working directory at the same time: one is still writing .meshes/uw_....msh.h5 while the other opens it for reading.

underworld3/discretisation/discretisation_mesh.py:178: in _from_plexh5
    viewer = PETSc.ViewerHDF5().create(filename, "r", comm=comm)
petsc4py.PETSc.Error: error code 76

This is not only a testing problem. Any two UW3 runs sharing a working directory hit it — a user running two models side by side, or a parameter sweep launched as concurrent single-rank jobs. The cache filename is derived from the mesh parameters, so identical geometry is exactly the case that collides.

Why it cannot currently be worked around

The cache directory is the string literal ".meshes", hardcoded 42 times across 5 modules (cartesian.py, spherical.py, annulus.py, segmented.py, geographic.py), with no environment override and no parameter. There is no way to give a process its own cache.

Note this also violates the Style Charter's DRY clause (§5, "DRY after the 2nd occurrence") 42 times over.

Measured impact on the test suite

Running the level-1 suite under pytest-xdist (-n 4 --dist loadfile, BLAS threads pinned) takes 2:55 against 9:45 serial — a 3.3x speedup — but 6 tests fail and 3 error, every one of them traceable to this race:

• tests/test_1014_stokes_multigrid.py (collection error)
• tests/test_0850_faults.py — three tests
• tests/test_0007_snapshot_inmemory.py, tests/test_0101_kdtree.py, tests/test_0650_recursion_prevention_regression.py
• tests/parallel/test_0765_internal_boundary_integral_mpi.py

So this defect is the single thing standing between us and a 3x faster test suite, locally and in CI.

Proposed fix

1. One helper owning the cache location — _mesh_cache_dir() — reading UW_MESH_CACHE_DIR and defaulting to .meshes, replacing all 42 literals.
2. tests/conftest.py sets it per xdist worker from PYTEST_XDIST_WORKER, so workers cannot collide.
3. Write atomically regardless: generate to a unique temporary name and os.replace() into place, so a reader either sees the old complete file or the new complete file, never a partial one. This is what makes concurrent user processes safe, which (1) and (2) alone do not.

(3) is the real fix; (1) and (2) make it configurable and unblock the test suite.

Underworld development team with AI support from Claude Code

[lmoresi]

Fixed by #565, merged to development (a0c6053). Closing by hand — GitHub only auto-closes on merges to the default branch, and we merge to development.

Both writes (gmsh's .msh and the .msh.h5) now land under a process-unique name and are renamed into place; UW_MESH_CACHE_DIR replaces the hardcoded .meshes; and the MPI barrier the rename makes necessary is explicit. The negative control is the part worth keeping: with every worker forced to SHARE one directory the run is equally green, so the atomicity is what fixes the race — the per-worker directory only stops workers redoing the same gmsh work.

Two generators were also writing their .msh into the working directory rather than .meshes/ (QuarterAnnulus, SegmentofAnnulus); both now use the helper.

Still open and caused by nothing here: #567, which the fix made REACHABLE — xdist previously died on this race in 9 seconds and never got far enough to expose it.