Swarm.migrate: shrink the working set instead of reclassifying every round

[lmoresi]

Addresses the dominant term in #551.

What was costing the time

Swarm.migrate's round loop re-offered the whole local array to
points_in_domain on every round, so the classification work per rank grew with
the round count. Measured through global_evaluate on a fixed 47k-point global
set (2-D simplex, cellSize=1/100, identical total work at every rank count):

np	global_evaluate	points_in_domain	points offered	the 50-cell walk
1	0.208 s	0.058 s (28%)	46 901	0.061 s (29%)
2	0.326 s	0.159 s (49%)	141 516	0.108 s (33%)
4	0.443 s	0.259 s (58%)	237 826	0.162 s (37%)

The walk holds a roughly constant share. What grows is the classification, from
28% to 58% of the call, because a fixed 47k-point set is offered to it 238k
times at np=4.

The change

A point this rank has found to be in its domain stays in its domain — neither
the coordinates nor the mesh change during migration — so it does not need
retesting. The loop remembers what it has claimed and classifies only the rest.

Keyed by coordinate, not by index. dm.migrate does not preserve the local
ordering: measured, retained points are not left at the front, so an index from
the previous round names a different particle after the move. Two particles at
the same coordinate share the answer, so a key collision is harmless. Exact
bit-pattern comparison through a void row view — 15 ms against
points_in_domain's 58 ms at 47k, so it pays whenever more than about a quarter
of the local array is already claimed, which it is from the second round on.

Measured after

np	global_evaluate	points offered to points_in_domain
1	0.208 -> 0.213 s	46 901 -> 46 901
2	0.326 -> 0.232 s	141 516 -> 47 558
4	0.443 -> 0.257 s	237 826 -> 50 025

42% off at np=4, and the growth with rank count is gone — the classification now
sees roughly the local set once whatever np is. np=1 is unchanged within noise
(one extra membership pass, no rounds to save). The walk shrinks with it, since
it is called from inside the classification.

This is the larger of the two faults named in #551's "amplifier" section. The
other — no cheap rejection inside the walk — attacks the smaller, flatter term
and is untouched here.

The collective

The points_in_domain call is unconditional, and that is load-bearing.
It is collective — it reaches get_max_radius() before any short-circuit
precisely so a rank with nothing to classify still joins the reduction (the #405
treatment, stated in its own source). Guarding it on whether this rank has
undecided points deadlocks at np=4, with every rank inside the call, as soon as
one rank runs out of them.

Verified

• Behaviour-preserving by measurement, not by inference. The per-rank
  partition fingerprint after migration — the sorted coordinates each rank owns,
  hashed — is byte-identical to development at np=2 and np=4.
• Full ./uw test: 1556 passed, 32 skipped, 2 xfailed, matching development.
• Parallel swarm/migration/evaluate tests at np=2: 29 passed, 6 skipped.
• test_0776_migrate_working_set_mpi.py covers the loop: the global particle
  count is conserved, every point lands on a rank whose mesh contains it, and the
  global multiset of coordinates is unchanged. It carries a premise test that the
  fixture actually migrates anything — populate alone leaves every particle
  already owned, and the loop never runs.
• Its own negative control: reintroducing the conditional collective makes
  test_0776 time out at np=4 (rc=241); restoring the fix makes it pass. The
  test catches the defect it was written for.

Underworld development team with AI support from Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lmoresi]

Adversarial review

Reviewed at 7811686. Four findings.

1. The claimed set grows without bound within a call, and is rebuilt on every
call. claimed_keys is a NumPy array concatenated once per round, so a swarm
of N local points costs O(N) memory per migrate and O(rounds x N) copying to
build it. At the sizes measured here that is invisible against the classification
it removes, but migrate is called every timestep in an advecting model and the
concatenation is not amortised across calls. A pre-allocated buffer, or keeping
the set between calls keyed on a population generation, would remove it; neither
is done here because neither is needed at the sizes we measured, which is a
statement about our benchmark rather than about a production run.

2. np.isin on a void view is O(n log n) by sort, and it is now on the hot
path. Measured 15 ms against 58 ms for the classification at 47k, so it pays —
but the margin is a factor of four, not an order of magnitude, and it narrows as
the already-claimed fraction falls. A run where almost every point moves every
round (a violently advecting swarm, or a first migrate after a large mesh
deformation) pays the membership cost for very little saving. It never costs
more than the classification it replaces, so the worst case is a wash rather
than a regression, but the win is workload-dependent in a way the numbers above
do not show.

3. The equivalence evidence covers np=2 and np=4 on one mesh. The partition
fingerprint is byte-identical, which is the right instrument, and it was taken on
a single 2-D simplex box with one particle distribution. It says the change is
behaviour-preserving there; it does not cover 3-D, deformed or adapted meshes,
split-node fault swarms with coincident coordinates, or np=8. The coincident-
coordinate case is the one worth naming: two particles at the identical position
on opposite faces of a fault produce the same key, and the argument that this is
harmless — they share the containment answer — is sound but untested.

4. The premise test constrains the fixture, not the property.
test_premise_the_fixture_actually_migrates asserts some point crosses ranks,
which is what makes the other two tests meaningful. It does not assert that the
round loop runs more than once, and the saving only exists from the second round
onward. A fixture that migrated everything in a single round would pass all three
tests while exercising none of the bookkeeping this PR adds.

Checked and clean. The points_in_domain call is unconditional, and the
negative control confirms it must be: reintroducing the conditional deadlocks at
np=4 (rc=241) and the fix clears it. Every rank reaches the reduction on every
round regardless of how much work it has.

Underworld development team with AI support from Claude Code

[lmoresi]

The np=4 parallel-test hang is pre-existing, not from this PR

Reporting this because an earlier run made it look otherwise.

Running tests/parallel/ -k "swarm or migrat or evaluate" at np=4 hangs on this
branch. It hangs identically on development at 8b7c8b9, so it is not caused by
the change here. Bisected to
test_0760_swarm_cache_migration.py::test_global_evaluate_after_migration, which
passes at np=2 and hangs at np=4 on the trunk; filed as #611, along with the
reason nobody has seen it — that file matches neither of the two globs
scripts/test.sh runs under mpirun.

One earlier np=4 result on this branch should be discarded rather than trusted:
it was running in the background while the negative control rebuilt the
environment underneath it (broken build, then restored), so it was executing
against site-packages being swapped mid-run. That is why it is not quoted in the
description.

What stands for this PR: full ./uw test 1556 passed matching development; the
np=2 parallel selection 29 passed, 6 skipped; byte-identical partition
fingerprints at np=2 and np=4; and test_0776 passing at np=2 and np=4 with a
working negative control.