Address CI failures

[kaushikcfd]

• Looks like CI is failing on main.
• Opened this PR to diagnose/fix it.

[kaushikcfd]

The arraycontext failure should be addressed by inducer/arraycontext#206. Will spend some time on the failure in test_distributed.

[kaushikcfd]

The thing I'm struggling with over here is that in Py3.11 the results are non-deterministic for different interpreter runs with the same PYTHONHASHSEED. I don't understand how that can even happen.

[inducer]

I don't know if this is at all related, but if I run test_distributed on Python 3.11 on my system, I get some ominous warnings about attrs:

$ python -m pytest test_distributed.py --sw --tb=native
===================================================================================================================================================================================================================================== test session starts ======================================================================================================================================================================================================================================
platform linux -- Python 3.11.0, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/andreas/src/pytato
plugins: pudb-0.7.0, anyio-3.6.1, hypothesis-6.47.2
collected 4 items                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
stepwise: no previously failed tests, not skipping.

test_distributed.py F

=========================================================================================================================================================================================================================================== FAILURES ===========================================================================================================================================================================================================================================
_______________________________________________________________________________________________________________________________________________________________________________________________________________________________ test_distributed_execution_basic _______________________________________________________________________________________________________________________________________________________________________________________________________________________________
Traceback (most recent call last):
  File "/home/andreas/src/pytato/test/test_distributed.py", line 73, in test_distributed_execution_basic
    run_test_with_mpi(2, _do_test_distributed_execution_basic)
  File "/home/andreas/src/pytato/test/test_distributed.py", line 52, in run_test_with_mpi
    check_call([
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mpiexec', '-np', '2', '-x', 'RUN_WITHIN_MPI=1', '--oversubscribe', '-x', 'INVOCATION_INFO=gASVQAAAAAAAAACMEHRlc3RfZGlzdHJpYnV0ZWSUjCRfZG9fdGVzdF9kaXN0cmlidXRlZF9leGVjdXRpb25fYmFzaWOUk5QphpQu', '/home/andreas/src/env-3.11/bin/python', '/home/andreas/src/pytato/test/test_distributed.py']' returned non-zero exit status 1.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stderr call -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/andreas/src/env-3.11/lib/python3.11/site-packages/attr/_make.py:918: RuntimeWarning: Running interpreter doesn't sufficiently support code object introspection.  Some features like bare super() or accessing __class__ will not work with slotted classes.
  set_closure_cell(cell, cls)
/home/andreas/src/env-3.11/lib/python3.11/site-packages/attr/_make.py:918: RuntimeWarning: Running interpreter doesn't sufficiently support code object introspection.  Some features like bare super() or accessing __class__ will not work with slotted classes.
  set_closure_cell(cell, cls)
/home/andreas/src/env-3.11/lib/python3.11/site-packages/attr/_make.py:918: RuntimeWarning: Running interpreter doesn't sufficiently support code object introspection.  Some features like bare super() or accessing __class__ will not work with slotted classes.
  set_closure_cell(cell, cls)
/home/andreas/src/env-3.11/lib/python3.11/site-packages/attr/_make.py:918: RuntimeWarning: Running interpreter doesn't sufficiently support code object introspection.  Some features like bare super() or accessing __class__ will not work with slotted classes.
  set_closure_cell(cell, cls)
Traceback (most recent call last):
  File "/home/andreas/src/pytato/test/test_distributed.py", line 317, in <module>
Traceback (most recent call last):
  File "/home/andreas/src/pytato/test/test_distributed.py", line 317, in <module>
    run_test_with_mpi_inner()
  File "/home/andreas/src/pytato/test/test_distributed.py", line 65, in run_test_with_mpi_inner
    f(cl.create_some_context, *args)
  File "/home/andreas/src/pytato/test/test_distributed.py", line 88, in _do_test_distributed_execution_basic
    run_test_with_mpi_inner()
  File "/home/andreas/src/pytato/test/test_distributed.py", line 65, in run_test_with_mpi_inner
    f(cl.create_some_context, *args)
  File "/home/andreas/src/pytato/test/test_distributed.py", line 88, in _do_test_distributed_execution_basic
    halo = staple_distributed_send(x, dest_rank=(rank-1) % size, comm_tag=42,
    halo = staple_distributed_send(x, dest_rank=(rank-1) % size, comm_tag=42,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/andreas/src/pytato/pytato/distributed.py", line 303, in staple_distributed_send
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/andreas/src/pytato/pytato/distributed.py", line 303, in staple_distributed_send
    return DistributedSendRefHolder(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/andreas/src/pytato/pytato/distributed.py", line 225, in __init__
    super().__init__(axes=passthrough_data.axes, tags=tags)
    return DistributedSendRefHolder(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/andreas/src/pytato/pytato/distributed.py", line 225, in __init__
    super().__init__(axes=passthrough_data.axes, tags=tags)
    ^^^^^^^
TypeError: super(type, obj): obj must be an instance or subtype of type
    ^^^^^^^
TypeError: super(type, obj): obj must be an instance or subtype of type
--------------------------------------------------------------------------

[kaushikcfd]

I cannot reproduce the attrs issue locally. Looks like the CI got terminated likely because of a deadlock in test_distributed, but AFAICT it also doesn't report the error you observe. I'm on Python=3.11.0 and attrs=22.1.0.

[matthiasdiener]

fwiw, 3.11 runs perfectly fine on my M1:

$ pytest test/test_distributed.py
====================================== test session starts =======================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/mdiener/Work/emirge/pytato
collected 4 items

test/test_distributed.py ....                                                              [100%]

======================================= 4 passed in 6.75s ========================================

[kaushikcfd]

Found one of the issues that led to this. The random DAG generator emitted comm nodes with the same comm tags.

[kaushikcfd]

Here's a minimal reproducer with 7 nodes that contains the non-determinism issue --

# file: foo.py
import pytato as pt


size = 4
rank = 2
comm_tag = 42

x = pt.make_placeholder("x", 10, "float64")
send1 = pt.staple_distributed_send(
        x, dest_rank=(rank-1) % size, comm_tag=comm_tag,
        stapled_to=pt.make_distributed_recv(
            src_rank=(rank+1) % size, comm_tag=comm_tag,
            shape=x.shape, dtype=x.dtype))

comm_tag += 1

send2 = pt.staple_distributed_send(
        send1, dest_rank=(rank-1) % size, comm_tag=comm_tag,
        stapled_to=pt.make_distributed_recv(
            src_rank=(rank+1) % size, comm_tag=comm_tag,
            shape=send1.shape, dtype=send1.dtype))

comm_tag += 1

send3 = pt.staple_distributed_send(
        send2, dest_rank=(rank-1) % size, comm_tag=comm_tag,
        stapled_to=pt.make_distributed_recv(
            src_rank=(rank+1) % size, comm_tag=comm_tag,
            shape=send2.shape, dtype=send2.dtype))

partition = pt.find_distributed_partition(
    pt.make_dict_of_named_arrays({"out": send3}))
assert set(partition.parts.keys()) == set(range(4))

and running it with the following command:

$ export PYTHONHASHSEED=6
$ for I in {0..40}; do python foo.py; done

I observe a failure rate of 5% (even after fixing PYTHONHASHSEED 🤯🤯🤯)

[kaushikcfd]

After:

[160692c] pyrsistent.pmap -> immutables.Map; adds an assert to ensure the linear scheduling

I no longer see any issues in the regressions/running tests locally. I think we should merge this here.

[kaushikcfd]

If the CI passes, I will hit merge here. Fairly trivial change. There was an error in the regression itself. So please take a look at 8d6f31c once you get a chance.