Pytato+distributed: verify partition if in debug mode

[inducer]

Draft because:

• Needs Check that distributed DAG is deadlock-free pytato#366

cc @matthiasdiener

[kaushikcfd]

Why don't we check for deadlocks whenever the user calls execute_distribute_parttion for the first time?

[inducer]

Why don't we check for deadlocks whenever the user calls execute_distribute_parttion for the first time?

How do we tell that it's the first time? By setting a flag on the partition?

I'm also a bit wary of doing this implicitly at the pytato level, since it's fairly expensive.

[kaushikcfd]

I was thinking something on the lines of:

@memoize_on_first_arg
def _verify_no_cycles(parts, comm):
    ...

def execute_distribute_partiions(parts, comm):
    if __debug__:
        _verify_no_cyles(parts, comm)

    ...

But the problem here is comm isn't hashable. Not sure if that's avoidable.

[inducer]

That's no different than setting a flag! (just with a more awkward interface)

[kaushikcfd]

That's no different than setting a flag! (just with a more awkward interface)

Ah fair.

[kaushikcfd]

Is this guaranteed to terminate cleanly or is there a chance we might need a self.actx.mpi_communicator.abort() if a failure was seen?

[inducer]

IMO, this might better be addressed by using python -m mpi4py in the MPI command line for these runs. This will automatically abort/kill other ranks if one goes down with an exception.

See, e.g. #295, inducer/pytato#389

[kaushikcfd]

Thanks, yep it does get resolved if we do python -m mpi4py .... If that's the only way one is supposed to run grudge apps, feel free to close this.

[inducer]

It's not the only way, but it's a good way to avoid hangs. Realistically, it's something every user of mpi4py-dependent code should know. Mirgecom does some hand-holding along these lines:

https://github.com/illinois-ceesd/mirgecom/blob/0d8635eb16717d62d83a0e62ddcdb0fb7866c4d7/mirgecom/mpi.py#L109-L159

But I find that too heavy-handed for pytato or grudge.