Boundary lazy comm v2

[matthiasdiener]

Followup of #89.

Needs:

• Graph partitioner pytato#139
• Distributed v3 pytato#148
• Fix array container support in nodal_sum #219

[matthiasdiener]

As far as I can see, this does successfully do the code generation for the partitioned code now, but fails in the "normal" pytato code generation now since DistributedSendRefHolder etc. is still part of the DAG, which the CodegenMapper can not handle. I'm not sure what the right approach here is @inducer - should the partitioning remain part of transform_dag(), or should it go somewhere else?

[inducer]

The RefHolder should have been removed by partitioning. I'm not sure why it's still around.

[matthiasdiener]

The RefHolder should have been removed by partitioning. I'm not sure why it's still around.

Sorry, I wasn't very clear.
The RefHolder is removed correctly from what I can tell. However, the partitioned code isn't saved anywhere at the moment, so the subsequent execution still uses the unpartitioned code and fails. I was wondering whether transform_dag() is the right place to do the partitioning, and where the new execution behavior should be implemented (in LazilyCompilingFunctionCaller::__call__ ?)

[inducer]

Sure, the execution logic needs to change, and I agree that transform_dag is not the correct place for the partitioning. We need an array context subclass that knows about distributed execution, and that needs to worry about distributed execution in the appropriate places. (freeze and precompiled function execution as you mentioned)

[matthiasdiener]

I think this is ready for another review @inducer.

[inducer]

FYI I introduced an MPIPyOpenCLArrayContext. This was needed to avoid just ignoring symbolic tags in the eager case, which didn't sit well with me. This in turn shifted the "source of truth" for the MPI communicator from the discretization to the array context, which is just as well IMO.