do not pickle cached hash of arrays

[matthiasdiener]

Prevents errors of the type:

pytato.distributed.verify.MissingSendError: no send for 'CommunicationOpIdentifier(src_rank=1, dest_rank=0, comm_tag=('fluid', (<class 'mirgecom.diffusion._DiffusionStateTag'>, (<class 'y3prediction.prediction._SmoothCharDiffFluidCommTag'>, Placeholder(shape=(), dtype='int64', name='_actx_in_1'))), (0,)))'

(see e.g. https://github.com/illinois-ceesd/mirgecom/actions/runs/11958222838/job/33337251412?pr=926)

cc @MTCam

Followup of #521.

Please squash

[inducer]

Completely agree that this needs to be fixed. Just a few nits to pick below.

[matthiasdiener]

A somewhat unorthodox thought - should we even cache the hash of dataclasses, or could we rely on that any "large" field of the dataclass (say, an immutabledict) already caches their own hash?

What dataclasses does to generate a hash is to create a tuple of field values and hash that tuple:
https://github.com/python/cpython/blob/a8536102828ac70000fc6b4c67e0b439ab0ddf98/Lib/dataclasses.py#L886-L892

In case caching the hash is not needed, neither the custom hashing nor the caching of the hash would be necessary (but we would have to set unsafe_hash=True to enable hashing when frozen=False for optimized executions).

This would also affect #566.

[matthiasdiener]

A somewhat unorthodox thought - should we even cache the hash of dataclasses, or could we rely on that any "large" field of the dataclass (say, an immutabledict) already caches their own hash?

Nevermind, for recursive data structures like the DAG we will need cached hashes.

[matthiasdiener]

A gentle ping for a review @inducer

[inducer]

Thanks, LGTM!