fix(core): avoid panic if workflow stream still up during eviction

[chris-olszewski]

What was changed

When the workflow stream has a zero-size cache (max_cached_workflows = 0) and ignore_evicts_on_shutdown = true (the default for tests), there is a race between stream shutdown and eviction completion processing.

After the final workflow activation completes, the zero-cache mechanism produces an eviction activation. When initiate_shutdown() is called, it cancels the shutdown token and enqueues a BumpStream message into the local channel. If an eviction completion is enqueued after BumpStream, the stream processes BumpStream first, sees shutdown_done() = true, and exits. This drops the eviction completion's response channel sender without fulfilling
it. This causes rx.await to return Err, which triggers dbg_panic! in tests.

For whatever reason, this only seemed to get hit on GitHub Action Ubuntu Arm runners.

Mirror the existing send_local-failure handling for empty completions for the case if the stream shuts down after WFActCompleteMsg was sent.

Why?

I've seen this test fail multiple times over the past week.

Checklist

1. Closes N/A

2. How was this tested:
   Added eviction_completion_during_shutdown_does_not_panic that reproduces the race by exploiting FIFO ordering on the local channel. It uses make_wft_stream_interminable() to prevent PollerDead from arriving, processes the full workflow, polls the eviction activation, then calls initiate_shutdown() followed immediately by the eviction completion. This guarentees BumpStream is ahead of WFActCompleteMsg in the channel. Without the fix the test panics with the same error seen in CI; with the fix it passes.

3. Any docs updates needed?
   N/A

[chris-olszewski]

Quick note

[chris-olszewski]

I didn't include a request_eviction call here. I can add it if that makes sense, but I think the workflow stream will always be shut down if the response send side of the channel has been dropped.

[Sushisource]

Nice, this fix makes sense. This one has been lurking forever and I never really found a satisfying way to fix it.

I kept trying to do this by ensuring that the other half of the channel never went away, which I do still think is sort of the "real" fix, but, at the same time if this was only ever happening with ignore_evicts_on_shutdown being set, then I really don't care and this is 100% fine.