workflow: support using a shared sandbox for all workflow invocations

[msullivan]

Importing everything on every workflow invocation is quite slow,
especially when using pydantic or similar.

I am planning to add automatic support for using pydantic to
serialize/validate arguments, and regenerating the pydantic validators
on every invocation will make it even more expensive. Using a shared
sandbox eliminates that problem.

Using a shared sandbox means that we still get all of the benefits of
preventing nondeterminstic calls, but that modifications to global
variables will stay visible.
But I think a good workflow shouldn't really be using globals
anyway, so...

I think I'm planning to basically immediately make a PR that will
switch the default to share_sandboxes=True, but for cleanliness I'm
separating it.

To do this, I:

• Add a Sandbox class entered with enter()
• Add a policy flag for it
• Make Workflows responsible for producing a sandbox
• Hold a sandbox-scoped lock while importing the module, to avoid
  weird cyclic init races.
• A bunch of tweaks to serde's registries

Questions:

• Are we nervous about sharing a sandbox for genuinely concurrent
  runs? If so, we can maintain a pool of unused sandboxes and create
  a new one when necessary.

[fantix]

Are we nervous about sharing a sandbox for genuinely concurrent runs? If so, we can maintain a pool of unused sandboxes and create a new one when necessary.

Each workflow run in the shared sandbox is sequential and serialized, right? Like workflow A starts an event loop, run to suspension, shutdown that loop, THEN workflow B can only start a new event loop after that, and do B things? I think I'm okay with shared sandboxes for this case.

[msullivan]

Are we nervous about sharing a sandbox for genuinely concurrent runs? If so, we can maintain a pool of unused sandboxes and create a new one when necessary.

Each workflow run in the shared sandbox is sequential and serialized, right? Like workflow A starts an event loop, run to suspension, shutdown that loop, THEN workflow B can only start a new event loop after that, and do B things? I think I'm okay with shared sandboxes for this case.

We will run multiple workflows in separate threads, so they can be genuinely concurrent. A and B in the same sandbox at the same time.

If we are nervous about that, we can make it so that a sandbox is in use by one thread at a time, and can then get reused later.

[fantix]

We will run multiple workflows in separate threads, so they can be genuinely concurrent. A and B in the same sandbox at the same time.

If we are nervous about that, we can make it so that a sandbox is in use by one thread at a time, and can then get reused later.

I feel like it's probably fine with the asyncio Runner thread-local thing, as far as each event loop is utilized sequentially for all runs within each thread.

[msullivan]

We will run multiple workflows in separate threads, so they can be genuinely concurrent. A and B in the same sandbox at the same time.
If we are nervous about that, we can make it so that a sandbox is in use by one thread at a time, and can then get reused later.

I feel like it's probably fine with the asyncio Runner thread-local thing, as far as each event loop is utilized sequentially for all runs within each thread.

Yeah, the event loop isn't shared.