CoreCLR aborts when Objective-C dealloc re-enters managed code during pthread destruction on macOS

[rolfbjarne]

Description

After updating dotnet/macios from the .NET 11 runtime (11.0.0-rc.2.26465.108) to the .NET 12 runtime (12.0.0-alpha.1.26471.108), the macOS monotouch-test runner crashes while a worker thread exits. CoreAnimation releases a CALayer during pthread TLS cleanup; its Objective-C destructor calls into managed code to unregister the NSObject. CoreCLR has already destroyed the thread's runtime state and now aborts on the attempted re-entry:

CLR: Assert failure: !"Attempt to execute managed code after the .NET runtime thread state has been destroyed."
    src/runtime/src/coreclr/vm/ceemain.cpp:1894

The macOS crash report's relevant stack is:

CheckThreadStateNotDestroyed
SetupThread
JIT_ReversePInvokeEnterRare
xamarin_unregister_nsobject
xamarin_notify_dealloc
XamarinObject::~XamarinObject
-[...Layer .cxx_destruct]
-[CALayer dealloc]
CA::Transaction::commit
CA::Transaction::release_thread
_pthread_tsd_cleanup
_pthread_exit

Reproduction

In dotnet/macios, run MonoTouchFixtures.CoreAnimation.LayerTest.TestCALayerDelegateDispose on macOS with the .NET 12 runtime package above. Before the local test workaround was applied, the test created a CALayer subclass on a worker thread, set its delegate, called Dispose(), and returned from the thread. Running the full monotouch-test macOS suite crashed consistently, and running just this test also reproduced the abort. The earlier .NET 11 runtime passed this test.

Adding CATransaction.Flush() immediately after layer.Dispose() on the worker thread lets the native layer be released before that thread's runtime state is destroyed. With that workaround, the complete suite passes. This is a useful test fix, but it does not address other embedding callbacks that might occur during native TLS cleanup.

Suspected regression

dotnet/runtime#132448 introduced CheckThreadStateNotDestroyed() in SetupThread() and, on Apple platforms, a pthread-key marker that persists through later destructor callbacks. I verified that the guard is absent from the prior runtime source and present in the new runtime source. The new detection explains the abort; it does not establish that CoreAnimation's deferred release is itself new.

Mono previously handled this exact class of problem in dotnet/runtime commit 43998aa3cc6: thread_exited_dtor() restores a pthread-key marker so mono_thread_info_is_exiting() detects re-entry from later destructors (explicitly including Objective-C dealloc). Embedding code calls mono_thread_detach_if_exiting() after its managed callback to detach the temporary runtime thread. CoreCLR handles runtime attach/detach automatically; the new CoreCLR guard aborts before any post-callback cleanup can run.

Question

Is reverse P/Invoke from an Objective-C destructor during pthread teardown expected to be unsupported by CoreCLR, or should this scenario be supported with an approach analogous to Mono's attach/detach handling? If it must remain unsupported, is there a supported way for embedding code to detect that thread teardown is underway and avoid invoking managed code, without leaking the managed/native object registration?

Note

This issue was prepared with GitHub Copilot assistance.

[rolfbjarne]

CC @jkotas, @eduardo-vp (since this started with #132448)

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
See info in area-owners.md if you want to be subscribed.

[jkotas]

@eduardo-vp I think we should disable the new check until we figure out what to do here.

should this scenario be supported with an approach analogous to Mono's attach/detach handling?

Possibly.

Another option is to ignore the problem like what we do on Windows with IJW. (It is a potential memory leak and reliability problem.)