.NET 11 breaks named Mutex interop with every earlier .NET on Linux and macOS

[danielo-unity3d]

Description

A .NET 11 process and a .NET 8/9/10 process that open the same named Mutex on Unix silently corrupt each other's owner/abandonment record: the .NET 11 side throws AbandonedMutexException on every acquisition that follows a normal release by the older side, while exclusion keeps working and nothing reports a version mismatch.

Since #117635, a named Mutex on Unix is implemented in managed code (src/libraries/System.Private.CoreLib/src/System/Threading/NamedMutex.Unix.cs, plus LowLevelCrossProcessMutex in src/native/libs/System.Native/pal_crossprocessmutex.c) instead of the CoreCLR PAL (src/coreclr/pal/src/synchobj/mutex.cpp).

Both implementations use the same shared-memory file (/tmp/.dotnet/shm/global/<name>, or the session/user-scoped variants), the same 8-byte header, SharedMemoryType.Mutex, and the same SyncSystemVersion = 1.
A .NET 11 process and a .NET ≤ 10 process that open a mutex of the same name therefore accept each other's file as valid and use it together.

The layout of the shared data behind the header changed, though:

field	.NET ≤ 10 PAL (mutex.hpp @ release/10.0, L166–176)	.NET 11 managed (NamedMutex.Unix.cs SharedData, L585–590; pal_crossprocessmutex.c LowLevelCrossProcessMutex)
lock (pthread_mutex_t on Linux, UINT32 m_timedWaiterCount on macOS)	same	same
lockOwnerProcessId	UINT32	uint
lockOwnerThreadId	UINT64, the OS thread id (THREADSilentGetCurrentThreadId()), InvalidSharedThreadId (all 0xFF) when unowned	uint, Thread.CurrentThread.ManagedThreadId, InvalidThreadId = (uint)-1 when unowned
isAbandoned	bool after the 64-bit field: @16 (macOS), @56 (Linux x64)	byte after the 32-bit field: @12 (macOS), @48 (Linux x64)

The byte .NET 11 reads as IsAbandoned therefore lies inside the PAL's 64-bit m_lockOwnerThreadId.
The PAL writes all 0xFF into that field on every release, and .NET 11's TryAcquireLock consults IsAbandoned after every acquisition (L126–129):

if (IsAbandoned)
{
    IsAbandoned = false;
    result = MutexTryAcquireLockResult.AcquiredLockButMutexWasAbandoned;
}

So every time a .NET 11 process acquires a named mutex that a .NET ≤ 10 process last released, WaitOne throws AbandonedMutexException, although nothing was abandoned.

Nothing detects the mismatch: the file is page-aligned, the version byte is 1 on both sides, and exclusion itself keeps working because the lock sits at the same offset in both layouts.
The two processes do exclude each other correctly; only the owner/abandonment protocol is corrupted, which makes the failure easy to attribute to a crash in the other process instead.

The reverse direction is mostly benign (edit: maybe not so benign after review): the PAL detects abandonment through m_lockOwnerProcessId, which is at the same offset in both layouts, and its m_isAbandoned byte lies past the end of the .NET 11 struct, where nothing writes.

Reproduction Steps

The layout mismatch can be shown from the .NET 10 side alone by decoding the shared-memory file; the two-runtime steps at the end show the behavioural consequence.

NamedMutexLayout.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>disable</ImplicitUsings>
  </PropertyGroup>
</Project>

Program.cs:

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;

var mode = args.Length > 0 ? args[0] : "";
var name = args.Length > 1 ? args[1] : "layout-repro";
var globalName = @"Global\" + name;

switch (mode)
{
    case "hold": // acquire, wait for Enter, release, wait for Enter, exit
    {
        using var mutex = new Mutex(false, globalName);
        Console.WriteLine($"[.NET {Environment.Version}] hold: acquired={mutex.WaitOne(0)} (pid {Environment.ProcessId}). Enter releases.");
        Console.ReadLine();
        mutex.ReleaseMutex();
        Console.WriteLine($"[.NET {Environment.Version}] hold: released. Enter exits.");
        Console.ReadLine();
        return 0;
    }
    case "probe": // one non-blocking acquisition
    {
        using var mutex = new Mutex(false, globalName);
        try
        {
            if (!mutex.WaitOne(0)) { Console.WriteLine($"[.NET {Environment.Version}] probe: HELD by someone else"); return 2; }
            Console.WriteLine($"[.NET {Environment.Version}] probe: CLEAN acquisition");
        }
        catch (AbandonedMutexException)
        {
            Console.WriteLine($"[.NET {Environment.Version}] probe: ABANDONED - AbandonedMutexException");
        }
        mutex.ReleaseMutex();
        return 0;
    }
    case "dump": // decode the shared data under both layouts
    {
        var bytes = File.ReadAllBytes(Path.Combine("/tmp/.dotnet/shm/global", name));
        var data = bytes.AsSpan(8); // after the 8-byte header
        Console.WriteLine($"shared data: {BitConverter.ToString(bytes, 8, 24).Replace('-', ' ')}");

        // Linux: pthread_mutex_t first (40 bytes on glibc x86_64, 48 on aarch64); macOS: UINT32 timed-waiter count.
        var pidOffset = OperatingSystem.IsMacOS() ? 4 : (RuntimeInformation.ProcessArchitecture == Architecture.Arm64 ? 48 : 40);
        var tidOffset = pidOffset + 4;
        var palTidOffset = (tidOffset + 7) / 8 * 8; // UINT64, 8-byte aligned

        Console.WriteLine($"PAL (.NET <= 10) reads: pid @{pidOffset} = {U32(data, pidOffset):X8}  tid(u64) @{palTidOffset} = {U64(data, palTidOffset):X16}  isAbandoned @{palTidOffset + 8} = {data[palTidOffset + 8]}");
        Console.WriteLine($".NET 11 reads:          pid @{pidOffset} = {U32(data, pidOffset):X8}  tid(u32) @{tidOffset} = {U32(data, tidOffset):X8}          isAbandoned @{tidOffset + 4} = {data[tidOffset + 4]}");
        return 0;
    }
    default:
        Console.Error.WriteLine("usage: hold|probe|dump <name>");
        return 1;
}

static uint U32(ReadOnlySpan<byte> d, int o) => BitConverter.ToUInt32(d.Slice(o, 4));
static ulong U64(ReadOnlySpan<byte> d, int o) => BitConverter.ToUInt64(d.Slice(o, 8));

Step 1 — what a .NET 10 process leaves in the file (no .NET 11 needed):

dotnet build -c Release

# terminal 1: hold the mutex on .NET 10
bin/Release/net10.0/NamedMutexLayout hold repro

# terminal 2: dump while held; then press Enter in terminal 1 (release) and dump again
bin/Release/net10.0/NamedMutexLayout dump repro
bin/Release/net10.0/NamedMutexLayout dump repro

Step 2 — the behavioural consequence, with both SDKs installed (add net11.0 to TargetFrameworks):

# terminal 1 (.NET 10): hold, press Enter once to release, keep the process alive
dotnet run -c Release -f net10.0 -- hold repro

# terminal 2 (.NET 11): acquire what the .NET 10 process just released
dotnet run -c Release -f net11.0 -- probe repro

Expected behavior

• Step 1: the field that .NET 11 will read as IsAbandoned is 0 after a normal release by a .NET 10 process.
• Step 2: probe: CLEAN acquisition — the mutex was released normally by a live process. (Running both sides on .NET 10, or both on .NET 11, does print CLEAN.)

Actual behavior

Step 1, macOS 26.6.2 arm64, .NET runtime 10.0.12:

# while held by the .NET 10 process
shared data: 00 00 00 00 7E B2 00 00 C4 00 32 00 00 00 00 00 00 00 00 00 00 00 00 00
PAL (.NET <= 10) reads: pid @4 = 0000B27E  tid(u64) @8 = 00000000003200C4  isAbandoned @16 = 0
.NET 11 reads:          pid @4 = 0000B27E  tid(u32) @8 = 003200C4          isAbandoned @12 = 0

# after the .NET 10 process released it (file still open)
shared data: 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF FF 00 00 00 00 00 00 00 00
PAL (.NET <= 10) reads: pid @4 = FFFFFFFF  tid(u64) @8 = FFFFFFFFFFFFFFFF  isAbandoned @16 = 0
.NET 11 reads:          pid @4 = FFFFFFFF  tid(u32) @8 = FFFFFFFF          isAbandoned @12 = 255

The upper half of the PAL's cleared 64-bit thread id is what .NET 11 reads as IsAbandoned.
On Linux the same happens at @48/@56: the low byte of the PAL's thread id becomes .NET 11's IsAbandoned, 0xFF after a release.

Step 2: probe: ABANDONED - AbandonedMutexException on the .NET 11 side, after every normal release by the .NET 10 side.

Regression?

Yes. Any two CoreCLR processes on .NET ≤ 10 (framework-dependent or self-contained, any mix of versions) share the PAL layout and interoperate.
.NET 11 is the first version whose named Mutex writes a different layout under the same SyncSystemVersion, so a .NET 11 process next to a .NET 8/9/10 process on the same machine no longer does.

(NativeAOT on Unix is a separate history: its named mutexes were per-process until .NET 11, #48720 / #110348, so there was nothing to be compatible with there.)

Known Workarounds

• Do not share a named mutex between a .NET 11 process and a .NET ≤ 10 process on Linux/macOS: use a different name per runtime major, or coordinate through something else (flock on a file of your own, a Unix socket).
• Catch AbandonedMutexException and ignore it when the other side is known to be a .NET ≤ 10 process. Fragile: real abandonment is then indistinguishable from the phantom one.
• What we did: carry our own port of the .NET 11 implementation with the PAL's layout (64-bit OS thread id, isAbandoned after it) so that our NativeAOT .NET 10 build interoperates with our shipped CoreCLR builds (AT SOME COST!). That is code we would like to delete on .NET 11, which needs .NET 11's Mutex to read and write the PAL layout.

Configuration

Which version of .NET is the code running on?

The PAL side of the repro: .NET SDK 10.0.401, runtime 10.0.12.
The .NET 11 side: main as of today (NamedMutex.Unix.cs SharedData: uint _lockOwnerThreadId; byte _isAbandoned;, SyncSystemVersion = 1); exclusion between .NET 10 and .NET 11 RC1 processes was also verified to work, which is what hides this.

What OS and version, and what distro if applicable?
Reproduced on macOS 26.6.2 (build 25G83).
Linux (Ubuntu 22.04, glibc) has the same mismatch with the fields shifted by sizeof(pthread_mutex_t): PAL isAbandoned @56, .NET 11 IsAbandoned @48.

What is the architecture (x64, x86, ARM, ARM64)?

ARM64 for the captured output; the offsets on x64 are identical on macOS and differ on Linux only by sizeof(pthread_mutex_t) (40 vs 48).

Do you know whether it is specific to that configuration?

No: it is a property of the two struct definitions and applies to every Linux and macOS configuration. Windows is unaffected (kernel named mutexes, no shared-memory file).

Not using Blazor

Other information

Suggested fix, in order of preference:

1. Keep the PAL layout in the managed implementation: a 64-bit owner thread id at the PAL's offset with isAbandoned after it.
   The managed side can still store its managed thread id in the low 32 bits; what matters for compatibility is where isAbandoned lives and what "unowned" looks like (InvalidSharedThreadId, all 0xFF, in the full 64 bits).
   This keeps .NET 11 interoperable with every earlier CoreCLR, which is the property named mutexes have on Windows.
2. Failing that, bump SyncSystemVersion (or otherwise make the header identify the layout) so that mismatched implementations fail loudly at open time instead of silently misreading each other.
   A .NET 11 process then cannot coordinate with a .NET 10 one at all, but that is strictly better than the current silent corruption of the abandonment protocol.

The repro's dump mode assumes glibc's sizeof(pthread_mutex_t) on Linux (40 on x86_64, 48 on aarch64); musl differs.

Related: #48720, #110348 (NativeAOT named mutexes per-process on Unix, fixed by #117635 — the change this issue is about).

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
See info in area-owners.md if you want to be subscribed.

[jkotas]

cc @jkoritzinsky

[am11]

We can keep the .NET 10 layout (both pthread and file-lock paths):

diff --git a/src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelCrossProcessMutex.cs b/src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelCrossProcessMutex.cs
index b479b52d9ea..305188ffda5 100644
--- a/src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelCrossProcessMutex.cs
+++ b/src/libraries/Common/src/Interop/Unix/System.Native/Interop.LowLevelCrossProcessMutex.cs
@@ -25,11 +25,11 @@ internal static unsafe partial class Sys
 
         [LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId", SetLastError = true)]
         [SuppressGCTransition]
-        internal static partial void LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(void* mutex, out uint processId, out uint threadId);
+        internal static partial void LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(void* mutex, out uint processId, out ulong threadId);
 
         [LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId", SetLastError = true)]
         [SuppressGCTransition]
-        internal static partial void LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(void* mutex, uint processId, uint threadId);
+        internal static partial void LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(void* mutex, uint processId, ulong threadId);
 
         [LibraryImport(Libraries.SystemNative, EntryPoint = "SystemNative_LowLevelCrossProcessMutex_IsAbandoned", SetLastError = true)]
         [SuppressGCTransition]
diff --git a/src/libraries/System.Private.CoreLib/src/System/Threading/NamedMutex.Unix.cs b/src/libraries/System.Private.CoreLib/src/System/Threading/NamedMutex.Unix.cs
index 077f68c63b2..15745509d7d 100644
--- a/src/libraries/System.Private.CoreLib/src/System/Threading/NamedMutex.Unix.cs
+++ b/src/libraries/System.Private.CoreLib/src/System/Threading/NamedMutex.Unix.cs
@@ -69,10 +69,13 @@ public void Abandon()
 
     internal abstract class NamedMutexProcessDataBase(SharedMemoryProcessDataHeader<NamedMutexProcessDataBase> header) : ISharedMemoryProcessData
     {
+        // The shared data layout must stay compatible with the CoreCLR PAL implementation used by .NET 10 and earlier
+        // (src/coreclr/pal/src/synchobj/mutex.cpp), which uses the same shared memory files and the same SyncSystemVersion.
+        // Processes from both implementations may open the same named mutex at the same time.
         private const byte SyncSystemVersion = 1;
         protected const int PollLoopMaximumSleepMilliseconds = 100;
         protected const uint InvalidProcessId = unchecked((uint)-1);
-        protected const uint InvalidThreadId = unchecked((uint)-1);
+        protected const ulong InvalidThreadId = unchecked((ulong)-1);
 
         // Use PThread mutex-backed named mutexes if possible.
         // macOS has support for the features we need in the pthread mutexes on arm64
@@ -298,7 +301,7 @@ internal sealed unsafe class NamedMutexProcessDataWithPThreads(SharedMemoryProce
 
         protected override bool IsLockOwnedByThreadInThisProcess(Thread thread)
         {
-            Interop.Sys.LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(_sharedData, out uint ownerProcessId, out uint ownerThreadId);
+            Interop.Sys.LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(_sharedData, out uint ownerProcessId, out ulong ownerThreadId);
             return ownerProcessId == (uint)Environment.ProcessId &&
                    ownerThreadId == (uint)thread.ManagedThreadId;
         }
@@ -581,12 +584,14 @@ protected override unsafe MutexTryAcquireLockResult AcquireLockCore(int timeoutM
             }
         }
 
+        // Must match the file lock variant of NamedMutexSharedData in the CoreCLR PAL used by .NET 10 and earlier
+        // (src/coreclr/pal/src/include/pal/mutex.hpp): the owner thread ID is 64 bits wide and _isAbandoned follows it.
         [StructLayout(LayoutKind.Sequential)]
         internal ref struct SharedData
         {
             private uint _timedWaiterCount;
             private uint _lockOwnerProcessId;
-            private uint _lockOwnerThreadId;
+            private ulong _lockOwnerThreadId;
             private byte _isAbandoned;
 
             public uint TimedWaiterCount { get => _timedWaiterCount; set => _timedWaiterCount = value; }
@@ -602,7 +607,7 @@ public uint LockOwnerProcessId
                 }
             }
 
-            public uint LockOwnerThreadId
+            public ulong LockOwnerThreadId
             {
                 get
                 {
diff --git a/src/native/libs/System.Native/pal_crossprocessmutex.c b/src/native/libs/System.Native/pal_crossprocessmutex.c
index 0d18731c6e9..3cc1f95ff68 100644
--- a/src/native/libs/System.Native/pal_crossprocessmutex.c
+++ b/src/native/libs/System.Native/pal_crossprocessmutex.c
@@ -69,16 +69,19 @@ static int32_t AcquirePThreadMutexWithTimeout(pthread_mutex_t* mutex, int32_t ti
 #endif
 }
 
+// This layout lives in the shared memory file and is shared with other processes. It must match the layout written by the
+// CoreCLR PAL in .NET 10 and earlier (NamedMutexSharedData in src/coreclr/pal/src/include/pal/mutex.hpp), since both
+// implementations use the same SyncSystemVersion. In particular, OwnerThreadId is 64 bits wide and IsAbandoned follows it.
 struct LowLevelCrossProcessMutex
 {
     pthread_mutex_t Mutex;
     uint32_t OwnerProcessId;
-    uint32_t OwnerThreadId;
+    uint64_t OwnerThreadId;
     uint8_t IsAbandoned;
 };
 
 #define INVALID_PROCESS_ID (uint32_t)(-1)
-#define INVALID_THREAD_ID (uint32_t)(-1)
+#define INVALID_THREAD_ID (uint64_t)(-1)
 
 int32_t SystemNative_LowLevelCrossProcessMutex_Size(void)
 {
@@ -136,7 +139,7 @@ int32_t SystemNative_LowLevelCrossProcessMutex_Destroy(LowLevelCrossProcessMutex
     return ConvertErrorPlatformToPal(pthread_mutex_destroy(&mutex->Mutex));
 }
 
-void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t* pOwnerProcessId, uint32_t* pOwnerThreadId)
+void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t* pOwnerProcessId, uint64_t* pOwnerThreadId)
 {
     assert(mutex != NULL);
     assert(pOwnerProcessId != NULL);
@@ -146,7 +149,7 @@ void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelC
     *pOwnerThreadId = mutex->OwnerThreadId;
 }
 
-void SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t ownerProcessId, uint32_t ownerThreadId)
+void SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t ownerProcessId, uint64_t ownerThreadId)
 {
     assert(mutex != NULL);
 
diff --git a/src/native/libs/System.Native/pal_crossprocessmutex.h b/src/native/libs/System.Native/pal_crossprocessmutex.h
index d48de4bda39..8dd4233897c 100644
--- a/src/native/libs/System.Native/pal_crossprocessmutex.h
+++ b/src/native/libs/System.Native/pal_crossprocessmutex.h
@@ -18,9 +18,9 @@ PALEXPORT int32_t SystemNative_LowLevelCrossProcessMutex_Release(LowLevelCrossPr
 
 PALEXPORT int32_t SystemNative_LowLevelCrossProcessMutex_Destroy(LowLevelCrossProcessMutex* mutex);
 
-PALEXPORT void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t* pOwnerProcessId, uint32_t* pOwnerThreadId);
+PALEXPORT void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t* pOwnerProcessId, uint64_t* pOwnerThreadId);
 
-PALEXPORT void SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t ownerProcessId, uint32_t ownerThreadId);
+PALEXPORT void SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t ownerProcessId, uint64_t ownerThreadId);
 
 PALEXPORT uint8_t SystemNative_LowLevelCrossProcessMutex_IsAbandoned(LowLevelCrossProcessMutex* mutex);
 
diff --git a/src/native/libs/System.Native/pal_crossprocessmutex_unsupported.c b/src/native/libs/System.Native/pal_crossprocessmutex_unsupported.c
index a155fb5708a..61b33b76073 100644
--- a/src/native/libs/System.Native/pal_crossprocessmutex_unsupported.c
+++ b/src/native/libs/System.Native/pal_crossprocessmutex_unsupported.c
@@ -47,14 +47,14 @@ int32_t SystemNative_LowLevelCrossProcessMutex_Destroy(LowLevelCrossProcessMutex
     return Error_EINVAL;
 }
 
-void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t* pOwnerProcessId, uint32_t* pOwnerThreadId)
+void SystemNative_LowLevelCrossProcessMutex_GetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t* pOwnerProcessId, uint64_t* pOwnerThreadId)
 {
     (void)mutex;
     (void)pOwnerProcessId;
     (void)pOwnerThreadId;
 }
 
-void SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t ownerProcessId, uint32_t ownerThreadId)
+void SystemNative_LowLevelCrossProcessMutex_SetOwnerProcessAndThreadId(LowLevelCrossProcessMutex* mutex, uint32_t ownerProcessId, uint64_t ownerThreadId)
 {
     (void)mutex;
     (void)ownerProcessId;

linux-arm/arm64 stays on pthread (file locks in .NET 10), an acceptable break IMO.

[jkoritzinsky]

We can't take a break here, so I'll revert linux arm/arm64 to file locks as well.

[danielo-unity3d]

Thank you for the prompt fix.
I'll try it soon, when it gets released.