byte*[] parameter is corrupted when marshalled to native code on x64 (regressed in 11.0.0-preview.5)

[TommiIversen]

Description

A P/Invoke parameter declared as an array of pointers no longer arrives intact on x64. The callee receives a pointer to a copy of the array instead of the pinned array, and the elements in that copy are wrong: the low 32 bits of each pointer are correct, the upper 32 bits are garbage.

Nothing throws — native code simply gets invalid pointers. In our case libswscale's sws_scale faults with 0xC0000005 on the first call, where the same IL against the same native library runs fine on .NET 10. It reaches the runtime through FFmpeg.AutoGen, whose generated binding declares the plane arrays as byte*[].

Both shapes are affected: a plain [DllImport], and a [UnmanagedFunctionPointer] delegate from Marshal.GetDelegateForFunctionPointer. In the same call an int[] parameter is passed at its pinned address with correct contents, so only arrays whose element type is a pointer are affected. IntPtr[] is copied as well, but its values survive.

Reproduction Steps

Console app, net10.0;net11.0, AllowUnsafeBlocks and ImplicitUsings enabled. No native dependency: the callee is an [UnmanagedCallersOnly] managed method, so it can print what it actually received.

using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

unsafe
{
    Console.WriteLine($"runtime={Environment.Version}");

    var planes = new byte*[4];
    var strides = new int[4];
    var buffer = new byte[64];

    fixed (byte* pinned = buffer)
    {
        planes[0] = pinned;
        planes[1] = pinned + 16;
        strides[0] = 1111;
        strides[1] = 2222;

        fixed (byte** expectedPlanes = planes)
        fixed (int* expectedStrides = strides)
            Console.WriteLine($"expected: planes={(long) expectedPlanes:x} strides={(long) expectedStrides:x} " +
                              $"planes[0]={(long) planes[0]:x} planes[1]={(long) planes[1]:x}");

        var target = (IntPtr) (delegate* unmanaged[Cdecl]<byte**, int*, int, int>) &Callee.Receive;

        Console.WriteLine("-- byte*[] and int[] --");
        Marshal.GetDelegateForFunctionPointer<ArrayShape>(target)(planes, strides, 7);

        Console.WriteLine("-- IntPtr[] and int[] --");
        var nativeInts = new IntPtr[4];
        nativeInts[0] = (IntPtr) planes[0];
        nativeInts[1] = (IntPtr) planes[1];
        Marshal.GetDelegateForFunctionPointer<IntPtrShape>(target)(nativeInts, strides, 7);

        Console.WriteLine("-- byte** and int* --");
        var pointerDelegate = Marshal.GetDelegateForFunctionPointer<PointerShape>(target);
        fixed (byte** p = planes)
        fixed (int* s = strides)
            pointerDelegate(p, s, 7);
    }
}

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
internal unsafe delegate int ArrayShape(byte*[] planes, int[] strides, int n);

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
internal unsafe delegate int IntPtrShape(IntPtr[] planes, int[] strides, int n);

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
internal unsafe delegate int PointerShape(byte** planes, int* strides, int n);

internal static unsafe class Callee
{
    [UnmanagedCallersOnly(CallConvs = [typeof(CallConvCdecl)])]
    internal static int Receive(byte** planes, int* strides, int n)
    {
        Console.WriteLine($"   callee: planes={(long) planes:x} strides={(long) strides:x} n={n}");
        if (planes != null) Console.WriteLine($"   callee: planes[0]={(long) planes[0]:x} planes[1]={(long) planes[1]:x}");
        if (strides != null) Console.WriteLine($"   callee: strides[0]={strides[0]} strides[1]={strides[1]}");
        return 42;
    }
}

Expected behavior

.NET 10.0.2, same source, same machine. The callee gets the pinned array itself and the elements are intact:

runtime=10.0.2
expected: planes=17ebd49e518 strides=17ebd49e550 planes[0]=17ebd49e578 planes[1]=17ebd49e588
-- byte*[] and int[] --
   callee: planes=17ebd49e518 strides=17ebd49e550 n=7
   callee: planes[0]=17ebd49e578 planes[1]=17ebd49e588
   callee: strides[0]=1111 strides[1]=2222

Actual behavior

.NET 11.0.0-rc.1.26425.128:

runtime=11.0.0
expected: planes=255e5c98bf0 strides=255e5c98c28 planes[0]=255e5c98c50 planes[1]=255e5c98c60
-- byte*[] and int[] --
   callee: planes=255e178a9e0 strides=255e5c98c28 n=7
   callee: planes[0]=3a000a18e5c98c50 planes[1]=88001600de9ca268
   callee: strides[0]=1111 strides[1]=2222

planes is a copy (255e178a9e0, not the pinned 255e5c98bf0), while strides in the same call is at its pinned address. planes[0] should be 255e5c98c50; the low 32 bits (e5c98c50) are right and the upper 32 bits are not. planes[1] is unrelated to the real value. Across runs the upper half is sometimes garbage and sometimes zero, which looks like an element size of 4 bytes written into an 8-byte slot of a buffer that isn't fully initialised.

Regression?

Yes, and it narrows to one preview. Correct on .NET 10.0.2 and on 11.0 preview 1 through preview 4. Corrupt from preview 5 onwards:

runtime	result
11.0.0-preview.1.26104.118 … preview.4.26230.115	correct
11.0.0-preview.5.26302.115	corrupt
11.0.0-preview.6.26359.118, preview.7.26381.103	corrupt
11.0.0-rc.1.26425.128	corrupt
11.0.0-rc.2.26467.112 (current daily)	corrupt

The only array-marshalling change in that window looks to be #126911 ("Move built-in array marshalling to managed", merged 2026-05-01), which rewrote the LPArray path in StubHelpers.cs and dropped the VTHACK_* element tags. The IArrayElementMarshaler type it introduces is absent from preview 4's System.Private.CoreLib.dll and present in preview 5's, so the timing fits — but that is correlation only, we have not built the runtime with and without it.

Unaffected by DOTNET_TieredCompilation=0 and by DOTNET_RuntimeAsync=0/=1.

Known Workarounds

Keep the array away from the marshaller: declare the parameter as byte** and pin at the call site, or call the entry point through a delegate* unmanaged[Cdecl]. Both are correct on every runtime tested. For a generated binding you do not own, the delegate field can be replaced with a managed lambda that pins and forwards.

[assembly: DisableRuntimeMarshalling] is not a workaround. The call then throws MarshalDirectiveException: Cannot marshal 'parameter #1' on .NET 10 as well, since arrays are managed types.

Configuration

Windows 11 Pro 10.0.26200, x64, x64 process, CoreCLR, Debug build. Runtimes as listed above; SDK 11.0.100-rc.1.26425.128.

Not tested: Linux, arm64, 32-bit, NativeAOT, trimming, the LibraryImport source generator.

Other information

[DllImport("swscale-10.dll", EntryPoint = "sws_scale", CallingConvention = CallingConvention.Cdecl)] declared with the same byte*[] signature crashes identically on .NET 11 and runs correctly on .NET 10, so this is not specific to GetDelegateForFunctionPointer.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.