Potential soundness issue in Gecko FFI for auto_thin_vec

[Shnatsel]

An audit with GPT-5.5 has uncovered a potential soundness issue in Gecko FFI combined with auto_thin_vec.

ThinVec::shallow_size_of only checks if capacity is 0, and if not, assumes it's handling a valid heap pointer:

thin-vec/src/lib.rs

Lines 2132 to 2138 in 6db6b4e

	#[cfg(feature = "malloc_size_of")]
	impl<T> MallocShallowSizeOf for ThinVec<T> {
	fn shallow_size_of(&self, ops: &mut MallocSizeOfOps) -> usize {
	if self.capacity() == 0 {
	// If it's the singleton we might not be a heap pointer.
	return 0;
	}

But this isn't sufficient to distinguish heap pointers from stack pointers in gecko-ffi auto arrays, where the active header can be stack-backed with nonzero capacity, so a normal malloc-size callback receives a stack pointer. This trips Address Sanitizer on the following unit test:

#[cfg(all(
    feature = "gecko-ffi",
    feature = "malloc_size_of"
))]
#[test]
fn malloc_size_of_auto_array_passes_stack_pointer_to_malloc() {
    // Issue: auto arrays store their active header inline on the stack, but
    // `ThinVec<T>::shallow_size_of` only checks `capacity() == 0` before
    // passing the header pointer to `MallocSizeOfOps::malloc_size_of`.
    // For stack-backed auto arrays the capacity is nonzero, so a normal
    // malloc-size callback receives a stack pointer and trips ASAN's
    // malloc_usable_size ownership check.
    use malloc_size_of::{MallocShallowSizeOf, MallocSizeOfOps};
    use std::ffi::c_void;

    extern "C" {
        fn malloc_usable_size(ptr: *const c_void) -> usize;
    }

    unsafe extern "C" fn malloc_size_of(ptr: *const c_void) -> usize {
        // This is representative of Gecko's malloc-size callback: it is only
        // valid for heap allocation pointers.
        unsafe { malloc_usable_size(ptr) }
    }

    crate::auto_thin_vec!(let t: [u8; 4]);
    let mut ops = MallocSizeOfOps::new(malloc_size_of, None, None);
    let _ = MallocShallowSizeOf::shallow_size_of(&**t, &mut ops);
}

The Address Sanitizer failure can be reproduced with:

RUSTFLAGS='-Zsanitizer=address' ASAN_OPTIONS=detect_leaks=0 \
  cargo +nightly test -Zbuild-std --release \
  --target x86_64-unknown-linux-gnu \
  --features gecko-ffi,malloc_size_of \
  malloc_size_of_auto_array_passes_stack_pointer_to_mallo

I don't have any evidence that this is happening today in the Gecko codebase (after admittedly only a cursory inspection).

[emilio]

Yeah it's a real bug, but yeah unlikely to come up in practice since you generally don't call malloc_size_of methods on objects that are on the stack. Worth fixing tho.