[Test] Run Apple device smoke tests on scouting queues

[matouskozak]

Test-only PR. Do not merge.

Use the Apple scouting queues from dotnet/xharness#1714:

Device	Helix queue
iPhone	osx.amd64.iphone.scouting.open
Apple TV	osx.amd64.appletv.scouting.open

Run four Release smoke-test jobs under the main runtime pipeline: iOS and tvOS, each with CoreCLR and NativeAOT.

Use the existing smoke-test selection (RunSmokeTestsOnly=true) to limit device queue use. Defer the full library suites, Checked tvOS job, runtime-test suite, and separate Apple build tests. Keep the Apple app CMake deployment target at 15.0.

Run these jobs on PR updates without path filters. Keep one Helix monitor. Leave the normal queue definitions unchanged outside this test pipeline.

Skip all other public pipelines, including runtime-ioslike, through the shared pipeline template without allocating build agents. Keep the separate VMR PR trigger disabled.

The YAML matrix and smoke-project selection were checked locally. Device CI results for this smoke-only configuration are pending. Previously queued full-test work is not cancelled by this change.

Note

This PR was generated by GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[matouskozak]

/azp run runtime-ioslike

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to 'os-ios': @vitek-karas, @kotlarmilos, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[matouskozak]

/azp run runtime-ioslike

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[matouskozak]

/azp run runtime-ioslike

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[matouskozak]

Device-test failures compared with main

PR snapshot: 22946b4bc985, build 1613523, 17 failed Helix work items. Matching failures across jobs are grouped below. All configurations are arm64 Release.

Main-device baseline: build 1614142, physical Apple TVs on tvOS 26.2/26.6. Main's physical-iPhone CoreCLR/NativeAOT smoke jobs also pass in build 1615169, but do not include the PR's failing work items.

1. PR failures whose tests pass on main devices

BitArray: Clone_LongLength_Works

PR: iOS CoreCLR, System.Collections.Tests, one failed case. Main: the same test passes on CoreCLR, physical tvOS 26.6.

[FAIL] System.Collections.Tests.BitArray_CtorTests.Clone_LongLength_Works
System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

PR console | Main test results

XML: allocation failures followed by an unhandled OOM

PR: iOS CoreCLR, System.Private.Xml.Tests, 16 logged failed cases: WrappedReaderTest.RunTests (2), SubtreeReaderTest.RunTests (3), FactoryReaderTest.RunTests (6), CharCheckingReaderTest.RunTests (4), and TCFullEndElement+TCWriteValue.writeValue_27 (1). The reader failures are Base64/BinHex overflow cases. The writer case uses CustomWriter, synchronous execution, TimeSpan to UInt16, param=1, isValid=false.

Main: matching test-case names pass on CoreCLR, physical tvOS 26.2; the assembly finishes successfully. Some parameterized cases share display names.

[FAIL] System.Xml.WrappedReaderTests.WrappedReaderTest.RunTests(testCase: 105376: ReadBase64 runs into an Overflow)
System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

The PR subsequently aborts while reporting results:

Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

App terminated due to signal 6.

PR console | Main test results

CPU usage: EnvironmentTests.TestCpuUsage

PR: iOS CoreCLR and NativeAOT, System.Runtime.Extensions.Tests, one failure per job. Main: the test passes on both runtimes on physical tvOS 26.2.

CoreCLR:

Unexpected Zero privileged time while having user time: 00:00:00.4796340

NativeAOT:

Unexpected Zero privileged time while having user time: 00:00:00.5820430

PR consoles: CoreCLR, NativeAOT. Main results: CoreCLR, NativeAOT.

2. PR failures not executed by the inspected main-device jobs

These are not established pass-on-main cases: the corresponding tests are omitted, filtered, skipped, or have no equivalent runtime configuration.

Unix file modes: SetUser is lost

PR: iOS CoreCLR and NativeAOT, 48 failed cases each in System.IO.FileSystem.Tests. Affects SetThenGet in DirectoryInfo_GetSetUnixFileMode, FileInfo_GetSetUnixFileMode, File_GetSetUnixFileMode, and File_GetSetUnixFileMode_SafeFileHandle; every failure differs by the missing SetUser bit.

Assert.Equal() Failure: Values differ
Expected: SetUser
Actual:   None

Main tvOS: these parameter values are explicitly filtered out, not passing.

PR consoles: CoreCLR, NativeAOT.

NetworkInformation: timeout on both runtimes, plus two CoreCLR failures

PR: iOS CoreCLR and NativeAOT, System.Net.NetworkInformation.Functional.Tests:

Run timed out after 4500 seconds

CoreCLR additionally logs IPGlobalProperties_AccessAllMethods_NoErrors failing in BsdIPGlobalProperties.GetTcpConnections:

System.OverflowException : Arithmetic operation resulted in an overflow.

And IPGlobalProperties_UdpListeners_Succeed(address: 127.0.0.1) failing in GetActiveUdpListeners:

System.PlatformNotSupportedException : The information requested is unavailable on the current platform.

Main tvOS: the assembly is excluded for device hangs under #60713. That documents an existing coverage gap, not proof of the exact same underlying hang.

PR consoles: CoreCLR, NativeAOT.

Tracing: three distinct signatures

PR: iOS CoreCLR, System.Diagnostics.Tracing.Tests, eight cases:

• Six EventPipeListener cases: TestsWrite.Test_Write_T; TestsWriteEvent.Test_WriteEvent and Test_WriteEvent_ByteArray, each with useSelfDescribingEvents=false/true; and Test_WriteEvent_ComplexData_SelfDescribing. All fail with ServerNotAvailableException; the diagnostic message includes:

Unable to connect to Process 4034. The total length of the diagnostic socket path may exceed 108 characters.

• TestsManifestNegative.Test_GenerateManifest_InvalidEventSources:

Assert.Throws() Failure: No exception was thrown
Expected: typeof(System.ArgumentException)

• TestsUserErrors.Test_BadEventSource_MismatchedIds:

Expected: "EventSourceMessage"
Actual:   "Event1"

Main tvOS: the assembly is excluded. PR console

Reflection.Emit: save-validation hits an app-bundle permission error

PR: iOS CoreCLR, AssemblySaveAssemblyBuilderTests.PersistedAssemblyBuilder_SaveValidations:

Assert.Throws() Failure: Exception type was not an exact match
Expected: typeof(System.InvalidOperationException)
Actual:   typeof(System.UnauthorizedAccessException)

Access to System.Reflection.Emit.Tests.app/File is denied, with inner error Operation not permitted. Main tvOS: the assembly is excluded. PR console

NativeAOT runtime: UnwindFpBleedTest

PR: iOS and tvOS NativeAOT, one failure each in Exceptions:

Assert.Equal() Failure: Values differ
Expected: 100
Actual:   1

Main: no equivalent Apple NativeAOT runtime-test job was found. The CoreCLR tvOS counterpart is explicitly Skip in the result XML; its green Exceptions work item has 0 passed / 2 skipped tests.

PR consoles: iOS, tvOS. Main CoreCLR console.

NativeAOT runtime: b425314.Mutate.TestEntryPoint

PR: tvOS NativeAOT, Regression_d, one failed test containing two scenario timeouts; expected return 100, actual 101:

Timeout: The `CastClass' scenario exceeded the time limit (180s) without reaching the basic work thresholds.
Timeout: The `Unbox' scenario exceeded the time limit (180s) without reaching the basic work thresholds.

Main: no equivalent tvOS NativeAOT runtime-test job was found. The CoreCLR tvOS counterpart is Skip; its green Regression_d work item has 0 passed / 2 skipped tests.

PR console | Main CoreCLR console

NativeAOT runtime: missing inputs before app launch

PR: MobileSmokeTest and nativeaot on both iOS and tvOS, four failed work items. Representative iOS errors:

Unhandled exception. System.CommandLine.CommandLineException: No files matching /tmp/helix/working/A4E8091A/w/B3ED0A15/e/publish/MobileSmokeTest.dll

Unhandled exception. System.CommandLine.CommandLineException: No files matching /tmp/helix/working/A4E8091A/w/B4310A23/e/publish/nativeaot.dll

ILC exits 134, followed by missing-provisioning/signing errors and wrapper exit 21. These are pre-launch failures, not four executed test failures. The inspected main device jobs do not run these runtime-test work items.

PR consoles: iOS MobileSmokeTest, iOS nativeaot, tvOS MobileSmokeTest, tvOS nativeaot.

Note

This analysis was prepared with GitHub Copilot.

[matouskozak]

Device-test failures compared with main

PR snapshot: 22946b4bc985, build 1613523, 17 failed Helix work items. Matching failures across jobs are grouped below. All configurations are arm64 Release.

Main-device baseline: build 1614142, physical Apple TVs on tvOS 26.2/26.6. Main's physical-iPhone CoreCLR/NativeAOT smoke jobs also pass in build 1615169, but do not include the PR's failing work items.

1. PR failures whose tests pass on main devices

BitArray: Clone_LongLength_Works

PR: iOS CoreCLR, System.Collections.Tests, one failed case. Main: the same test passes on CoreCLR, physical tvOS 26.6.

[FAIL] System.Collections.Tests.BitArray_CtorTests.Clone_LongLength_Works
System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

PR console | Main test results

XML: allocation failures followed by an unhandled OOM

PR: iOS CoreCLR, System.Private.Xml.Tests, 16 logged failed cases: WrappedReaderTest.RunTests (2), SubtreeReaderTest.RunTests (3), FactoryReaderTest.RunTests (6), CharCheckingReaderTest.RunTests (4), and TCFullEndElement+TCWriteValue.writeValue_27 (1). The reader failures are Base64/BinHex overflow cases. The writer case uses CustomWriter, synchronous execution, TimeSpan to UInt16, param=1, isValid=false.

Main: matching test-case names pass on CoreCLR, physical tvOS 26.2; the assembly finishes successfully. Some parameterized cases share display names.

[FAIL] System.Xml.WrappedReaderTests.WrappedReaderTest.RunTests(testCase: 105376: ReadBase64 runs into an Overflow)
System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

The PR subsequently aborts while reporting results:

Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

App terminated due to signal 6.

PR console | Main test results

CPU usage: EnvironmentTests.TestCpuUsage

PR: iOS CoreCLR and NativeAOT, System.Runtime.Extensions.Tests, one failure per job. Main: the test passes on both runtimes on physical tvOS 26.2.

CoreCLR:

Unexpected Zero privileged time while having user time: 00:00:00.4796340

NativeAOT:

Unexpected Zero privileged time while having user time: 00:00:00.5820430

PR consoles: CoreCLR, NativeAOT. Main results: CoreCLR, NativeAOT.

2. PR failures not executed by the inspected main-device jobs

These are not established pass-on-main cases: the corresponding tests are omitted, filtered, skipped, or have no equivalent runtime configuration.

Unix file modes: SetUser is lost

PR: iOS CoreCLR and NativeAOT, 48 failed cases each in System.IO.FileSystem.Tests. Affects SetThenGet in DirectoryInfo_GetSetUnixFileMode, FileInfo_GetSetUnixFileMode, File_GetSetUnixFileMode, and File_GetSetUnixFileMode_SafeFileHandle; every failure differs by the missing SetUser bit.

Assert.Equal() Failure: Values differ
Expected: SetUser
Actual:   None

Main tvOS: these parameter values are explicitly filtered out, not passing.

PR consoles: CoreCLR, NativeAOT.

NetworkInformation: timeout on both runtimes, plus two CoreCLR failures

PR: iOS CoreCLR and NativeAOT, System.Net.NetworkInformation.Functional.Tests:

Run timed out after 4500 seconds

CoreCLR additionally logs IPGlobalProperties_AccessAllMethods_NoErrors failing in BsdIPGlobalProperties.GetTcpConnections:

System.OverflowException : Arithmetic operation resulted in an overflow.

And IPGlobalProperties_UdpListeners_Succeed(address: 127.0.0.1) failing in GetActiveUdpListeners:

System.PlatformNotSupportedException : The information requested is unavailable on the current platform.

Main tvOS: the assembly is excluded for device hangs under #60713. That documents an existing coverage gap, not proof of the exact same underlying hang.

PR consoles: CoreCLR, NativeAOT.

Tracing: three distinct signatures

PR: iOS CoreCLR, System.Diagnostics.Tracing.Tests, eight cases:

• Six EventPipeListener cases: TestsWrite.Test_Write_T; TestsWriteEvent.Test_WriteEvent and Test_WriteEvent_ByteArray, each with useSelfDescribingEvents=false/true; and Test_WriteEvent_ComplexData_SelfDescribing. All fail with ServerNotAvailableException; the diagnostic message includes:

Unable to connect to Process 4034. The total length of the diagnostic socket path may exceed 108 characters.

• TestsManifestNegative.Test_GenerateManifest_InvalidEventSources:

Assert.Throws() Failure: No exception was thrown
Expected: typeof(System.ArgumentException)

• TestsUserErrors.Test_BadEventSource_MismatchedIds:

Expected: "EventSourceMessage"
Actual:   "Event1"

Main tvOS: the assembly is excluded. PR console

Reflection.Emit: save-validation hits an app-bundle permission error

PR: iOS CoreCLR, AssemblySaveAssemblyBuilderTests.PersistedAssemblyBuilder_SaveValidations:

Assert.Throws() Failure: Exception type was not an exact match
Expected: typeof(System.InvalidOperationException)
Actual:   typeof(System.UnauthorizedAccessException)

Access to System.Reflection.Emit.Tests.app/File is denied, with inner error Operation not permitted. Main tvOS: the assembly is excluded. PR console

NativeAOT runtime: UnwindFpBleedTest

PR: iOS and tvOS NativeAOT, one failure each in Exceptions:

Assert.Equal() Failure: Values differ
Expected: 100
Actual:   1

Main: no equivalent Apple NativeAOT runtime-test job was found. The CoreCLR tvOS counterpart is explicitly Skip in the result XML; its green Exceptions work item has 0 passed / 2 skipped tests.

PR consoles: iOS, tvOS. Main CoreCLR console.

NativeAOT runtime: b425314.Mutate.TestEntryPoint

PR: tvOS NativeAOT, Regression_d, one failed test containing two scenario timeouts; expected return 100, actual 101:

Timeout: The `CastClass' scenario exceeded the time limit (180s) without reaching the basic work thresholds.
Timeout: The `Unbox' scenario exceeded the time limit (180s) without reaching the basic work thresholds.

Main: no equivalent tvOS NativeAOT runtime-test job was found. The CoreCLR tvOS counterpart is Skip; its green Regression_d work item has 0 passed / 2 skipped tests.

PR console | Main CoreCLR console

NativeAOT runtime: missing inputs before app launch

PR: MobileSmokeTest and nativeaot on both iOS and tvOS, four failed work items. Representative iOS errors:

Unhandled exception. System.CommandLine.CommandLineException: No files matching /tmp/helix/working/A4E8091A/w/B3ED0A15/e/publish/MobileSmokeTest.dll

Unhandled exception. System.CommandLine.CommandLineException: No files matching /tmp/helix/working/A4E8091A/w/B4310A23/e/publish/nativeaot.dll

ILC exits 134, followed by missing-provisioning/signing errors and wrapper exit 21. These are pre-launch failures, not four executed test failures. The inspected main device jobs do not run these runtime-test work items.

PR consoles: iOS MobileSmokeTest, iOS nativeaot, tvOS MobileSmokeTest, tvOS nativeaot.

Note

This analysis was prepared with GitHub Copilot.

cc: @BrzVlad @vitek-karas do you think any of these are severe enough to investigate before .NET 11? For context this comes from testing CoreCLR iOS mobile on tvOS/iOS27 with XCode 27. There is couple of System.OutOfMemoryException which I haven't seen on main so far.

[vitek-karas]

The OOMs - worth investigating but I don't think these are blocking for 11. They're probably intentionally large tests (looks like it).

Priviledged time - this behavior is already allowed on tvOS #105322 - so possible it also affects iOS we just didn't see it yet.

FileSystem - The test is disabled on tvOS for a while already - so this could be Apple specific behavior. Not very important.

Networking - this is probably interesting - we should understand what's going on at least - so I'd say we should investigate this.

EventSource - weird behavior especially the last failure - worth investigation, but lower priority I'd say (not many uses for event pipe). We should see if event pipe in general works, if so, then a corner case failure is not high-pri.

Reflection - surprised we even run these tests on mobile - writing assemblies is not exactly a typical thing for a mobile app to do - low priority (also feels like a test issue, not product issue)

NativeAOT regression test failure - Interesting failures - floating point math in one case - but doesn't feel important enough for 11 servicing.

NativeAOT command line failures - this is at build time - weird. Worth investigating.

[matouskozak]

Took a deeper look into the networking failures and System.Net.NetworkInformation.Functional.Tests hangs on iOS 27 devices under both CoreCLR and NativeAOT until the 4,500 second timeout. Device logs show repeated sandbox denials for net.inet.udp.pcblist (CoreCLR) and net.inet.tcp.pcblist (NativeAOT). Any ideas @simonrozsival ?

I think we don't run these tests on main but I'm not sure why. I found #60713 but that seems to be a very old issue and possibly not relevant anymore.