Test failure: baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd

[v-wenyuxu]

Failed in: runtime-coreclr outerloop 20240721.3

Failed tests:

coreclr osx x64 Checked @ OSX.1200.Amd64.Open
    - baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd

Error message:

 
cmdLine:/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/baseservices-exceptions/../unhandled/unhandledTester/unhandledTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 7/21/2024 10:34:29 PM, end: 7/21/2024 10:44:29 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/AFE209A8/w/B882099F/uploads/unhandled/unhandledTester/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/AFE209A8/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true unhandledTester.dll ''
"Unhandled exception. System.Exception: Test"
"   at TestUnhandledException.Program.Main(String[] args)"
""
Test process unhandled.dll with argument main exited
Test process exited with expected error code and produced expected output
"Unhandled exception. System.DllNotFoundException: Unable to load shared library '__internal' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: "
"dlopen(/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/__internal.dylib, 0x0001): tried: '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/__internal.dylib' (no such file)"
"dlopen(/tmp/helix/working/AFE209A8/p/__internal.dylib, 0x0001): tried: '/tmp/helix/working/AFE209A8/p/__internal.dylib' (no such file), '/private/tmp/helix/working/AFE209A8/p/__internal.dylib' (no such file)"
"dlopen(__internal.dylib, 0x0001): tried: '__internal.dylib' (no such file), '/usr/local/lib/__internal.dylib' (no such file), '/usr/lib/__internal.dylib' (no such file), '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/__internal.dylib' (no such file)"
"dlopen(/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/lib__internal.dylib, 0x0001): tried: '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/lib__internal.dylib' (no such file)"
"dlopen(/tmp/helix/working/AFE209A8/p/lib__internal.dylib, 0x0001): tried: '/tmp/helix/working/AFE209A8/p/lib__internal.dylib' (no such file), '/private/tmp/helix/working/AFE209A8/p/lib__internal.dylib' (no such file)"
"dlopen(lib__internal.dylib, 0x0001): tried: 'lib__internal.dylib' (no such file), '/usr/local/lib/lib__internal.dylib' (no such file), '/usr/lib/lib__internal.dylib' (no such file), '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/lib__internal.dylib' (no such file)"
"dlopen(/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/__internal, 0x0001): tried: '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/__internal' (no such file)"
"dlopen(/tmp/helix/working/AFE209A8/p/__internal, 0x0001): tried: '/tmp/helix/working/AFE209A8/p/__internal' (no such file), '/private/tmp/helix/working/AFE209A8/p/__internal' (no such file)"
"dlopen(__internal, 0x0001): tried: '__internal' (no such file), '/usr/local/lib/__internal' (no such file), '/usr/lib/__internal' (no such file), '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/__internal' (no such file)"
"dlopen(/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/lib__internal, 0x0001): tried: '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/unhandled/unhandledTester/lib__internal' (no such file)"
"dlopen(/tmp/helix/working/AFE209A8/p/lib__internal, 0x0001): tried: '/tmp/helix/working/AFE209A8/p/lib__internal' (no such file), '/private/tmp/helix/working/AFE209A8/p/lib__internal' (no such file)"
"dlopen(lib__internal, 0x0001): tried: 'lib__internal' (no such file), '/usr/l

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor139|0_140(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2024-07-22T06:32:39.884Z	runtime-coreclr crossgen2 outerloop	Checked-no_tiered_compilation	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"BuildNumber":"20240721.1","System.StageName":"Build","Project":"public","System.JobAttempt":"1","System.JobName":"__default","AzurePipelinesTestRunId":"18969038","Reason":"Schedule","DefinitionId":"134","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","DefinitionName":"runtime-coreclr crossgen2 outerloop","BuildId":"749562","operatingSystem":"Windows.11.Arm64.Open","architecture":"arm64","System.PhaseAttempt":"1","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","System.StageAttempt":"1","configuration":"Checked-no_tiered_compilation","CollectionUri":"https://dev.azure.com/dnceng-public/"}	Schedule
2024-07-22T06:32:39.884Z	runtime-coreclr crossgen2 outerloop	Checked-no_tiered_compilation	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"BuildNumber":"20240721.1","System.StageName":"Build","Project":"public","System.JobAttempt":"1","System.JobName":"__default","AzurePipelinesTestRunId":"18969038","Reason":"Schedule","DefinitionId":"134","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","DefinitionName":"runtime-coreclr crossgen2 outerloop","BuildId":"749562","operatingSystem":"Windows.11.Arm64.Open","architecture":"arm64","System.PhaseAttempt":"1","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","System.StageAttempt":"1","configuration":"Checked-no_tiered_compilation","CollectionUri":"https://dev.azure.com/dnceng-public/"}	Schedule
2024-07-21T22:16:19.843Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_disabler2r_jitstress2	osx.1200.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.StageName":"Build","architecture":"arm64","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","System.JobName":"__default","BuildId":"749452","CollectionUri":"https://dev.azure.com/dnceng-public/","Project":"public","System.StageAttempt":"1","AzurePipelinesTestRunId":"18964482","System.JobAttempt":"1","configuration":"Checked-gcstress0xc_disabler2r_jitstress2","BuildNumber":"20240721.1","Reason":"Schedule","DefinitionName":"runtime-coreclr gcstress-extra","operatingSystem":"OSX.1200.ARM64.Open","DefinitionId":"113","System.PhaseAttempt":"1","System.PhaseName":"run_test_p1__osx_arm64_checked"}	Schedule
2024-07-20T22:18:40.955Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_jitstress2	osx.1200.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"DefinitionId":"113","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","Reason":"Schedule","System.JobAttempt":"1","System.PhaseAttempt":"1","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","Project":"public","BuildNumber":"20240720.1","System.StageName":"Build","configuration":"Checked-gcstress0xc_jitstress2","BuildId":"749040","System.StageAttempt":"1","AzurePipelinesTestRunId":"18949352","DefinitionName":"runtime-coreclr gcstress-extra","architecture":"arm64","CollectionUri":"https://dev.azure.com/dnceng-public/"}	Schedule
2024-07-20T06:35:44.084Z	runtime-coreclr crossgen2 outerloop	Checked	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"DefinitionName":"runtime-coreclr crossgen2 outerloop","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","architecture":"arm64","configuration":"Checked","Reason":"Schedule","System.StageName":"Build","System.PhaseAttempt":"1","System.StageAttempt":"1","DefinitionId":"134","System.JobName":"__default","BuildNumber":"20240719.1","operatingSystem":"Windows.11.Arm64.Open","BuildId":"748543","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","AzurePipelinesTestRunId":"18933048","Project":"public"}	Schedule
2024-07-14T07:27:47.664Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress2	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.JobAttempt":"1","BuildId":"740354","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","System.StageAttempt":"1","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_jitstress2","System.PhaseAttempt":"1","Project":"public","BuildNumber":"20240713.1","architecture":"arm64","System.JobName":"__default","Reason":"Schedule","AzurePipelinesTestRunId":"18682830","DefinitionId":"141","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageName":"Build","operatingSystem":"Windows.11.Arm64.Open","DefinitionName":"runtime-coreclr crossgen2-composite gcstress"}	Schedule

More failures

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2024-06-21T06:31:49.233Z	runtime-coreclr crossgen2 outerloop	Checked	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"BuildId":"715192","DefinitionId":"134","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","System.PhaseAttempt":"1","BuildNumber":"20240620.1","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","Reason":"Schedule","System.StageName":"Build","configuration":"Checked","System.JobAttempt":"1","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","operatingSystem":"Windows.11.Arm64.Open","AzurePipelinesTestRunId":"17894852","DefinitionName":"runtime-coreclr crossgen2 outerloop","System.JobName":"__default","Project":"public","architecture":"arm64"}	Schedule
2024-06-20T06:36:22.042Z	runtime-coreclr crossgen2 outerloop	Checked-no_tiered_compilation	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.JobName":"__default","System.StageAttempt":"1","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","operatingSystem":"Windows.11.Arm64.Open","architecture":"arm64","CollectionUri":"https://dev.azure.com/dnceng-public/","DefinitionName":"runtime-coreclr crossgen2 outerloop","BuildId":"713731","Reason":"Schedule","AzurePipelinesTestRunId":"17854254","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","Project":"public","System.PhaseAttempt":"1","DefinitionId":"134","System.StageName":"Build","System.JobAttempt":"1","configuration":"Checked-no_tiered_compilation","BuildNumber":"20240619.1"}	Schedule
2024-06-17T07:33:18.599Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitminopts_heapverify1	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.PhaseAttempt":"1","configuration":"Checked-gcstress0xc_jitminopts_heapverify1","Project":"public","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","Reason":"Schedule","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","BuildId":"709719","System.JobName":"__default","BuildNumber":"20240616.1","architecture":"arm64","DefinitionId":"141","System.StageAttempt":"1","System.JobAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","AzurePipelinesTestRunId":"17727758"}	Schedule
2024-05-19T07:25:26.68Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress2	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","BuildId":"680500","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","configuration":"Checked-gcstress0xc_jitstress2","System.StageName":"Build","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","BuildNumber":"20240518.1","architecture":"arm64","AzurePipelinesTestRunId":"16879150","Project":"public","System.PhaseAttempt":"1","DefinitionId":"141","Reason":"Schedule","System.StageAttempt":"1","System.JobAttempt":"1","System.JobName":"__default","DefinitionName":"runtime-coreclr crossgen2-composite gcstress"}	Schedule
2024-05-19T07:25:26.68Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress2	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","BuildId":"680500","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","configuration":"Checked-gcstress0xc_jitstress2","System.StageName":"Build","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","BuildNumber":"20240518.1","architecture":"arm64","AzurePipelinesTestRunId":"16879150","Project":"public","System.PhaseAttempt":"1","DefinitionId":"141","Reason":"Schedule","System.StageAttempt":"1","System.JobAttempt":"1","System.JobName":"__default","DefinitionName":"runtime-coreclr crossgen2-composite gcstress"}	Schedule
2024-04-21T06:30:58.345Z	runtime-coreclr crossgen2 outerloop	Checked	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.StageName":"Build","System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2 outerloop","DefinitionId":"134","Project":"public","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobAttempt":"1","System.PhaseAttempt":"1","architecture":"arm64","AzurePipelinesTestRunId":"16093914","BuildId":"650980","Reason":"Schedule","BuildNumber":"20240420.1","configuration":"Checked","System.JobName":"__default","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","operatingSystem":"Windows.11.Arm64.Open"}	Schedule
2024-04-19T06:33:51.091Z	runtime-coreclr crossgen2 outerloop	Checked	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"operatingSystem":"Windows.11.Arm64.Open","BuildId":"649001","CollectionUri":"https://dev.azure.com/dnceng-public/","Project":"public","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64","System.StageName":"Build","DefinitionName":"runtime-coreclr crossgen2 outerloop","System.PhaseAttempt":"1","DefinitionId":"134","BuildNumber":"20240418.1","AzurePipelinesTestRunId":"16042244","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","configuration":"Checked","System.StageAttempt":"1","System.JobName":"__default"}	Schedule
2024-04-14T07:35:44.843Z	runtime-coreclr crossgen2-composite gcstress	Checked-heapverify1	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"architecture":"arm64","BuildNumber":"20240413.1","configuration":"Checked-heapverify1","AzurePipelinesTestRunId":"15849442","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobAttempt":"1","Reason":"Schedule","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","DefinitionId":"141","System.JobName":"__default","System.StageName":"Build","System.PhaseAttempt":"1","Project":"public","BuildId":"642632","operatingSystem":"Windows.11.Arm64.Open","System.StageAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","DefinitionName":"runtime-coreclr crossgen2-composite gcstress"}	Schedule
2024-04-14T07:35:44.843Z	runtime-coreclr crossgen2-composite gcstress	Checked-heapverify1	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"architecture":"arm64","BuildNumber":"20240413.1","configuration":"Checked-heapverify1","AzurePipelinesTestRunId":"15849442","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobAttempt":"1","Reason":"Schedule","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","DefinitionId":"141","System.JobName":"__default","System.StageName":"Build","System.PhaseAttempt":"1","Project":"public","BuildId":"642632","operatingSystem":"Windows.11.Arm64.Open","System.StageAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","DefinitionName":"runtime-coreclr crossgen2-composite gcstress"}	Schedule
2024-04-05T06:40:22.997Z	runtime-coreclr crossgen2 outerloop	Checked	windows.11.arm64.open	arm64	baseservices/exceptions/unhandled/unhandledTester/unhandledTester.cmd	Failed	{"System.StageName":"Build","DefinitionId":"134","BuildId":"631352","operatingSystem":"Windows.11.Arm64.Open","CollectionUri":"https://dev.azure.com/dnceng-public/","AzurePipelinesTestRunId":"15513776","BuildNumber":"20240404.1","System.JobName":"__default","System.PhaseName":"run_test_p1_R2R_Composite_windows_arm64_checked","System.JobAttempt":"1","configuration":"Checked","Project":"public","System.JobId":"f1be4a6e-76af-52cb-72c1-7edccd0341b3","System.StageAttempt":"1","architecture":"arm64","DefinitionName":"runtime-coreclr crossgen2 outerloop","System.PhaseAttempt":"1","Reason":"Schedule"}	Schedule

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[janvorli]

Duplicate of #100495

[janvorli]

Part is also a mix of issues unrelated to the unhandled exception test - in one of the runs on macOS (arm64 2024-07-21T22:16:19.843Z runtime-coreclr gcstress-extra Checked-gcstress0xc_disabler2r_jitstress2), huge number of the tests in the exception handling suire are crashing due to what looks like GC holes.

[janvorli]

It seems that the issues here are caused by #104336 that was reverted yesterday. I am closing this issue for now.