From c336dbfbdeccee934974a14fb0d89fca24a20460 Mon Sep 17 00:00:00 2001 From: Ivan Diaz Date: Tue, 17 Jan 2023 17:01:35 -0800 Subject: [PATCH 1/6] Added the documentation regarding the new libraries test modes. --- docs/workflow/testing/libraries/testing.md | 61 ++++++++++++++++++---- 1 file changed, 52 insertions(+), 9 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index cb17c48b6cc7a1..4d90f52af44f27 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -5,62 +5,67 @@ These example commands automate the test run and all pre-requisite build steps in a single command from a clean enlistment. - Run all tests - Builds clr in release, libs+tests in debug: + ``` build.cmd/sh -subset clr+libs+libs.tests -test -rc Release ``` - Run all tests - Builds Mono in release, libs+tests in debug: + ``` build.cmd/sh -subset mono+libs+libs.tests -test -rc Release ``` - Run all tests - Build Mono and libs for x86 architecture in debug (choosing debug for runtime will run very slowly): + ``` build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 ``` ## Partial Build and Test Runs -Doing full build and test runs takes a long time and is very inefficient if you need to iterate on a change. -For greater control and efficiency individual parts of the build + testing workflow can be run in isolation. -See the [Building instructions](../../building/libraries/README.md) for more info on build options. +Doing full build and test runs takes a long time and is very inefficient if you need to iterate on a change. For greater control and efficiency individual parts of the build + testing workflow can be run in isolation. See the [Building instructions](../../building/libraries/README.md) for more info on build options. ### Test Run Pre-requisites -Before any tests can run we need a complete build to run them on. This requires building (1) a runtime, and -(2) all the libraries. Examples: + +Before any tests can run we need a complete build to run them on. This requires building (1) a runtime, and (2) all the libraries. Examples: - Build release clr + debug libraries + ``` build.cmd/sh -subset clr+libs -rc Release ``` - Build release mono + debug libraries + ``` build.cmd/sh -subset mono+libs -rc Release ``` -Building the `libs` subset or any of individual library projects automatically copies product binaries into the testhost folder -in the bin directory. This is where the tests will load the binaries from during the run. However System.Private.CorLib is an -exception - the build does not automatically copy it to the testhost folder. If you [rebuild System.Private.CoreLib](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) you must also build the `libs.pretest` subset to ensure S.P.C is copied before running tests. +Building the `libs` subset or any of individual library projects automatically copies product binaries into the testhost folder in the bin directory. This is where the tests will load the binaries from during the run. However System.Private.CorLib is an exception - the build does not automatically copy it to the testhost folder. If you [rebuild System.Private.CoreLib](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) you must also build the `libs.pretest` subset to ensure S.P.C is copied before running tests. ### Running tests for all libraries - Build and run all tests in release configuration. + ``` build.cmd/sh -subset libs.tests -test -c Release ``` - Build the tests without running them + ``` build.cmd/sh -subset libs.tests ``` - Run the tests without building them + ``` build.cmd/sh -subset libs.tests -test -testnobuild ``` - The following example shows how to pass extra msbuild properties to ignore tests ignored in CI. + ``` build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI ``` @@ -68,17 +73,20 @@ build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI ### Running tests for a single library The easiest (and recommended) way to build and run the tests for a specific library, is to invoke the `Test` target on that library: + ```cmd cd src\libraries\System.Collections.Immutable\tests dotnet build /t:Test ``` It is possible to pass parameters to the underlying xunit runner via the `XUnitOptions` parameter, e.g.: + ```cmd dotnet build /t:Test /p:XUnitOptions="-class Test.ClassUnderTests" ``` Which is very useful when you want to run tests as `x86` on a `x64` machine: + ```cmd dotnet build /t:Test /p:TargetArchitecture=x86 ``` @@ -88,6 +96,7 @@ There may be multiple projects in some directories so you may need to specify th ### Running a single test on the command line To quickly run or debug a single test from the command line, set the XunitMethodName property, e.g.: + ```cmd dotnet build /t:Test /p:XunitMethodName={FullyQualifiedNamespace}.{ClassName}.{MethodName} ``` @@ -95,10 +104,44 @@ dotnet build /t:Test /p:XunitMethodName={FullyQualifiedNamespace}.{ClassName}.{M ### Running outer loop tests To run all tests, including "outer loop" tests (which are typically slower and in some test suites less reliable, but which are more comprehensive): + ```cmd dotnet build /t:Test /p:Outerloop=true ``` ### Running tests on a different target framework -Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks. \ No newline at end of file +Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks. + +### Running tests in custom compilation modes + +There are several custom compilation modes for tests. These are enabled by setting a switch during the configuration. These switches are described in the following table: + +| Mode | Description | Prerequisite Subsets | +| -------------- | --------------------------------------------------------- | -------------------- | +| TestSingleFile | Test using the single file compilation mode | libs+clr | +| TestNativeAot | Test by compiling using NativeAOT | libs+clr.aot | +| TestReadyToRun | Test compilation of the tests/libraries into R2R binaries | libs+clr | + +To run a test in a specific mode, simply build the tests after building the prerequisite subsets, and specify the test mode in the command-line. For example, to use the _TestReadyToRun_ mode in Release configuration: + +```bash +dotnet build -c Release -t:Test -p:TestReadyToRun=true +``` + + +It is important to highlight that these tests do not use the standard XUnit test runner. Instead, they run with the [SingleFileTestRunner](/src/libraries/Common/tests/SingleFileTestRunner/SingleFileTestRunner.cs). The set of available commands is listed here: + +- `-xml` +- `-notrait` +- `-class` +- `-class-` +- `-noclass` +- `-method` +- `-method-` +- `-nomethod` +- `-namespace` +- `-namespace-` +- `-nonamespace` +- `-parallel` + From 7f71c59e24a00ec7d5dbd09a2a6ed988ddb48f53 Mon Sep 17 00:00:00 2001 From: Ivan Diaz Date: Fri, 20 Jan 2023 11:31:18 -0800 Subject: [PATCH 2/6] Enable the fundamentals behind libraries tests compiled via crossgen2. --- eng/testing/tests.props | 1 + eng/testing/tests.singlefile.targets | 44 ++++++------ .../SingleFileTestRunner.cs | 68 ++++++++++++++++++- ...m.Diagnostics.FileVersionInfo.Tests.csproj | 3 + .../System.Diagnostics.Process.Tests.csproj | 3 + 5 files changed, 95 insertions(+), 24 deletions(-) diff --git a/eng/testing/tests.props b/eng/testing/tests.props index 6f976dcf9fb793..ab58a76a58c3b5 100644 --- a/eng/testing/tests.props +++ b/eng/testing/tests.props @@ -8,6 +8,7 @@ true $(MSBuildThisFileDirectory)ILLinkDescriptors\ true + true diff --git a/eng/testing/tests.singlefile.targets b/eng/testing/tests.singlefile.targets index 582932087ac1bd..347041f209f40d 100644 --- a/eng/testing/tests.singlefile.targets +++ b/eng/testing/tests.singlefile.targets @@ -2,8 +2,6 @@ Exe - $(DefineConstants);SINGLE_FILE_TEST_RUNNER - $([MSBuild]::NormalizeDirectory('$(OutDir)', 'publish')) $([MSBuild]::NormalizePath('$(BundleDir)', '$(RunScriptOutputName)')) $(OutputRid) @@ -18,9 +16,16 @@ true + + true + false + + $(CoreCLRILCompilerDir) $(CoreCLRCrossILCompilerDir) + clang-15 + clang-9 $(ROOTFS_DIR) $(CoreCLRILCompilerDir)netstandard/ILCompiler.Build.Tasks.dll $(CoreCLRAotSdkDir) @@ -34,6 +39,10 @@ true + + $(DefineConstants);SINGLE_FILE_TEST_RUN + + @@ -82,26 +91,17 @@ - - - - clang - - - - - + + + $([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'crossgen2'))crossgen2$(ExeSuffix) + + + + + + + values)) { noTraits.Add(traitKeyValue[0], values = new List()); } + values.Add(traitKeyValue[1]); + i++; } + if (args[i].Equals("-xml", StringComparison.OrdinalIgnoreCase)) { - xmlResultFileName=args[i + 1].Trim(); + xmlResultFileName = args[i + 1].Trim(); + i++; + } + + if (args[i].Equals("-class", StringComparison.OrdinalIgnoreCase)) + { + filters.IncludedClasses.Add(args[i + 1].Trim()); + i++; + } + + if (args[i].Equals("-noclass", StringComparison.OrdinalIgnoreCase) || + args[i].Equals("-class-", StringComparison.OrdinalIgnoreCase)) + { + filters.ExcludedClasses.Add(args[i + 1].Trim()); + i++; + } + + if (args[i].Equals("-method", StringComparison.OrdinalIgnoreCase)) + { + filters.IncludedMethods.Add(args[i + 1].Trim()); + i++; + } + + if (args[i].Equals("-nomethod", StringComparison.OrdinalIgnoreCase) || + args[i].Equals("-method-", StringComparison.OrdinalIgnoreCase)) + { + filters.ExcludedMethods.Add(args[i + 1].Trim()); + i++; + } + + if (args[i].Equals("-namespace", StringComparison.OrdinalIgnoreCase)) + { + filters.IncludedNamespaces.Add(args[i + 1].Trim()); + i++; + } + + if (args[i].Equals("-nonamespace", StringComparison.OrdinalIgnoreCase) || + args[i].Equals("-namespace-", StringComparison.OrdinalIgnoreCase)) + { + filters.ExcludedNamespaces.Add(args[i + 1].Trim()); + i++; + } + + if (args[i].Equals("-parallel", StringComparison.OrdinalIgnoreCase)) + { + string parallelismArg = args[i + 1].Trim().ToLower(); + var (parallelizeAssemblies, parallelizeTestCollections) = parallelismArg switch + { + "all" => (true, true), + "assemblies" => (true, false), + "collections" => (false, true), + "none" => (false, false), + _ => throw new ArgumentException($"Unknown parallelism option '{parallelismArg}'.") + }; + + assemblyConfig.ParallelizeAssembly = parallelizeAssemblies; + assemblyConfig.ParallelizeTestCollections = parallelizeTestCollections; + i++; } } diff --git a/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj b/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj index 8a18d381c0b226..b9873d355e5f6d 100644 --- a/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj +++ b/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj @@ -50,5 +50,8 @@ PreserveNewest + + + diff --git a/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj b/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj index 35a748804d1bd2..be1ce9e524d32a 100644 --- a/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj +++ b/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj @@ -72,5 +72,8 @@ Content PreserveNewest + + + From 3f2de8f23b258630126d284ad7c2da4540cede24 Mon Sep 17 00:00:00 2001 From: Ivan Diaz Date: Fri, 20 Jan 2023 13:26:57 -0800 Subject: [PATCH 3/6] Fixed a typo with SINGLE_FILE_TEST_RUNNER. --- eng/testing/tests.singlefile.targets | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eng/testing/tests.singlefile.targets b/eng/testing/tests.singlefile.targets index 347041f209f40d..aed7c5f2c534df 100644 --- a/eng/testing/tests.singlefile.targets +++ b/eng/testing/tests.singlefile.targets @@ -40,7 +40,7 @@ - $(DefineConstants);SINGLE_FILE_TEST_RUN + $(DefineConstants);SINGLE_FILE_TEST_RUNNER From 8b3df1d976b04c4a8928813cc69c4dbd5a413a33 Mon Sep 17 00:00:00 2001 From: Ivan Diaz Date: Mon, 23 Jan 2023 13:59:02 -0800 Subject: [PATCH 4/6] Addressed review comments. --- eng/testing/tests.singlefile.targets | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/eng/testing/tests.singlefile.targets b/eng/testing/tests.singlefile.targets index aed7c5f2c534df..c40a71e6bd3b52 100644 --- a/eng/testing/tests.singlefile.targets +++ b/eng/testing/tests.singlefile.targets @@ -24,8 +24,6 @@ $(CoreCLRILCompilerDir) $(CoreCLRCrossILCompilerDir) - clang-15 - clang-9 $(ROOTFS_DIR) $(CoreCLRILCompilerDir)netstandard/ILCompiler.Build.Tasks.dll $(CoreCLRAotSdkDir) @@ -39,7 +37,7 @@ true - + $(DefineConstants);SINGLE_FILE_TEST_RUNNER @@ -93,12 +91,8 @@ - - $([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'crossgen2'))crossgen2$(ExeSuffix) - - - + @@ -109,4 +103,19 @@ AfterTargets="Build" DependsOnTargets="Publish;ArchiveTests" /> + + + clang + + + + + + + From 7ac2212226f6b500d19f39cae80460617f42964f Mon Sep 17 00:00:00 2001 From: Ivan Diaz Date: Tue, 24 Jan 2023 09:56:48 -0800 Subject: [PATCH 5/6] Restored an accidentally deleted comment. --- eng/testing/tests.singlefile.targets | 35 +++++++++++++++++----------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/eng/testing/tests.singlefile.targets b/eng/testing/tests.singlefile.targets index c40a71e6bd3b52..295cc97608efc7 100644 --- a/eng/testing/tests.singlefile.targets +++ b/eng/testing/tests.singlefile.targets @@ -89,20 +89,13 @@ - - - - - - - - - - + @@ -118,4 +111,18 @@ + + + + + + + + + + From 6ae3ed7560a17056850a77d27aac00abfae1ff47 Mon Sep 17 00:00:00 2001 From: Ivan Diaz Date: Fri, 17 Feb 2023 20:06:45 -0800 Subject: [PATCH 6/6] Fixed wrong Crossgen2 path and added comment with link to an important tracking SDK bug. --- eng/testing/tests.singlefile.targets | 6 +++++- .../System.Diagnostics.FileVersionInfo.Tests.csproj | 2 +- .../tests/System.Diagnostics.Process.Tests.csproj | 2 +- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/eng/testing/tests.singlefile.targets b/eng/testing/tests.singlefile.targets index 295cc97608efc7..cb745fc7662f07 100644 --- a/eng/testing/tests.singlefile.targets +++ b/eng/testing/tests.singlefile.targets @@ -113,8 +113,12 @@ + + $(CoreCLRCrossgen2Dir)crossgen2$(ExeSuffix) + + - + diff --git a/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj b/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj index b9873d355e5f6d..12807ef4fc413c 100644 --- a/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj +++ b/src/libraries/System.Diagnostics.FileVersionInfo/tests/System.Diagnostics.FileVersionInfo.Tests/System.Diagnostics.FileVersionInfo.Tests.csproj @@ -51,7 +51,7 @@ - + diff --git a/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj b/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj index be1ce9e524d32a..c3f01adc3bc673 100644 --- a/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj +++ b/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj @@ -73,7 +73,7 @@ PreserveNewest - +