OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest on CentOS Stream 9

[omajid]

Description

While building .NET 6 using .NET 6 in in CentOS Stream 9 (aka RHEL 9 in-development), I am running into error : Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest

error : Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest 
error :    at Interop.Crypto.RsaSignHash(SafeEvpPKeyHandle pkey, RSASignaturePaddingMode paddingMode, IntPtr digestAlgorithm, ReadOnlySpan`1 hash, Span`1 destination) 
error :    at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature) 
error :    at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) 
error :    at Microsoft.CodeAnalysis.SigningUtilities.CalculateRsaSignature(IEnumerable`1 content, RSAParameters privateKey) 
error :    at Microsoft.CodeAnalysis.DesktopStrongNameProvider.<>c__DisplayClass12_0.<SignBuilder>b__0(IEnumerable`1 content) 
error :    at System.Reflection.PortableExecutable.PEBuilder.Sign(BlobBuilder peImage, Blob strongNameSignatureFixup, Func`2 signatureProvider) 
error :    at System.Reflection.PortableExecutable.ManagedPEBuilder.Sign(BlobBuilder peImage, Func`2 signatureProvider) 
error :    at Microsoft.CodeAnalysis.DesktopStrongNameProvider.SignBuilder(ExtendedPEBuilder peBuilder, BlobBuilder peBlob, RSAParameters privateKey) 
error :    at Microsoft.Cci.PeWriter.WritePeToStream(EmitContext context, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, Boolean metadataOnly, Boolean isDeterministic, Boolean emitTestCoverageData, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.Compilation.SerializePeToStream(CommonPEModuleBuilder moduleBeingBuilt, DiagnosticBag metadataDiagnostics, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getMetadataPeStreamOpt, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, RebuildData rebuildData, Boolean metadataOnly, Boolean includePrivateMembers, Boolean isDeterministic, Boolean emitTestCoverageData, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream(CommonPEModuleBuilder moduleBeingBuilt, EmitStreamProvider peStreamProvider, EmitStreamProvider metadataPEStreamProvider, EmitStreamProvider pdbStreamProvider, RebuildData rebuildData, Func`2 testSymWriterFactory, DiagnosticBag diagnostics, EmitOptions emitOptions, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit(TouchedFileLogger touchedFilesLogger, Compilation& compilation, ImmutableArray`1 analyzers, ImmutableArray`1 generators, ImmutableArray`1 additionalTextFiles, AnalyzerConfigSet analyzerConfigSet, ImmutableArray`1 sourceFileAnalyzerConfigOptions, ImmutableArray`1 embeddedTexts, DiagnosticBag diagnostics, CancellationToken cancellationToken, CancellationTokenSource& analyzerCts, Boolean& reportAnalyzer, AnalyzerDriver& analyzerDriver) 
error :    at Microsoft.CodeAnalysis.CommonCompiler.RunCore(TextWriter consoleOutput, ErrorLogger errorLogger, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.CommonCompiler.Run(TextWriter consoleOutput, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.<>c__DisplayClass1_0.<Run>b__0(TextWriter tw) 
error :    at Microsoft.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output[T](Boolean utf8Output, TextWriter textWriter, Func`2 func) 
error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader) 
error :    at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter, String pipeName) 
error :    at Microsoft.CodeAnalysis.CommandLine.BuildClient.Run(IEnumerable`1 arguments, RequestLanguage language, CompileFunc compileFunc, CompileOnServerFunc compileOnServerFunc) 
error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.MainCore(String[] args) 

Full log is here: https://centos.softwarefactory-project.io/zuul/t/centos/build/b00c0fe1895c4e1487350108a41214da

Could this be caused by Disable SHA1 signature creation and verification by default?

Reproduction Steps

It seems like building runtime in a CentOS Stream 9 container should be enough to trigger the bug. Will test and update this step later.

Expected behavior

I can build .NET itself.

Actual behavior

I can't build .NET, because the compiler needs signing (via OpenSSL) to work.

Regression?

It's a regression somewhere. Most likely it's not .NET itself that's to blame, because this same source code (no changes) built a few days ago.

Known Workarounds

I am testing if an export OPENSSL_ENABLE_SHA1_SIGNATURES=1 will work around the issue.

Edit: Confirmed. It does make the build move past this particular error.

Configuration

• .NET 6 source-build using the 6.0.102 tag of dotnet/installer
• CentOS Stream 9 on x86_64

Other information

No response

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

While building .NET 6 using .NET 6 in in CentOS Stream 9 (aka RHEL 9 in-development), I am running into error : Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest

error : Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest 
error :    at Interop.Crypto.RsaSignHash(SafeEvpPKeyHandle pkey, RSASignaturePaddingMode paddingMode, IntPtr digestAlgorithm, ReadOnlySpan`1 hash, Span`1 destination) 
error :    at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature) 
error :    at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) 
error :    at Microsoft.CodeAnalysis.SigningUtilities.CalculateRsaSignature(IEnumerable`1 content, RSAParameters privateKey) 
error :    at Microsoft.CodeAnalysis.DesktopStrongNameProvider.<>c__DisplayClass12_0.<SignBuilder>b__0(IEnumerable`1 content) 
error :    at System.Reflection.PortableExecutable.PEBuilder.Sign(BlobBuilder peImage, Blob strongNameSignatureFixup, Func`2 signatureProvider) 
error :    at System.Reflection.PortableExecutable.ManagedPEBuilder.Sign(BlobBuilder peImage, Func`2 signatureProvider) 
error :    at Microsoft.CodeAnalysis.DesktopStrongNameProvider.SignBuilder(ExtendedPEBuilder peBuilder, BlobBuilder peBlob, RSAParameters privateKey) 
error :    at Microsoft.Cci.PeWriter.WritePeToStream(EmitContext context, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, Boolean metadataOnly, Boolean isDeterministic, Boolean emitTestCoverageData, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.Compilation.SerializePeToStream(CommonPEModuleBuilder moduleBeingBuilt, DiagnosticBag metadataDiagnostics, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getMetadataPeStreamOpt, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, RebuildData rebuildData, Boolean metadataOnly, Boolean includePrivateMembers, Boolean isDeterministic, Boolean emitTestCoverageData, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream(CommonPEModuleBuilder moduleBeingBuilt, EmitStreamProvider peStreamProvider, EmitStreamProvider metadataPEStreamProvider, EmitStreamProvider pdbStreamProvider, RebuildData rebuildData, Func`2 testSymWriterFactory, DiagnosticBag diagnostics, EmitOptions emitOptions, Nullable`1 privateKeyOpt, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit(TouchedFileLogger touchedFilesLogger, Compilation& compilation, ImmutableArray`1 analyzers, ImmutableArray`1 generators, ImmutableArray`1 additionalTextFiles, AnalyzerConfigSet analyzerConfigSet, ImmutableArray`1 sourceFileAnalyzerConfigOptions, ImmutableArray`1 embeddedTexts, DiagnosticBag diagnostics, CancellationToken cancellationToken, CancellationTokenSource& analyzerCts, Boolean& reportAnalyzer, AnalyzerDriver& analyzerDriver) 
error :    at Microsoft.CodeAnalysis.CommonCompiler.RunCore(TextWriter consoleOutput, ErrorLogger errorLogger, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.CommonCompiler.Run(TextWriter consoleOutput, CancellationToken cancellationToken) 
error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.<>c__DisplayClass1_0.<Run>b__0(TextWriter tw) 
error :    at Microsoft.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output[T](Boolean utf8Output, TextWriter textWriter, Func`2 func) 
error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader) 
error :    at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter, String pipeName) 
error :    at Microsoft.CodeAnalysis.CommandLine.BuildClient.Run(IEnumerable`1 arguments, RequestLanguage language, CompileFunc compileFunc, CompileOnServerFunc compileOnServerFunc) 
error :    at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.MainCore(String[] args) 

Full log is here: https://centos.softwarefactory-project.io/zuul/t/centos/build/b00c0fe1895c4e1487350108a41214da

Could this be caused by Disable SHA1 signature creation and verification by default?

Reproduction Steps

It seems like building runtime in a CentOS Stream 9 container should be enough to trigger the bug. Will test and update this step later.

Expected behavior

I can build .NET itself.

Actual behavior

I can't build .NET, because the compiler needs signing (via OpenSSL) to work.

Regression?

It's a regression somewhere. Most likely it's not .NET itself that's to blame, because this same source code (no changes) built a few days ago.

Known Workarounds

I am testing if an export OPENSSL_ENABLE_SHA1_SIGNATURES=1 will work around the issue.

Configuration

• .NET 6 source-build using the 6.0.102 tag of dotnet/installer
• CentOS Stream 9 on x86_64

Other information

No response

Author:	omajid
Assignees:	-
Labels:	area-System.Security, untriaged
Milestone:	-

[vcsjones]

Could this be caused by Disable SHA1 signature creation and verification by default?

Seems likely. It's trying to produce a strong name signature and that uses RSA-SHA1:

https://github.com/dotnet/roslyn/blob/315c2e149ba7889b0937d872274c33fcbfe9af5f/src/Compilers/Core/Portable/PEWriter/SigningUtilities.cs#L22

[omajid]

Thanks for the pointer.

I tried a build with a workaround to enable SHA1 and it seems to be working. It does seem to be down to SHA1 signature creation being disabled.

How difficult would be be to switch to something slightly better for a future release of .NET? I have no knowledge of Windows, but I found a blog post that suggests SHA256 and SHA512 are also supported for signing?

[vcsjones]

How difficult would be be to switch to something slightly better for a future release of .NET? I have no knowledge of Windows,

My understanding is that this RSA-SHA1 is being used for .NET's strong naming functionality, not Windows Authenticode. The Authenticode signatures are likely already SHA256.

I don't know what it would take to move the ecosystem past RSA-SHA1 for the purpose that it is being used. Perhaps @bartonjs or @GrabYourPitchforks know, directly or indirectly.

[vcsjones]

For what it's worth, this breaks any .NET project that uses strong name signing, not just building .NET itself.

To reproduce, create a new C# project. Add this somewhere:

[assembly:System.Reflection.AssemblyKeyFile("testkey.snk")]

Grab a strong name key. You can use testkey.snk in this repository. Put it next to the csproj.

It will fail to build with the same error.

[jaredpar]

My understanding is that this RSA-SHA1 is being used for .NET's strong naming functionality, not Windows Authenticode

That is correct for this scenario / stack trace

[bartonjs]

ECMA-335 only describes RSA(SSA)-SHA1(-PKCS1v1_5):

The Strong Name (SN) signing process uses standard hash and cipher algorithms for Strong name
signing. An SHA-1 hash over most of the PE file is generated. That hash value is RSA-signed with the
SN private key. For verification purposes the public key is stored into the PE file as well as the signed
hash value.

There's a concept of "extended strong name", which is a SHA256 countersignature on the SHA-1 signature (similar to how Authenticode bootstrapped SHA256); but I don't think that there's a Strong Name scenario that avoids SHA-1.

While Strong Name isn't important for .NET Core / .NET 5+, I can imagine RHEL users building components (such as the Roslyn C# compiler) that need to support Strong Name for .NET Framework-based customers (either a .NET Standard library or even building directly for a .NET Framework TFM).

I don't know how dotnet build works (e.g. does it launch a second process?), but perhaps there's a way for the RHEL build of the SDK to always assert OPENSSL_ENABLE_SHA1_SIGNATURES=1 in the environment when using dotnet build?

[vcsjones]

Additionally, I did a quick search to see if there are other things in the dotnet org that use RSA+SHA1 and strong name signing is the only meaningful result.

https://cs.github.com/?scopeName=All+repos&scope=&q=org%3Adotnet+%2F%28Sign%7CVerify%29%28Hash%7CData%29%5C%28.*HashAlgorithmName.SHA1%2F+NOT+path%3A%2Ftests%3F%2F

[omajid]

I reported this to the roslyn folks directly as well, at dotnet/roslyn#59880, particularly because I am not sure if roslyn's usage of SHA1 qualifies as "security" or not.

[jaredpar]

@omajid I responded to the issue in dotnet/roslyn. In summary though this isn't a decision the roslyn compiler is making. The .NET platform requires SHA-1 be used for strong naming. The roslyn compiler, and every other .NET compiler that supports strong naming, is simply implementing the specification here. There are no other options besides SHA-1 unless the underlying format is changed.

[omajid]

Thanks, @jaredpar

The .NET platform requires SHA-1 be used for strong naming

What should the platform do when (RSA+)SHA1 is not available on the OS (because of policies like FIPS, or general system hardening)? Should it hard-fail like it does now? Should it have a separate this-is-not-security implementation of RSA+SHA1?

Is there a way to tell OpenSSL "we want to use RSA+SHA1, but not for security, we promise"?

There are no other options besides SHA-1 unless the underlying format is changed.

Given how SHA1 is being phased out everywhere, what would it take to update the underlying format?

[omajid]

Perhaps there's a way for the RHEL build of the SDK to always assert OPENSSL_ENABLE_SHA1_SIGNATURES=1 in the environment when using dotnet build?

I slept over this one, and I wonder if this will allow using RSA+SHA1 even in contexts where the OS really doesn't want it to be used, eg for TLS when downloading nuget packages (that happens as part of dotnet build, right?).

And it wont help when FIPS enablement permanently disables RSA+SHA1.

[vcsjones]

Is there a way to tell OpenSSL "we want to use RSA+SHA1, but not for security, we promise"?

Correct me if I am wrong, but this is a patch on top of OpenSSL developed by RedHat, not something that is in OpenSSL upstream, correct? It would then be up to RedHat to implement a work around, if one needs to be provided.

Based on the changeset, it looks like developers can:

1. Set rh-allow-sha1-signatures in openssl.cnf to permanently re-allow SHA1 signatures.
2. Use the environment variable, as you have already described.

Even if RedHat provided an option for "pinky swear it's not for security", it would require the runtime detect this custom API from RedHat, and somehow expose this option to roslyn and fsharp compilers.

Alternatively, the roslyn and fsharp compilers could consider using a private implementation, say one written in managed code, of RSA and not use OpenSSL at all. I don't know if that is at all feasible or not.