ArgumentOutOfRangeException at System.Net.Security.SslStream.ProcessBlob

[liiri]

Description

I'm getting recurring but non consistent exceptions in couple of my environments running the same code under the Docker image mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim

This only happened in environments where I have an SSL certificate provisioned by certbot, but I don't know if its related. Certificate is signed and up to date.

Reproduction Steps

not consistent

Expected behavior

No errors on Kestrel level

Actual behavior

Microsoft.AspNetCore.Server.Kestrel: Unhandled exception while processing 0HMDG00D520FK.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize)
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1.ExecuteAsync()

Regression?

Did not observe this in dotnet 5 or core 3.1 that I run beforehand

Known Workarounds

No response

Configuration

Dotnet 6.0.100
OS is Ubuntu 18.0.4 x86_64 , running Docker image mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim`

Other information

No response

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I'm getting recurring but non consistent exceptions in couple of my environments running the same code under the Docker image mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim

This only happened in environments where I have an SSL certificate provisioned by certbot, but I don't know if its related. Certificate is signed and up to date.

Reproduction Steps

not consistent

Expected behavior

No errors on Kestrel level

Actual behavior

Microsoft.AspNetCore.Server.Kestrel: Unhandled exception while processing 0HMDG00D520FK.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize)
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1.ExecuteAsync()

Regression?

Did not observe this in dotnet 5 or core 3.1 that I run beforehand

Known Workarounds

No response

Configuration

Dotnet 6.0.100
OS is Ubuntu 18.0.4 x86_64 , running Docker image mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim`

Other information

No response

Author:	liiri
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[wfurt]

Can you get packet captures @liiri and step in with debugger to get value of frameSize. There were some changes in 6.0 to process more frames at once when possible so there can be new bugs. However there is not enough information to investigate. I cannot say for sure but the certificate seems unlikely, unless it is somewhat unusual - like very big. This part of the code really deals with IO and TLS frames so it is more matter of size and framing.

[liiri]

I will try to get more information, would have if it was simple. This never occurred in an environment where a debugger is handy to attach to.

Can you offer any workaround to catch this exception or retry the blob receive?

[wfurt]

This is inbound connection, right? Any idea if this happens with particular client?

One more thought:
Can you possibly try https://docs.microsoft.com/en-us/dotnet/api/system.appdomain.firstchanceexception?view=net-6.0
And than possibly "Environment.FailFast() if e is System.ArgumentOutOfRangeException" That would create core dump if the system is properly configured. (needs coredump_filter=0x3f https://github.com/dotnet/diagnostics/blob/main/documentation/debugging-coredump.md)
There are other ways how to get dump but if we can get one from when this happen, we can likely solve the mystery.

[liiri]

This is inbound connection, from browser, most likely Chrome

[karelz]

@liiri did you get chance to try it out?

This issue has been marked needs more info since it may be missing important information. Please refer to our contribution guidelines for tips on how to report issues effectively.