[API Proposal]: Expose TLS signature algorithm families for certificate selection

[wfurt]

Background and motivation

During a post-quantum cryptography rollout, servers need to keep classical certificates available while selectively offering ML-DSA certificates to compatible clients. Today, ServerOptionsSelectionCallback can select a certificate based on SNI and advertised TLS versions, but SslClientHelloInfo does not indicate which signature algorithm families the client advertised. Selecting an ML-DSA certificate unconditionally breaks classical clients, while always selecting RSA or ECDSA prevents gradual PQC deployment on the same endpoint.

This proposal exposes a compact family-level view of the ClientHello signature_algorithms extension. It is intended as an allocation-free pre-filter for choosing among configured RSA, ECDSA, EdDSA, ML-DSA, and SLH-DSA certificate candidates. It deliberately does not claim that every certificate or chain in a reported family is compatible; the application must still apply its certificate policy.

TLS signature algorithms are independent from TLS supported groups. Negotiated supported-group reporting, including hybrid ML-KEM groups, is tracked separately by #132239. Additional PQC SslStream test coverage is tracked by #134227.

The existing workaround is to use the experimental TlsSession raw ClientHello API and implement a TLS parser. Normal SslStream, Kestrel, and QUIC callback users cannot otherwise inspect this information.

API proposal

namespace System.Net.Security;

[Flags]
public enum TlsSignatureAlgorithmFamilies
{
    None = 0,
    Rsa = 1,
    ECDsa = 2,
    EdDsa = 4,
    MLDsa = 8,
    SlhDsa = 16,
}

public readonly partial struct SslClientHelloInfo
{
    // EXISTING
    // public SslClientHelloInfo(string serverName, SslProtocols sslProtocols);

    public SslClientHelloInfo(
        string serverName,
        SslProtocols sslProtocols,
        TlsSignatureAlgorithmFamilies signatureAlgorithmFamilies);

    public TlsSignatureAlgorithmFamilies SignatureAlgorithmFamilies { get; }
}

Prototype: wfurt@21c91ef

API usage

await sslStream.AuthenticateAsServerAsync(
    (stream, hello, state, cancellationToken) =>
    {
        SslStreamCertificateContext certificate =
            (hello.SignatureAlgorithmFamilies & TlsSignatureAlgorithmFamilies.MLDsa) != 0
                ? mlDsaCertificate
                : (hello.SignatureAlgorithmFamilies & TlsSignatureAlgorithmFamilies.ECDsa) != 0
                    ? ecdsaCertificate
                    : rsaCertificate;

        return ValueTask.FromResult(new SslServerAuthenticationOptions
        {
            ServerCertificateContext = certificate,
        });
    },
    state: null,
    cancellationToken);

The server controls preference ordering. MLDsa means that the client advertised at least one recognized ML-DSA signature scheme; it does not mean that every ML-DSA parameter set or certificate chain is compatible.

Alternative designs

Expose exact signature schemes

public ReadOnlyMemory<TlsSignatureScheme> SignatureAlgorithms { get; }
public ReadOnlyMemory<TlsSignatureScheme> CertificateSignatureAlgorithms { get; }

This is lossless and can distinguish ML-DSA parameter sets, RSA-PSS key forms, ECDSA curves, certificate-chain signature constraints, client ordering, and unknown schemes. However, it exposes substantially more TLS protocol detail, requires callers to implement family mapping and signature_algorithms_cert fallback semantics, and likely requires per-handshake storage for the lists. The proposed bitmask addresses the common certificate-family rollout policy without adding collection allocations. Exact scheme exposure can be added later if applications need to select among parameter sets rather than among families.

Expose both signature_algorithms and signature_algorithms_cert as family masks

signature_algorithms_cert constrains signatures within the certificate chain, not simply the leaf key family. For example, an ECDSA leaf can be issued by an RSA CA. Reducing that extension to a second family mask could encourage callers to incorrectly require the leaf family in both masks. The proposal instead documents that the family mask is only a broad leaf/private-key compatibility signal.

Expose raw ClientHello bytes

This is already available through the experimental TlsSession API. Requiring every SslStream or Kestrel application to implement a security-sensitive TLS parser is not appropriate for the common selection scenario.

Automatically choose from multiple certificates

Platform TLS providers differ in certificate-selection policy and do not receive equivalent candidate sets from SslStream today. Reusing ServerOptionsSelectionCallback keeps server preference explicit and consistent across Schannel, OpenSSL, Apple TLS, and MsQuic.

Include supported groups

Supported groups describe key exchange, while signature algorithms describe certificate authentication. This proposal does not add group requirements or enforcement. Negotiated-group reporting is covered by #132239.

Risks

• Family-level information is intentionally lossy. Applications choosing a specific parameter set must retain a classical fallback or use a future exact-scheme API.
• Unknown-only, absent, malformed, or unavailable signature information is represented as None.
• SslClientHelloInfo is shared by SslStream, experimental TlsSession, and QuicListener; the prototype populates the property in all three paths.
• The change is additive. The existing constructor remains and initializes the new property to None.

Usage in dotnet/runtime

Updated in the prototype

File	Description
System.Net.Security/SslStream.IO.cs	Populates the family mask for ServerOptionsSelectionCallback.
System.Net.Security/TlsSession.cs	Populates ClientHelloInfo for deferred server options.
System.Net.Quic/QuicListener.cs	Parses MsQuic's ClientHello crypto buffer before invoking ConnectionOptionsCallback.

No additional runtime adoption sites select among multiple certificates today.

Validation

• System.Net.Security functional tests: 5,077 total, 0 failed.
• System.Net.Security unit tests: 134 total, 0 failed.
• System.Net.Quic functional tests available on this host: 3 total, 0 failed.
• System.Net.Quic source and functional test projects build for all configured target frameworks.
• The deterministic TLS-handshake-message parser test passes locally.
• The live QUIC callback test is included; its class is disabled when MsQuic is unavailable.

Note

This proposal and prototype were created with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
12 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[wfurt]

Closing this draft because the requested deliverable is the API proposal itself, before committing to an implementation. The proposal is now tracked in #134630. A prototype can follow after the API design discussion.

Note

This comment was created with GitHub Copilot.