104080 custom rsapss salt length

[henning-krause]

Added support for custom PSS salt length for RSA based signature operations. Resolves #104080.

With this PR, I have extended the RsaSignaturePadding class with a custom salt length as discussed in #104080. I've also updated the CmsSigner and the CertificateRequest class to support this.

CoseSigner did not need functionality updates, because the spec does not support custom salt length. I have added checks to prevent this to be configured on the CoseSigner.

Tests where either added or updated to test the new functionality.

All RSA implementations which support this (MAC doesn't seem to support this) were updated.

[Copilot]

Pull Request Overview

This pull request adds support for custom PSS salt length for RSA-based signature operations. The implementation extends the RSASignaturePadding class with a new CreatePss(int saltLength) method and updates related cryptographic components to utilize custom salt lengths.

Key changes include:

• Extended RSASignaturePadding to support custom PSS salt lengths with new constants and factory method
• Updated all RSA implementation backends (OpenSSL, CNG, BCrypt, etc.) to handle custom salt lengths
• Added comprehensive test coverage for various salt length scenarios
• Updated high-level APIs like CmsSigner and CertificateRequest to support the new functionality

Reviewed Changes

Copilot reviewed 40 out of 41 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey_rsa.h	Added pssSaltLength parameter to RSA sign/verify functions
src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey_rsa.c	Implemented custom salt length support in OpenSSL backend
src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSASignaturePadding.cs	Added CreatePss method, PssSaltLength property, and related constants
src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPssX509SignatureGenerator.cs	Updated to use custom salt lengths from padding configuration
src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs	Modified PSS encoding/verification to accept custom salt lengths
src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.DigestInfo.cs	Added salt length calculation logic and moved digest info constants
src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/RSAPssX509SignatureGeneratorTests.cs	Added comprehensive tests for custom PSS salt lengths
src/libraries/System.Security.Cryptography.Pkcs/src/System/Security/Cryptography/Pkcs/CmsSigner.cs	Updated validation to accept any PSS mode padding
src/libraries/System.Security.Cryptography.Cose/src/System/Security/Cryptography/Cose/CoseSigner.cs	Added validation to prevent custom salt lengths in COSE

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs:300

• The size check if (emLen < 2 + hLen + sLen) can overflow when sLen is a large user-supplied value (via CreatePss), which can bypass the guard and lead to large allocations/invalid slicing later. Use subtraction (or long/checked arithmetic) to make the guard overflow-safe.

            // 3.  if emLen < hLen + sLen + 2, encoding error.
            //
            // sLen = hLen in this implementation.

            if (emLen < 2 + hLen + sLen)

src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs:393

• This guard uses hLen + sLen + 2, which can overflow for large sLen values and incorrectly skip returning false. That can result in negative indexes later in VerifyPss. Make the check overflow-safe (e.g., via subtraction) before proceeding.

            // 3. If emLen < hLen + sLen + 2, output "inconsistent" and stop.
            if (emLen < hLen + sLen + 2)
            {
                return false;
            }

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 3 comments.

[henning-krause]

@bartonjs All comments should now be fixed... Can you review my last changes? Thanks :-)

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 2 comments.

[github-actions]

Workflow state for the Holistic Review Orchestrator.

{
  "version": 5,
  "last_dispatched_commit": "d0d88a24fb7edde8e3b0826d606d91309e18ef35",
  "last_dispatched_base_ref": "main",
  "last_dispatched_base_sha": "17bbb207a46d1b2ae3741baa62cb88a98e2e9847",
  "last_reviewed_commit": "d0d88a24fb7edde8e3b0826d606d91309e18ef35",
  "last_reviewed_base_ref": "main",
  "last_reviewed_base_sha": "17bbb207a46d1b2ae3741baa62cb88a98e2e9847",
  "last_recorded_worker_run_id": "29684886999",
  "review_attempt_commit": "",
  "review_attempt_base_ref": "",
  "review_attempt_count": 0,
  "max_review_attempts": 5,
  "review_history_format": "holistic-review-disclosure-v1",
  "review_history": [
    {
      "commit": "d0d88a24fb7edde8e3b0826d606d91309e18ef35",
      "review_id": 4730678743
    }
  ]
}

[github-actions]

Holistic Review

Motivation: Justified. Issue #104080 is api-approved and documents a real interoperability gap: .NET hard-codes the PSS salt length to the hash length, so it cannot produce or validate PSS signatures that use a different salt length (e.g. RSA_PSS_SALTLEN_MAX or an explicit value), which other stacks and standards require.

Approach: Sound and consistent with the codebase. RSASignaturePadding.CreatePss(int) plus the PssSaltLengthIsHashLength/PssSaltLengthMax sentinels are stored on the immutable padding object; every backend (OpenSSL, CNG, BCrypt, Android, Apple, plus the managed RsaPaddingProcessor) resolves the sentinels through a single new RsaPaddingProcessor.CalculatePssSaltLength helper, and platforms that cannot honor a custom length (Apple, COSE) throw rather than silently ignore it. This directly follows the implementation checklist bartonjs posted on the approved issue (encode/verify, certificate PssParams, PssParamsAsn rehydration, CMS/COSE, and tests).

Summary: ⚠️ Needs Human Review. I found no blocking defects and verified the public surface matches the approved API exactly, but this is a large, security-sensitive cryptographic change spanning five platform backends plus CMS/COSE/X.509 encoding, and the area owner (bartonjs) has an in-flight review (previously CHANGES_REQUESTED). A crypto-domain maintainer should confirm the salt-length math and per-platform semantics before merge.

---

Detailed Findings

✅ API Approval — Matches approved shape

The new public surface in ref/System.Security.Cryptography.cs (const int PssSaltLengthIsHashLength = -1, const int PssSaltLengthMax = -2, int PssSaltLength { get; }, static RSASignaturePadding CreatePss(int saltLength)) is byte-for-byte identical to the shape approved by bartonjs (who applied the api-approved label on 2025-06-03) in the issue comment. Namespaces, type, member names, and signatures all match; no extra or missing public API.

✅ Cross-platform completeness — sentinels resolved centrally, unsupported paths throw

Sentinel values (-1/-2) are never leaked to native layers or the managed encoder: each caller resolves them via RsaPaddingProcessor.CalculatePssSaltLength(padding.PssSaltLength, KeySize, hashAlgorithm) before use (RSAOpenSsl, RSACng.SignVerify, RSABCrypt, RSAAndroid, RSAAppleCrypto). Backends that cannot honor a non-hLen salt reject it: Apple's ValidatePaddingParameters (called from both TrySignHash and VerifyHash) throws Cryptography_CustomPssSaltLengthNotSupported, and COSE's GetRSAAlgorithm throws because the spec mandates sLen==hLen. The EncodePss/VerifyPss emLen < 2 + hLen + sLen guard plus the PssSaltLengthTooLarge test protect against over-large explicit salts, and EncodePss moves the salt buffer to the heap when sLen > 128 to avoid an unbounded stackalloc.

✅ Test coverage — thorough and interop-anchored

Tests exercise explicit lengths, PssSaltLengthMax, and hLen equivalence, use externally-generated OpenSSL signatures as known-answer vectors, assert cross-verification between implicit Pss and explicit CreatePss(hLen), verify wrong salt lengths fail, and cover CMS/COSE/CertificateRequest. Coverage is gated by AreCustomSaltLengthsSupportedWithPss (excludes Apple), matching the platform capabilities.

💡 Stale comment in RsaPaddingProcessor.EncodePss

The comment // sLen = hLen in this implementation. (RsaPaddingProcessor.cs, in EncodePss near the emLen < 2 + hLen + sLen check) is now inaccurate since sLen comes from the caller-supplied saltLength. It sits on an unchanged line so I did not attach an inline comment; consider removing it to avoid confusion. Non-blocking.

Note

This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.

Generated by Holistic Review · 275.2 AIC · ⌖ 10.7 AIC · ⊞ 10K

[henning-krause]

@bartonjs Can you take a look at my last changes?

[Copilot]

🟡 Changes recommended

Four moderate correctness and resource-handling issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (4)

src/libraries/Common/src/System/Security/Cryptography/Asn1/PssParamsAsn.manual.cs:54

• SaltLength is read from untrusted ASN.1, but this branch passes every non-default value directly to CreatePss. That accepts the internal -1/-2 sentinels, so a CMS or PKCS#10 PSS parameter with a negative salt-length INTEGER is interpreted as hash-length/max instead of rejected, even though the ASN.1 field is an actual non-negative salt length. Reject negative SaltLength before constructing the padding.

                RSASignaturePadding.CreatePss(SaltLength);

src/libraries/Common/src/System/Security/Cryptography/RSAAndroid.cs:686

• EncodePss now rejects custom salt lengths, but this call is outside a try/finally. On Android, a too-large (or otherwise invalid) salt length throws after encodedRented is acquired, so the pool buffer is not returned; repeated invalid signing attempts can steadily increase allocation pressure. Keep the encoding and primitive call inside a try/finally that returns encodedRented.

                    RsaPaddingProcessor.EncodePss(hashAlgorithm, hash, encodedBytes, KeySize, RsaPaddingProcessor.CalculatePssSaltLength(padding.PssSaltLength, KeySize, hashAlgorithm));

src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs:390

• The same int overflow lets an oversized salt length pass this verification guard. For example, with sLen == int.MaxValue, the sum becomes negative and the method later indexes dbMask with a negative computed position, throwing instead of returning false for an invalid signature. Use a non-overflowing comparison such as sLen > emLen - hLen - 2.

            if (emLen < hLen + sLen + 2)

src/libraries/System.Security.Cryptography.Pkcs/src/System/Security/Cryptography/Pkcs/CmsSignature.RSA.cs:459

• For SHA-1, the shared PSS ASN.1 definition uses AlgorithmIdentifier values whose parameters are an explicit NULL (PssParamsAsn.xml:22-27). This new custom-salt path constructs both the hash and MGF1 hash identifiers without parameters, so a non-default salt causes explicit [0]/[1] fields with absent SHA-1 parameters; RFC 4055 consumers can reject that encoding. Populate the SHA-1 NULL parameters, or omit the default hash and MGF fields when they are SHA-1.

                    HashAlgorithm = new ValueAlgorithmIdentifierAsn { Algorithm = digestOid },

• Files reviewed: 38/39 changed files
• Comments generated: 1
• Review effort level: Lite