HTTP Compression - Request Compression

[EatonZ]

Background and Motivation

There is currently no support for HTTP request compression. Response decompression support has been present and maintained for many years now, but request compression has been oddly absent. Some web servers support request compression and this would be useful when sending large objects over slow connections. Adding opt-in request compression would be useful for programmers looking to optimize their network communication code in certain situations.

Proposed API (updated by @iremyux)

namespace System.Net.Http
{
  public enum CompressionMethod 
  {
    GZip = 0x1,
    Deflate = 0x2,
    Brotli = 0x4,
    Zstandard = 0x8
  }

  public sealed partial class CompressedContent : System.Net.Http.HttpContent
  {
    public CompressedContent(System.Net.Http.HttpContent content, CompressionMethod compressionMethod) { }
    public CompressedContent(System.Net.Http.HttpContent content, CompressionMethod compressionMethod, System.IO.Compression.CompressionLevel compressionLevel) { }
  }
}

Usage Examples (updated by @iremyux)

var request = new HttpRequestMessage(RequestMethod, URI);
{
   Content = new CompressedContent(json, CompressionMethod.Brotli)
};
var response = await httpClient.SendAsync(request);

var content = new StreamContent(largeFileStream);
request.Content = new CompressedContent(content, CompressionMethod.GZip, CompressionLevel.SmallestSize);

Alternative Designs (updated by @iremyux)

• The underlying compression algorithms offer other parameters beyond CompressionLevel : a leaveOpen flag which is not meaningful in this case and an algorithm-specific options object ( ZLibCompressionOptions ,  BrotliCompressionOptions ,  ZstandardCompressionOptions ). We decided the advanced per-algorithm tuning can be discussed later if there is demand for it.
• We considered numbering CompressionMethod sequentially. This was rejected in favor of aligning the values with the existing DecompressionMethods flags so the two compression-related enums share a consistent, memorable mapping for the same algorithms.
• A CompressionHandler (similar to the existing DecompressionHandler) that automatically compresses requests and handles server negotiation was discussed.

Risks

• 415 Unsupported Media Type: Servers that don't support compressed requests will reject them. Users must handle this themselves.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

There is currently no support for HTTP request compression. Response decompression support has been present and maintained for many years now, but request compression has been oddly absent. Some web servers support request compression and this would be useful when sending large objects over slow connections. Adding opt-in request compression would be useful for programmers looking to optimize their network communication code in certain situations.

Proposed API

Add a CompressedContent class. An example of an existing implementation is here. It needs a refresh for 2021 (Brotli support, etc) but otherwise showcases support is possible, and easy to implement.

The CompressedContent name also seems the best to use - it goes along with DecompressedContent.

Usage Examples

A snippet of code from my own project, which uses a modified version of the WebAPIContrib CompressedContent. content is one of ByteArrayContent, FormUrlEncodedContent, StreamContent, StringContent, or any other class inheriting HttpContent.

var request = new HttpRequestMessage(RequestMethod, URI);
request.Content = RequestDetails.RequestCompression switch
{
    DecompressionMethods.GZip => new CompressedContent(content, "gzip"),
    DecompressionMethods.Deflate => new CompressedContent(content, "deflate"),
    DecompressionMethods.Brotli => new CompressedContent(content, "br"),
    DecompressionMethods.All => throw new Exception("EasyHttp.HttpExecuteAsync: Only 1 request compression method can be specified."),
    _ => content
};
var response = await httpClient.SendAsync(request);

Alternative Designs

N/A at this time.

Risks

Maybe CRIME and BREACH?

Author:	EatonZ
Assignees:	-
Labels:	api-suggestion, area-System.Net.Http, untriaged
Milestone:	-

[karelz]

I assume this is achievable today by setting the headers and manually compressing the content. It probably also allows better flexibility in tweaking compression algorithm parameters (size vs. time tradeoff).
I think it will boil down to how often is the scenario needed.

We have similar new compression API on WebSocket: #31088 (comment)

[EatonZ]

@karelz Yes, it can be done today if you copy that example CompressedContent class into your project. I just feel it should be built-in so you don't need to do that.

Compression params would be easy - just add CompressionLevel to the constructor to let the user specify their preference for speed or size.

I am working on a modernized version of CompressedContent for my own project. If the .NET team would approve adding it, I'd be happy to put in a PR, but I'd need to know where this class would best fit.

[karelz]

@EatonZ we first need agreement on Networking team this is something valuable for enough customers (in our triage) and then we will need official API approval (see API review process, before we would be ready for a PR.
Expect it will take some time - also to let more folks to comment and not to rush it without considering alternatives / need.

[EatonZ]

Here's a complete example of how it could look:

public sealed class CompressedContent : HttpContent
{
    private readonly HttpContent originalContent;
    private readonly string contentEncoding;
    private readonly CompressionLevel compressionLevel;

    public CompressedContent(HttpContent originalContent, string contentEncoding, CompressionLevel compressionLevel)
    {
        this.originalContent = OriginalContent;
        this.contentEncoding = ContentEncoding;
        this.compressionLevel = CompressionLevel;

        foreach (var (key, value) in originalContent.Headers) Headers.TryAddWithoutValidation(key, value);
        Headers.ContentEncoding.Add(contentEncoding);
    }

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        Stream compressedStream = contentEncoding switch
        {
            "br" => new BrotliStream(stream, compressionLevel, true),
            "gzip" => new GZipStream(stream, compressionLevel, true),
            "deflate" => new ZLibStream(stream, compressionLevel, true) //Yes, ZLibStream over DeflateStream. See this note: https://github.com/dotnet/runtime/blob/7ab969c84ef05ba948c0075392716ce335b47744/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/DecompressionHandler.cs#L231
        };
        await using (compressedStream.ConfigureAwait(false))
        {
            await originalContent.CopyToAsync(compressedStream).ConfigureAwait(false);
        }
    }

    protected override bool TryComputeLength(out long length)
    {
        length = 0;
        return false;
    }
}

[karelz]

Triage: May be worthwhile, but only 1 customer asking for it + there is simple workaround. Let's wait if more people upvote the top post before adding yet another API.

[samsosa]

Most RESTful API implementations support implicit Content-Encoding: gzip.
We use it for all such kind API services, as it significantly reduces the amount
of JSON-Payload transferred. (... by up to 90 %)

Currently, we compress the payload before sending it using GZipStream and
set the appropriate request header.
We would not reject a clean, reasonable API within .NET. 😉

[EatonZ]

@karelz Any interest in getting this into .NET 6?