"OverflowError: string longer than 2147483647 bytes" when trying requests.put

[EB123]

Hi,

I'm trying to upload a file that weight about 3GB and I'm getting the following error:
"OverflowError: string longer than 2147483647 bytes"

If I understand correctly it seems like there's a 2GB limit? didnt manage to find any reference to such limiation or how to bypass it (if possible).

The code i'm using is:

datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    args = myfile.read()
resp = requests.put(url, data=args, verify=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 995, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1029, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 991, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 844, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 820, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/ssl.py", line 234, in sendall
    v = self.send(data[count:])
  File "/usr/local/lib/python2.7/ssl.py", line 203, in send
    v = self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

For smaller files this code works fine for me.

[demianbrecht]

Rather than reading the entire file and sending it across in a single request, would it be possible for you to use chunked transfer encoding? http://docs.python-requests.org/en/latest/user/advanced/#chunk-encoded-requests

[Lukasa]

This limitation is in httplib. You can easily avoid it by slightly changing your code:

datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    resp = requests.put(url, data=myfile, verify=False)

[sigmavirus24]

@Lukasa that's inaccurate. The traceback comes from an SSL wrapped socket. This has nothing to do with httplib from what I can see. OverflowErrors are raised when a value is larger than the underlying C integer size allowed. This can be seen if you call len(something_larger_than_four_gb) on a 32 bit system.

[eriktews]

Unfortunately it looks like it cannot be avoided when you do a POST request with several headers. Then the file (or the files) is always read completely.

It would be great when this could be avoided in requests since I often have to send files which are longer than the available main memory on the system.

[StewPoll]

I'm just gonna chime in.

If you're trying to send files via the Web that are larger than your
systems memory can handle, HTTP probably isn't the best protocol to do this
via.

On Tue, 11 Oct 2016, 5:54 AM Erik Tews notifications@github.com wrote:

Unfortunately it looks like it cannot be avoided when you do a POST
request with several headers. Then the file (or the files) is always read
completely.

It would be great when this could be avoided in requests since I often
have to send files which are longer than the available main memory on the
system.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kennethreitz/requests/issues/2717#issuecomment-252729478,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGk_7XIpx0UDRPXDuoopI7jDSfAlurgLks5qypf7gaJpZM4FqFGb
.

[eriktews]

That's true, but I don't get to decide on the protocol and endpoint.

Doing the request with curl works fine, and as a workaround, I'm currently printing a curl command to STDOUT so that the user can launch it.

[sigmavirus24]

@eriktews can you share how you're doing the upload? There are ways to stream uploads (like Lukasa's comment shows). Is there a reason you cannot do that (if you are not already trying that)? Also, can you provide your actual traceback?

[eriktews]

So the comment from Lukasa seems to work when you are uploading a single file, then you can do a streaming upload. But I have to do a normal post request with several variables in the data part and the file as a part of a multipart upload.

There is an API documentation at https://canvas.instructure.com/doc/api/file.file_uploads.html which shows a curl command in the "Step 2" section. Basically I wanna replicate that call with the requests package in streaming mode.

I don't have the traceback at the moment, but when I get it, I will post it here.

[sigmavirus24]

Have you tried using the MultipartEncoder from the requests-toolbelt? That allows you to stream multipart/form-data uploads like doing an upload of a single file. It was written specifically for this use case and was added to our docs.

[eriktews]

No, but that looks like right what I need. I will give it a try and see whether this works. I didn't know about the toolbelt package at all, so maybe you should reference it in the normal requests package documentation.

[sigmavirus24]

so maybe you should reference it in the normal requests package documentation.

It is :)

[adamn]

@Lukasa 's method does not work - even with httplib off the signing still happens for the transport itself. In my case I have a 2GB+ POST request (not a file, just POST data). This is for an elasticsearch bulk update. The endpoint only has HTTPS so I'm working through other solutions now.

Traceback (most recent call last):
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/elasticsearch/connection/http_requests.py", line 75, in perform_request
    timeout=timeout or self.timeout, verify=False)
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 356, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 891, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

[Lukasa]

Sorry, can you demonstrate your code please?