Python Requests - ChunkedEncodingError(e) - requests.iter_lines

Question

I'm getting a ChunkedEncodingError(e) using Python requests. I'm using the following to rip down JSON:

r = requests.get(url, headers=auth, stream=True)

And the iterating over each line, using the carriage return as a delimiter, which is how this API distinguishes between distinct JSON events.

for d in r.iter_lines(delimiter="\n"): d += "\n" sock.send(d)

I'm delimiting on the carriage return and then adding it back in as the endpoint I'm pushing the logs to actually expects a carriage return at the end of each event also. This seems to work for roughly 100k log files. When I try to make a larger call I'll get this following thrown:

for d in r.iter_lines(delimiter="\n"): logs_1 | File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 783, in iter_lines logs_1 | for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode): logs_1 | File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 742, in generate logs_1 | raise ChunkedEncodingError(e) logs_1 | requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

UPDATE: I've discovered the API is sending back a NoneType at some point as well. So how can I account for this null byte somewhere in the response without blowing everything up? Each individual event is ended with a \n, and I need to be able to inspect each even individually. Should I chunk the content instead of iter_lines? Then ensure there is no NoneType in the chunk? That way I don't try to iter_lines over a NoneType and it blows up?

gushitong · Accepted Answer · 2017-08-22 12:19:35Z

ChunkedEncodingError is caused by: httplib.IncompletedRead

import httplib def patch_http_response_read(func): def inner(*args): try: return func(*args) except httplib.IncompleteRead, e: return e.partial return inner httplib.HTTPResponse.read = patch_http_response_read(httplib.HTTPResponse.read)

I think this could be a patch. It allows you to deal with defective http servers.

Most servers transmit all data, but due implementation errors they wrongly close session and httplib raise error and bury your precious bytes.

Thank you for this. In my example the exception is actually thrown on r.iter_lines() so where would the above fit in there? I'm using requests with stream=True so I don't have to read the whole response into memory first.
As a clarification, this is indeed a valid approach. It won't work in python3, but there you can use requests.get(..., stream=True) + for line in response.iter_lines() inside a try-except.
For Python3 you need import http and replace httplib with http.client as well as except httplib.IncompleteRead as e

Bordotti · Accepted Answer · 2019-04-30 14:27:34Z

As I posted here mentioned by another guy IncompleteRead, you can use the "With" clause to make sure that your previous request has closed.

 with requests.request("POST", url_base, json=task, headers=headers) as report: print('report: ', report)

My implementation was already using a with clause, but I still get chunked encoding errors, so I don't think this is the root of the issue

abasar · Accepted Answer · 2022-04-17 06:16:47Z

If you are sharing a requests.Session object across multiple processes (multiprocessing), it may lead to this error. You can create a seperate Session per process (os.getpid()).

Collectives™ on Stack Overflow

Python Requests - ChunkedEncodingError(e) - requests.iter_lines

3 Answers 3

3 Comments

1 Comment

Comments

Linked

Hot Network Questions