json() on "requests" response raises UnicodeEncodeError

Question

I'm querying Github's Jobs API with python3, using the requests library, but running into an error parsing the response.

Library: http://docs.python-requests.org/en/latest/

Code:

import requests import json url = 'https://jobs.github.com/positions.json?' response = requests.get(url) print(response.json())

Error:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 321: ordinal not in range(128)

Using this API in the past with Ruby, I've never run into this issue.

I also tried converting it to a dictionary but it resulted in the same errors.

There's other questions on SO about the UnicodeEncodeError (mostly re: opening files), but I'm not familiar with Python and didn't find them helpful.

Possible duplicate of UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128) — Beefster
– Beefster, Commented Dec 28, 2017 at 22:19

Hetzroni · Accepted Answer · 2019-05-01 06:55:58Z

First, check that the response is actually JSON. Try printing response.text and see if it looks like a valid JSON object.

Assuming it is JSON: it's very "hack"-ey, but you can replace the non ASCII characters with their escaped Unicode representation:

def escape_unicode(c): return c.encode('ascii', 'backslashreplace').decode('ascii') response = ... text = response.text escaped = re.sub(r'[^\x00-\x7F]', lambda m: escape_unicode(m.group(0)), text) json_response = json.loads(escaped)

Collectives™ on Stack Overflow

json() on "requests" response raises UnicodeEncodeError

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related