1

Version: Python 2.7.10.

I have the following: 

r = requests.post(url = API_ENDPOINT, headers = headers, data = data) print(r.text) print(type(r.text)) print(r.text[2])

The output

{"type":"quote","symbol":"SPY","bid":266.52,"bidsz":5,"bidexch":"P","biddate":"1513040398000","ask":266.55,"asksz":100,"askexch":"P","askdate":"1513040399000"} {"type":"trade","symbol":"SPY","exch":"P","price":"266.31","size":"0","cvol":"83077533","date":"1513040400000","last":"266.31"} {"type":"summary","symbol":"SPY","open":"265.58","high":"266.38","low":"265.4793","prevClose":"265.51","close":"266.31"} <type 'unicode'> b

I would like to get the output "SPY".

I added the following:

new = simplejson.loads(r.text) print(new)

Now, I get the following: 

Traceback (most recent call last): File "example.py", line 63, in <module> new = simplejson.loads(r.text) File "/Library/Python/2.7/site-packages/simplejson/__init__.py", line 518, in loads return _default_decoder.decode(s) File "/Library/Python/2.7/site-packages/simplejson/decoder.py", line 373, in decode raise JSONDecodeError("Extra data", s, end, len(s)) simplejson.errors.JSONDecodeError: Extra data: line 1 column 160 - line 1 column 407 (char 159 - 406)

I changed:

r = requests.post(url = API_ENDPOINT, headers = headers, data = data).json()

Now, I get: 

Traceback (most recent call last): File "example.py", line 51, in <module> r = requests.post(url = API_ENDPOINT, headers = headers, data = data).json() File "/Library/Python/2.7/site-packages/requests/models.py", line 884, in json self.content.decode(encoding), **kwargs File "/Library/Python/2.7/site-packages/simplejson/__init__.py", line 518, in loads return _default_decoder.decode(s) File "/Library/Python/2.7/site-packages/simplejson/decoder.py", line 373, in decode raise JSONDecodeError("Extra data", s, end, len(s)) simplejson.errors.JSONDecodeError: Extra data: line 1 column 160 - line 1 column 407 (char 159 - 406)

I added:

new = json.dumps(r.text) print(new) print(type(new))

Now, output is: 

"{\"type\":\"quote\",\"symbol\":\"SPY\",\"bid\":266.52,\"bidsz\":5,\"bidexch\":\"P\",\"biddate\":\"1513040398000\",\"ask\":266.55,\"asksz\":100,\"askexch\":\"P\",\"askdate\":\"1513040399000\"} {\"type\":\"trade\",\"symbol\":\"SPY\",\"exch\":\"P\",\"price\":\"266.31\",\"size\":\"0\",\"cvol\":\"83077533\",\"date\":\"1513040400000\",\"last\":\"266.31\"} {\"type\":\"summary\",\"symbol\":\"SPY\",\"open\":\"265.58\",\"high\":\"266.38\",\"low\":\"265.4793\",\"prevClose\":\"265.51\",\"close\":\"266.31\"}" <type 'str'>

If I do: 

for line in r.text.splitlines(): d = json.loads(line)

I get:

Traceback (most recent call last): File "example.py", line 54, in <module> d = json.loads(line) File "/System/Library/Frameworks/Python. framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/System/Library/Frameworks/Python.framework/ Versions/2.7/lib/python2.7/json/decoder.py", line 369, in decode raise ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 1 column 158 - line 1 column 404 (char 157 - 403)

So I tried:

for line in r.text.splitlines(): print(line) print("\n\n")

And I got: 

{"type":"quote","symbol":"SPY","bid":267.18,"bidsz":1,"bidexch":"P","biddate":"1513213200000","ask":267.22,"asksz":3,"askexch":"P","askdate":"1513213200000"} {"type":"quote","symbol":"SPY","bid":267.18,"bidsz":1,"bidexch":"P","biddate":"1513213200000","ask":267.22,"asksz":3,"askexch":"P","askdate":"1513213200000"}

So even though there should be two lines, it interprets everything as a single line.

How can I convert r.text to a dictionary?

Improve this question
8
  • Try using json.loads Commented Dec 12, 2017 at 6:33
  • 1
    Are you using the requests module? If so, it provides a .json() method. But why are you using Python 2? Commented Dec 12, 2017 at 6:35
  • Why would r.text[2] return "b" in your example? Commented Dec 12, 2017 at 6:36
  • I guess server is not providing correct/well-formed json data. Note that the pasta (in your sample) does not have the ending double quote. Commented Dec 12, 2017 at 6:55
  • mshsayem - that was a typo on my end, sorry. Commented Dec 12, 2017 at 7:06

2 Answers 2

Reset to default
1

You should be using this .json():

r.json()
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

@Everett if you presented the output that you get, then everything should be fine. Probably you get error, and you should check for r.status_code, for not 2xx status response.
r.json() gives simplejson.errors.JSONDecodeError: Extra data: line 1 column 160 - line 1 column 407 (char 159 - 406).
@Everett i saw that in your updated question. Check comment above.
r.status_code is 200. What do you mean when you say "everything is fine"? Are you saying r.text["dinner"] will return "pasta"?
I am saying that you getting string object in response. r.text is a string. And if response is valid json, then it needs r.json() should return dict.
|
1

I believe what you are receiving is JSON Lines format. Each line of the response is a single json string. I simulated the response:

import json class r: pass r.text = u'''\ {"type":"quote","symbol":"SPY","bid":266.52,"bidsz":5,"bidexch":"P","biddate":"1513040398000","ask":266.55,"asksz":100,"askexch":"P","askdate":"1513040399000"} {"type":"trade","symbol":"SPY","exch":"P","price":"266.31","size":"0","cvol":"83077533","date":"1513040400000","last":"266.31"} {"type":"summary","symbol":"SPY","open":"265.58","high":"266.38","low":"265.4793","prevClose":"265.51","close":"266.31"} ''' print(r.text) print(type(r.text)) print(r.text[2]) # Parse JSON a line at a time: for line in r.text.splitlines(): d = json.loads(line) print d['symbol']

Output:

{"type":"quote","symbol":"SPY","bid":266.52,"bidsz":5,"bidexch":"P","biddate":"1513040398000","ask":266.55,"asksz":100,"askexch":"P","askdate":"1513040399000"} {"type":"trade","symbol":"SPY","exch":"P","price":"266.31","size":"0","cvol":"83077533","date":"1513040400000","last":"266.31"} {"type":"summary","symbol":"SPY","open":"265.58","high":"266.38","low":"265.4793","prevClose":"265.51","close":"266.31"} <type 'unicode'> t SPY SPY SPY
Improve this answer

9 Comments

The actual response is much longer - 3000-4000 characters. I just pasted a representative portion of it above. The entire structure of the response is exactly the same as what I have posted.
@Everett but as you can see, the representative portion you posted doesn't reproduce the problem. The error is at offset 160.
^Updated post accordingly.
@Everett After your update, is the data exactly as shown or have you added newlines to format it for display? I suspect each individual line is a separately json-formatted record (called json lines format). The json module itself can only process one complete json entry at a time, hence the "extra data" error, but if those extra newlines are in the data you won't be able to iterate over lines and process the json individually.
I added the newlines myself to format it for display. The actual data has no space between the first line and the second line - so there would be no space after the comma and it would be: ... 5,"bidexch" ...
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.