3

I am trying to construct below POST request using python requests.

Request header

Host: www.example.com User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:46.0) Gecko/20100101 Firefox/46.0 Accept: */* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br X-CSRFToken: LdoMdAvsJ3QeFZ79YK1ZKylohMYCEgif X-Requested-With: XMLHttpRequest Content-Type: multipart/form-data; boundary=---------------------------6820904081792922663698482414 Cache-Control: max-age=0 Referer: https://www.example.com/def/ Content-Length: 175 Cookie: HE_UTS_ID_CL=5f0ae4142d484a5dbc6579228be42f9a6bc68c95cdd64156af65fffb49e258d4; HE_UTS_ID_LP="/challenges/"; csrftoken=LdoMdAvsJ3QeFZ79YK1ZKylohMYCEgif; user_tz=Asia/Kolkata; _ga=GA1.2.1255646438.1464624573; lordoftherings="a5dr3g48ag2dg8s2b8r57gkil6ioip74:df99595bff3625e78f4caf1281450faf"; mp_mixpanel__c=6; mp_mixpanel__c3=4136; mp_mixpanel__c4=3974; mp_mixpanel__c5=25; hiring_challenge_anon_visited=visited; mp_4c30b8635363027dfb780d5a61785112_mixpanel=%7B%22distinct_id%22%3A%20%22155026da7bb4b-0b9cbbccc1992-3e6e034e-100200-155026da7bceb%22%2C%22url_path%22%3A%20%22%2Fchallenges%2F%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; HE_UTS_ID=c5bb2531e6f9465bb0d7115d953bdbe79c17ad924dfb467f9e76fa5c19e373b2 Connection: keep-alive

Request body

-----------------------------6820904081792922663698482414 Content-Disposition: form-data; name="submit" True -----------------------------6820904081792922663698482414--

Below is my code as of now.

import requests url = 'https://www.example.com/adadsdas?modern=true' headers = {'Host': 'www.abc.com',\ 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:46.0) Gecko/20100101 Firefox/46.0',\ 'Accept': '*/*',\ 'Accept-Language': 'en-US,en;q=0.5',\ 'Accept-Encoding': 'gzip, deflate, br',\ 'X-CSRFToken': '6jdyxOPzlE4ui1HzNa5RRIYhGAaRtalw',\ 'X-Requested-With': 'XMLHttpRequest',\ 'Referer': 'https://www.example.com/def/',\ 'Content-Length': '175',\ 'Content-Type': 'multipart/form-data; boundary=---------------------------6820904081792922663698482414',\ 'Cookie': 'HE_UTS_ID_CL=4179383969bd41a489340b5ceeb4cde3225d29cfa7734ea09a8a57b0fbd1e0cd; HE_UTS_ID_LP="/challenges/"; csrftoken=6jdyxOPzlE4ui1HzNa5RRIYhGAaRtalw; user_tz=Asia/Kolkata; _ga=GA1.2.1255646438.1464624573; lordoftherings="a5dr3g48ag2dg8s2b8r57gkil6ioip74:df99595bff3625e78f4caf1281450faf"; mp_mixpanel__c=6; mp_mixpanel__c3=4136; mp_mixpanel__c4=3974; mp_mixpanel__c5=25; hiring_challenge_anon_visited=visited; mp_4c30b8635363027dfb780d5a61785112_mixpanel=%7B%22distinct_id%22%3A%20%22155026da7bb4b-0b9cbbccc1992-3e6e034e-100200-155026da7bceb%22%2C%22url_path%22%3A%20%22%2Fchallenges%2F%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; HE_UTS_ID=efd6ef54d30f48efbc0be67b2283545daf69267d76e6403595f683a26a5adbf7',\ 'Connection': 'keep-alive',\ 'Cache-Control': 'max-age=0'} requests.post(url,headers=headers,files={'submit':(None,'True')})

I am getting below exception when passing files dict, What is the correct way to construct the POST request as mentioned above ? How to construct request body ? And to view raw request to check what is being constructed ?

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 87, in post return request('post', url, data=data, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 276, in request prep = req.prepare() File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 224, in prepare p.prepare_body(self.data, self.files) File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 369, in prepare_body (body, content_type) = self._encode_files(files, data) File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 131, in _encode_files body, content_type = encode_multipart_formdata(new_fields) File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/filepost.py", line 73, in encode_multipart_formdata content_type = get_content_type(filename) File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/filepost.py", line 27, in get_content_type return mimetypes.guess_type(filename)[0] or 'application/octet-stream' File "/usr/lib/python2.7/mimetypes.py", line 298, in guess_type return _db.guess_type(url, strict) File "/usr/lib/python2.7/mimetypes.py", line 114, in guess_type scheme, url = urllib.splittype(url) File "/usr/lib/python2.7/urllib.py", line 1074, in splittype match = _typeprog.match(url) TypeError: expected string or buffer
Improve this question
2
  • Can you share the actual link? Commented May 30, 2016 at 19:26
  • actual link is https://www.hackerearth.com/challenges/ , I makes a AJAX POST call to https://www.hackerearth.com/AJAX/filter-challenges/?modern=true . I can call this API without the request body successfully, but I want to learn how to construct request body. Commented May 30, 2016 at 19:50

1 Answer 1

Reset to default
4

Your error is as per the comments to this answer from using an ancient version of requestes 1.1.0 from 2013, so pip install -U requests will fix that.

To do the post you don't need to hardcode anything, you create a session and get whatever you need i.e the csrf token and requests will take care of the rest:

import requests headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:46.0) Gecko/20100101 Firefox/46.0', 'X-Requested-With': 'XMLHttpRequest', "referer": "https://www.hackerearth.com/challenges/"} with requests.Session() as s: s.get("https://www.hackerearth.com") headers["X-CSRFToken"] = s.cookies["csrftoken"] r = s.post("https://www.hackerearth.com/AJAX/filter-challenges/?modern=true", headers=headers, files={'submit': (None, 'True')}) print(r.json())

The three necessary things were, X-Requested-With as it has to be an ajax request, the referer and the csrf token.

To access the raw request data, you can access the attributes on r.request:

print(r.request.body) print(r.request.headers)
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

I hardcoded csrf token just to get to point with the post request which is not working when passing None in fields dict. I still get the same exception when using your code .
@g4ur4v, that is almost certainly because you are using an outdated version of requests, what version do you have installed? The code works perfectly with both 2.9.1 and 2.10.0
>>> requests.__version__ '1.1.0'
Oh lord, that is from 2013, you really need to pip install -U requests
it works now after upgrading. Thanks please add this to your answer so that I can accept it.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.