Making HTTP HEAD request with urllib2 from Python 2

Question

I'm trying to do a HEAD request of a page using Python 2.

I am trying

import misc_urllib2 ..... opender = urllib2.build_opener([misc_urllib2.MyHTTPRedirectHandler(), misc_urllib2.HeadRequest()])

with misc_urllib2.py containing

class HeadRequest(urllib2.Request): def get_method(self): return "HEAD" class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler): def __init__ (self): self.redirects = [] def http_error_301(self, req, fp, code, msg, headers): result = urllib2.HTTPRedirectHandler.http_error_301( self, req, fp, code, msg, headers) result.redirect_code = code return result http_error_302 = http_error_303 = http_error_307 = http_error_301

But I am getting

TypeError: __init__() takes at least 2 arguments (1 given)

If I just do

opender = urllib2.build_opener(misc_urllib2.MyHTTPRedirectHandler())

then it works fine

Anthony Geoghegan · Accepted Answer · 2016-06-27 13:56:15Z

This works just fine:

import urllib2 request = urllib2.Request('http://localhost:8080') request.get_method = lambda : 'HEAD' response = urllib2.urlopen(request) print response.info()

Tested with quick and dirty HTTPd hacked in python:

Server: BaseHTTP/0.3 Python/2.6.6 Date: Sun, 12 Dec 2010 11:52:33 GMT Content-type: text/html X-REQUEST_METHOD: HEAD

I've added a custom header field X-REQUEST_METHOD to show it works :)

Here is HTTPd log:

Sun Dec 12 12:52:28 2010 Server Starts - localhost:8080 localhost.localdomain - - [12/Dec/2010 12:52:33] "HEAD / HTTP/1.1" 200 -

Edit: there is also httplib2

import httplib2 h = httplib2.Http() resp = h.request("http://www.google.com", 'HEAD')

Python3 (3.3+) added support for the method keyword argument to the Request initializer - as an alternative to bypass the need for the lambda

Community · Accepted Answer · 2017-05-23 10:31:28Z

Try httplib

>>> import httplib >>> conn = httplib.HTTPConnection("www.google.com") >>> conn.request("HEAD", "/index.html") >>> res = conn.getresponse() >>> print res.status, res.reason 200 OK >>> print res.getheaders() [('content-length', '0'), ('expires', '-1'), ('server', 'gws'), ('cache-control', 'private, max-age=0'), ('date', 'Sat, 20 Sep 2008 06:43:36 GMT'), ('content-type', 'text/html; charset=ISO-8859-1')]

See How do you send a HEAD HTTP request in Python 2?

Does that allow you to set other request objs? See my OP, the MyHTTPRedirectHandler
We ran into a bug using this approach. Later someone wanted to check an HTTPS url, for which you have to use a different method: httplib.HTTPSConnection()

matiasg · Accepted Answer · 2010-12-12 15:37:42Z

The problem lies with your class HeadRequest, which inherits from urllib2.Request. According to doc, urllib2.Request.__init__ signature is

 __init__(self, url, data=None, headers={}, origin_req_host=None, unverifiable=False)

so you must pass an url argument to it. In your second try, you just do not use HeadRequest, this is why it works.

uingtea · Accepted Answer · 2017-06-14 19:35:45Z

you shoud not add HeadRequest to build_opener or add_handler it should be called like this

opener = urllib2.build_opener(MyHTTPRedirectHandler) response = opener.open(HeadRequest(url)) print response.getheaders()

Collectives™ on Stack Overflow

Making HTTP HEAD request with urllib2 from Python 2

4 Answers 4

1 Comment

2 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

2 Comments

Comments

Comments

Linked

Related