Using cURL to download a site's HTML source, but getting different file than intended

Question

I'm trying to use cURL and PHP to download the HTML source (as it appears in the browser) of here. But instead of the actual source code, this is returned instead (a meta refresh link set to 0).

<html> <head><title>Object moved</title></head> <body> <h2>Object moved to <a href="https://login.live.com/login.srf?wa=wsignin1.0&amp;rpsnv=11&amp;checkda=1&amp;ct=1321044850&amp;rver=6.1.6195.0&amp;wp=MBI&amp;wreply=http:%2F%2Fwww.windowsphone.com%2Fen-US%2Fapps%2Fea39f002-ac30-e011-854c-00237de2db9e&amp;lc=1033&amp;id=268289">here</a>. </h2> </body> </html>

I'm trying to spoof the referral header to be the site, but it seems I'm doing it wrong. Code is below. Any suggestions? Thanks

$ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'http://www.windowsphone.com/en-US/apps/ea39f002-ac30-e011-854c-00237de2db9e'); curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6'); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1); curl_setopt($ch, CURLOPT_AUTOREFERER, false); curl_setopt($ch, CURLOPT_REFERER, "http://www.windowsphone.com/en-US/apps/ea39f002-ac30-e011-854c-00237de2db9e"); $html = curl_exec($ch); curl_close($ch);

I've got a feeling I answered this one before: stackoverflow.com/questions/8046907/… — flesk
– flesk, Commented Nov 11, 2011 at 21:09

Community · Accepted Answer · 2017-05-23 10:27:30Z

7

Add the curl option to follow redirects:

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

If it is a meta refresh and not an HTTP moved header, see: PHP: Can CURL follow meta redirects

As mentioned by flesk, you may also need to store the cookies.

edited May 23, 2017 at 10:27

CommunityBot

11 silver badge

answered Nov 11, 2011 at 21:03

jli

6,6432 gold badges31 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user954912 Over a year ago

I'm writing the HTML returned to a separate file (to read from later). When I set FOLLOWLOCATION to true, the file became blank.

jli Over a year ago

This works fine if the redirect is just in the headers. I just tested it with your code, and the page returns a 200 OK with a meta refresh tag. See my edit.

user954912 Over a year ago

excellent, I'll check it out. Sorry I'm new to all this HTTP/spoofing/cURL

Nick Baluk · Accepted Answer · 2011-11-11 21:15:26Z

$ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'http://www.windowsphone.com/en-US/apps/ea39f002-ac30-e011-854c-00237de2db9e'); curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6'); curl_setopt($ch, CURLOPT_HEADER, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt"); curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt"); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30); curl_setopt($ch, CURLOPT_REFERER, "http://www.windowsphone.com"); $html = curl_exec($ch); curl_close($ch); echo $html;

This doesn't fix it. You need the follow location option to be set to true.

flesk · Accepted Answer · 2011-11-11 21:14:21Z

The problem isn't the referrer but that you need to enable cookies for it to work.

Try something like this:

curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt"); curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");

You have to query the page twice. First allow redirects to get the cookie from login.live.com, then query again with the cookie set.

Collectives™ on Stack Overflow

Using cURL to download a site's HTML source, but getting different file than intended

3 Answers 3

3 Comments

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

Comments

Linked

Related