File_get_contents, curl not working

Question

Something strange is going on, and I would like to know why.

On this url: http://api.promasters.net.br/cotacao/v1/valores?moedas=USD&alt=json, which works well in the browser, but when I tried to retrieve the content with php:

echo file_get_contents('http://api.promasters.net.br/cotacao/v1/valores?moedas=USD&alt=json');

printed nothing, with var_dump(...) = string(0) "", so i went a little further and used:

function get_page($url) { $curl = curl_init(); curl_setopt($curl, CURLOPT_RETURNTRANSFER, True); curl_setopt($curl, CURLOPT_URL, $url); $return = curl_exec($curl); curl_close($curl); return $return; } echo get_page('http://api.promasters.net.br/cotacao/v1/valores?moedas=USD&alt=json');

Also printed nothing, so i tried python (3.X):

import requests print(requests.get('http://api.promasters.net.br/cotacao/v1/valores?moedas=USD&alt=json').text)

And WORKED. Why is this happening? What's going on?

are you using this on a server or on your own pc? could be that the php module for those function is not set enabled which would indeed occur in blank. — Giovanni Le Grand
– Giovanni Le Grand, Commented May 31, 2016 at 14:09
you're just assuming success. echo won't print out the boolean false that you'd get back on failure. use var_dump instead. — Marc B
– Marc B, Commented May 31, 2016 at 14:10
Nop, the api it's on an external server, and i also tried echo file_get_contents('http://stackoverflow.com/questions/37548007/file-get-contents-curl-not-working?noredirect=1#comment62584729_37548007'); and it worked — Miguel
– Miguel, Commented May 31, 2016 at 14:10
try playing with the options, especially try adding curl_setopt($curl, CURLOPT_FOLLOWLOCATION, True); before the curl_exec — Lud
– Lud, Commented May 31, 2016 at 14:19

Jeff Puckett · Accepted Answer · 2016-05-31 14:58:06Z

3

It looks like they're blocking the user agent, or lack thereof, considering that php curl and file_get_contents doesn't seem to set the value in the request header.

You can fake this by setting it to something like Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

<?php function get_page($url) { $curl = curl_init(); curl_setopt($curl, CURLOPT_RETURNTRANSFER, True); curl_setopt($curl, CURLOPT_URL, $url); curl_setopt($curl,CURLOPT_USERAGENT,'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1'); $return = curl_exec($curl); curl_close($curl); return $return; } echo get_page('http://api.promasters.net.br/cotacao/v1/valores?moedas=USD&alt=json');

edited May 31, 2016 at 14:58

answered May 31, 2016 at 14:30

Jeff Puckett

41.5k19 gold badges125 silver badges177 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Miguel Over a year ago

Thanks it was that indeed. Just out of curiosity, what is the default user_agent and why it was working with curl on console and with python 3.x?

Jeff Puckett Over a year ago

I don't think php curl or file_get_contents has one. I created a test page that only has var_dump($_SERVER) and HTTP_USER_AGENT was not set. When I manually set the agent as in my answer, it showed as expected. When running curl from console, it sets the agent as curl/7.35.0 in my case

Miguel Over a year ago

Thanks, but it's strange that with console curl/python 3.x works. I would like to know why. Thanks very much :)

Progrock Over a year ago

Do a call to a script that dumps request headers, to inspect the python variants request.

Jeff Puckett Over a year ago

@fungusanthrax it identifies the browser to the server so that the server can potentially give different content based on user agent, such as a mobile site (before the days of responsive design). In some cases, servers ignore requests from bots or if the user agent string isn't set at all, they interpret it as something not worth answering, like a possible DDOS attack.

Progrock · Accepted Answer · 2016-05-31 15:14:30Z

I experienced the same behaviour.

Fetching the URL using the CLI Curl worked for me.

I then wrote a script with a file_get_contents call to another script that dumped all request headers to a file using getallheaders:

<?php file_put_contents('/tmp/request_headers.txt', var_export(getallheaders(),true));

Output of file:

array ( 'Host' => 'localhost', )

I then inspected the curl request headers,

$ curl -v URL

And tried adding one at a time to the file_get_contents request. It turned out a User agent header was needed.

<?php $opts = array( 'http'=>array( 'method'=>"GET", 'header'=> "User-Agent: examplebot\r\n" ) ); $context = stream_context_create($opts); $response = file_get_contents($url, false , $context);

This gave me a useful response.

Thanks, any ideia why the curl -v http://api.promasters.net.br/cotacao/v1/valores?moedas=USD&alt=json on terminal and with python worked without specifying an user agent?

Collectives™ on Stack Overflow

File_get_contents, curl not working

2 Answers 2

5 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Related