How to get website using cURL

Question

I am not able to fetch data using cURL in php AS I can do this using curl command in terminal like below -

curl http://www.universalapplianceparts.com/search.aspx?find=w10130694

This command provide expected result in terminal.

I want same result in php using curl

Below is my php code -

$url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694"; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL,"http://www.universalapplianceparts.com/"); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HEADER, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); $output = curl_exec ($ch); curl_close ($ch); var_dump($output);

Please let me know why I am not getting same result using php curl as I am getting in terminal.

Thanks

I want ot fetch html data in my browser as I am getting in terminal . I think something is missing in php code — John
– John, Commented Mar 10, 2016 at 7:17
As far as I can see, you are using a POST, while you're actually want to GET. I think if you change POST to GET and delete the postfields and header line, it should work... — therebelcoder
– therebelcoder, Commented Mar 10, 2016 at 7:23

Exception · Accepted Answer · 2016-03-10 07:35:23Z

Try this ::

 $url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694"; $ch1= curl_init(); curl_setopt ($ch1, CURLOPT_URL, $url ); curl_setopt($ch1, CURLOPT_HEADER, 0); curl_setopt($ch1,CURLOPT_VERBOSE,1); curl_setopt($ch1, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)'); curl_setopt ($ch1, CURLOPT_REFERER,'http://www.google.com'); //just a fake referer curl_setopt($ch1, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch1,CURLOPT_POST,0); curl_setopt($ch1, CURLOPT_FOLLOWLOCATION, 20); $htmlContent= curl_exec($ch1); echo $htmlContent;

Perfect.. thats what I needed. Perfectly works.. But Can you please explain it. So I can understand it properly that what I am doing.. I dont want to just use this I also want to learn. and it will also helps to many fellows also
look you first need to learn what is curl and how it works. please refer php.net/manual/en/book.curl.php this will help you..
This code provides proper OP but while writing regexp for particular div it again returns complete html. can you check once. this is my regexp preg_match_all('/.*<div.*class=\"product\-list\-options\".*>.*<a href=\"(.*)\">.*<\/a>.*<\/div>/s',$htmlContent,$matches); I need only href value

irfanengineer · Accepted Answer · 2016-03-10 07:32:28Z

Try This:

 $url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694"; $options = Array( CURLOPT_RETURNTRANSFER => TRUE, CURLOPT_FOLLOWLOCATION => TRUE, CURLOPT_AUTOREFERER => TRUE, CURLOPT_HTTPHEADER => array("Content-Type: application/text"), CURLOPT_TIMEOUT => 120, CURLOPT_MAXREDIRS => 10, CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8", CURLOPT_URL => $url, ); $ch = curl_init(); curl_setopt_array($ch, $options); $data = curl_exec($ch); curl_close($ch);

I am still getting error page while fetching data. can you check terminal command output it shows proper complete html code in terminal same output should be dispalyed in browser so I can copy content from that output

Ravindra Bhalothia · Accepted Answer · 2016-03-10 08:48:13Z

I tried your code and getting error related to length so you need to add content length as well in your curl request. Try below code hope it will help you :

$url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694"; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/text")); curl_setopt($ch, CURLOPT_USERAGENT, array("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8")); curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Length: 0')); curl_setopt($ch, CURLOPT_POST, 1); $result = curl_exec($ch); var_dump($result);

kindly add below as well curl_setopt($ch, CURLOPT_USERAGENT, array("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8"));
yes.. also I tried accepted answer but it returns complete html again in regexp. still my regexp should fetch only href content from div class in anchor tag, this is my regexp preg_match_all('/.*<div.*class=\"product\-list\-options\".*>.*<a href=\"(.*)\">.*<\/a>.*<\/div>/s',$htmlContent,$matches);

Collectives™ on Stack Overflow

How to get website using cURL

3 Answers 3

4 Comments

2 Comments

4 Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

2 Comments

4 Comments

Related