Curl with PHP to scrape a website not working

Question

I am trying to call Citi Bank's open API (https://developer.citi.com/) and this require me to scrape the screen to allow the user to login with his username and password.

This works if I simply put this URL with parameters in the browser.

https://sandbox.apihub.citi.com/gcb/api/authCode/oauth2/authorize?response_type=code&client_id=<my_client_id>&scope=pay_with_points&countryCode=SG&businessCode=GCB&locale=en_SG&state=12093&redirect_uri=<my_callback>

However, when I attempt to make the same call from my PHP codes with curl, it returns status code of 503.

<?php $header = array(); $header[] = 'Upgrade-Insecure-Requests: 1'; $header[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'; $header[] = 'Accept-Encoding: gzip, deflate, br'; $header[] = 'Accept-Language: en-US,en;q=0.8,ja;q=0.6,zh-CN;q=0.4,zh;q=0.2,zh-TW;q=0.2,th;q=0.2'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'https://sandbox.apihub.citi.com/gcb/api/authCode/oauth2/authorize?response_type=code&client_id=<my_client_id>=pay_with_points&countryCode=SG&businessCode=GCB&locale=en_SG&state=12093&redirect_uri=<my_callback_url>'); curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'); curl_setopt($ch, CURLOPT_HTTPHEADER, $header); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); curl_setopt($ch, CURLOPT_ENCODING, ''); curl_setopt($ch, CURLOPT_TIMEOUT, 20); $result = curl_exec($ch); curl_close ($ch); echo $result; ?>

I have attempted to change my request headers so that it is just like how it would look if I had entered it as a URL in my browser.

I must have missed out something that I need to configure in curl.

Would anyone have some idea? Thank you!

It's my callback URL. The authorize api will call back to what I state here, for example myurl.com/bank/hello.php. — Jackson Ng
– Jackson Ng, Commented Oct 18, 2017 at 9:29
did your read the documentation, i don't think you need those headers — madalinivascu
– madalinivascu, Commented Oct 18, 2017 at 9:32

Glorfindel · Accepted Answer · 2023-04-23 08:02:19Z

The issue might be due to https. There are few options available.

1. You can download the https://curl.haxx.se/ca/cacert.pem file and save it and then add this option

curl_setopt($ch, CURLOPT_CAINFO, "/path/to/cacert.pem");

2. You can download the certificate from the browser by going to that site and do the same as above. You might run into problems if they change their certificates, need to confirm with them.

3. This is not recommended but can be used temporarily for debugging purposes to find out, if this is the actual problem. It introduces MIMT attacks.

//Only use for debugging purposes. curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0); curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);

Thanks Kamal. I realized that when viewed in a browser, the page redirects and drops a cookie before redirecting to the next page. If I paste the cookie and curl with it, it works. But the page only generates the HTML portion. Parts that get written using JavaScript doesn't come up. I figured perhaps curl isn't the correct solution for this. I should have just done a page redirect. The downside is that the user sees my client_id. I need to think if it actually matters that my client_id isn't a secret.

Jay Bharat · Accepted Answer · 2021-07-05 08:24:28Z

-1

You can use header('location: '.$url);

answered Jul 5, 2021 at 8:24

Jay Bharat

8998 silver badges6 bronze badges

Collectives™ on Stack Overflow

Curl with PHP to scrape a website not working

2 Answers 2

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Related