1

I am collecting market prices from different exchanges, the exchanges are set up for 1000's of requests a second however I am concerned that when my website is under heavy use this cURL function will be too resource intensive on my server.

Here is the cURL function, which gets results from between 2 and 4 exchanges (depending if an exchange timeouts):

function curl($exchange,$timeout) { $a = curl_init(); curl_setopt($a, CURLOPT_TIMEOUT, $timeout); curl_setopt($a, CURLOPT_RETURNTRANSFER, 1); curl_setopt($a,CURLOPT_SSL_VERIFYPEER, false); curl_setopt($a, CURLOPT_HTTPHEADER, array('Accept: application/json', 'Content-Type: application/json')); curl_setopt($a, CURLOPT_URL, $exchange); $result = curl_exec($a); $httpCode = curl_getinfo($a, CURLINFO_HTTP_CODE); if($httpCode == 200) { return json_decode($result); } else { return false; } curl_close($a); }

I am using AJAX to load the script asynchronously since it takes a few seconds to complete. It is loaded on the homepage and I am anticipating ~15,000 unique hits a day.

Will I run into issues if cURL is called upon many times a second and if so is there a better alternative?

Improve this question

2 Answers 2

Reset to default
1

One option would be to implement a caching mechanism , this certainly will reduce the server's overhead . Frameworks like ZF , Symfony and Laravel have this mechanism build-in . For instance , in Laravel the implementation is simple as : 

Cache::put('key', 'value', $minutes); . // Retrieving the data if (Cache::has('key')) { // ...... $value = Cache::get('key'); // ....... }

On what persisted layer the data will be cached (file , Memcached or Redis) is up to us . In Laravel its just a single configuration option (provided that our server has the aforementioned services installed) . We should also implement a "Queue" service to run the "time consuming" tasks in the background (Beanstalkd , Ironio , Amazon's SQS) . Combined with a cron-job , our Queue service could update / refresh the cached data .On a shared hosted environment the most obvious choice is to use "file" for caching and a cloud based Queue (Ironio has also a free tier) . Hope my comment gave you a starting point .

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

Instead of doing a cURL request for every visitor, set up a scheduled task with Cron (et al) to update a local cache in, say, a MySQL table every few minutes. Then you just have the overhead of reading your cache instead of having to make multiple cURL requests for every page load.

To answer the question in the title: not hugely intensive. You'll have to wait for the network of course, but I don't think cURL will be your bottleneck, especially if you're only requesting data to update a local cache.

Improve this answer

3 Comments

Am I right in interpretting your answers as: a cron job running cURL and an INSERT statement every few minutes + another SQL SELECT statement on the site will be less resource intensive than running the cURL function for every user? Because I had thought of doing that but after thinking about it wasn't sure if that would be any better than just running cURL via AJAX for each visitor..
@MrMayo the cached approach is very scalable, and very fast, and the overhead of updating a cache every now and then is minimal. the "curl fetch again for each user" approach gets more resource expensive very fast, the more concurrent users you have. (the cache can be anything tho, doesn't have to be a mysql server db)
the fastest cache is probably just a locally stored file and using readfile() - then you're essentially using fopen/fread, and leveraging the filesystem/OS IO cache, and fopen is much faster than creating an autenticated mysql connection

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.