2

I am using curl_multi_exec() for just 5 url. Now i have this strange issue. When i run my code on xampp , it works perfect. i can see $running value initialized with 5 and then keeps decreasing. . But, when i tried it on other localhost(on arm architecture), $running gets initialized with 0. so my curl_multi_exec() never returns any response.

Here is the code snippet :

do { curl_multi_exec($master,$running); echo "<pre>"; var_dump($running ); echo "</pre>"; } while($running > 0); 

Here is my entire code:

 $nodes = array( 'https://www.example.com', 'https://www.example2.com', 'https://www.example3.com', 'https://www.example4.com', 'https://www.example5.com' ); $node_count = count($nodes); $curl_arr = array(); $master = curl_multi_init(); for($i = 0; $i < $node_count; $i++) { $url =$nodes[$i]; $curl_arr[$i] = curl_init($url); curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true); curl_multi_add_handle($master, $curl_arr[$i]); } do { curl_multi_exec($master,$running); echo "<pre>"; var_dump($running ); echo "</pre>"; } while($running > 0); for($i = 0; $i < $node_count; $i++) { $results = curl_multi_getcontent($curl_arr[$i]); var_dump($results); } 

I googled a few things and got to know curl ssl might be an issue. So, i installed another localhost(on ARM) with openssl and curl ssl enabled. Now i have two different localhost(both for ARM) with SSL enabled, this snippet works fine on one localhost and doesn't work on the other one.

And somehow i need that "other one" because it has lot more features.

Someone please guide what might be the issue with this $running initialization?

Any help is appreciated :)

Tried this also, but no success

 <?php // echo "<meta http-equiv='refresh' content='3'/>" ; include_once ("simple_html_dom.php"); libxml_use_internal_errors(true); function get_string_between($string, $start, $end){ $string = ' ' . $string; $ini = strpos($string, $start); if ($ini == 0) return ''; $ini += strlen($start); $len = strpos($string, $end, $ini) - $ini; return substr($string, $ini, $len); } function multi_thread_curl($url_array, $number_threads) { $curl_array = array_chunk($url_array, $number_threads, $preserve_keys = true); //Iterate through each batch of urls. foreach($curl_array as $threads) { //Create your cURL resources. foreach($threads as $key=>$value) { ${'ch' . $key} = curl_init(); curl_setopt(${'ch' . $key}, CURLOPT_URL, $value); curl_setopt(${'ch' . $key}, CURLOPT_SSL_VERIFYPEER, false); curl_setopt(${'ch' . $key}, CURLOPT_RETURNTRANSFER, true); curl_setopt(${'ch' . $key}, CURLOPT_TIMEOUT, 10); } //Create the multiple cURL handler. $mh = curl_multi_init(); //Add the handles. foreach($threads as $key=>$value) { curl_multi_add_handle($mh, ${'ch' . $key}); } $active = null; //execute the handles. do { $mrc = curl_multi_exec($mh, $active); } while ($mrc == CURLM_CALL_MULTI_PERFORM); while ($active && $mrc == CURLM_OK) { echo $active; if (curl_multi_select($mh) != -1) { do { $mrc = curl_multi_exec($mh, $active); } while ($mrc == CURLM_CALL_MULTI_PERFORM); } } //Get your data and close the handles. foreach($threads as $key=>$value) { $results[$key] = curl_multi_getcontent(${'ch' . $key}); curl_multi_remove_handle($mh, ${'ch' . $key}); } //Close the multi handle exec. curl_multi_close($mh); } return $results; } $nodes = array( 'https://www.example1.com', 'https://www.example2.com', 'https://www.example3.com', 'https://www.example4.com', 'https://www.example5.com', ); $node_count = count($nodes); echo "results: "; $number_threads = 5; $results = multi_thread_curl($nodes, $number_threads); print_r($results); echo 'done'; ?> 

Issue Here is : $active is always 5. Forever Loop :(

4
  • please post your curl code. And putting a multi exec curl function inside a while loop probably is not a good practice Commented May 2, 2018 at 3:28
  • 1
    @Joseph_J curl_multi_exec always works inside while to keep track of current running handles. You can visit : curl_multi_exec php.net official to have a look :) Commented May 2, 2018 at 3:43
  • My apologies, you are right. For some reason I was thinking it was a function you had written that contained the curl_multi_exec. Commented May 2, 2018 at 3:47
  • @Joseph_J That's OK :) . Any Help or comment is appreciated. :) Commented May 2, 2018 at 4:04

1 Answer 1

2

Here is a multi-thread-curl function that I put together using examples from PHP.net. I have used this function to get large amounts of URL's. It is capable of really speeding things up. I have had great success with it.

You could even expand on it and add a parameter for your curl options.

function multi_thread_curl($url_array, $number_threads) { $curl_array = array_chunk($url_array, $number_threads, $preserve_keys = true); //Iterate through each batch of urls. foreach($curl_array as $threads) { //Create your cURL resources. foreach($threads as $key=>$value) { ${'ch' . $key} = curl_init(); curl_setopt(${'ch' . $key}, CURLOPT_URL, $value); curl_setopt(${'ch' . $key}, CURLOPT_SSL_VERIFYPEER, false); curl_setopt(${'ch' . $key}, CURLOPT_RETURNTRANSFER, true); curl_setopt(${'ch' . $key}, CURLOPT_TIMEOUT, 10); } //Create the multiple cURL handler. $mh = curl_multi_init(); //Add the handles. foreach($threads as $key=>$value) { curl_multi_add_handle($mh, ${'ch' . $key}); } $active = null; //execute the handles. do { $mrc = curl_multi_exec($mh, $active); } while ($mrc == CURLM_CALL_MULTI_PERFORM); while ($active && $mrc == CURLM_OK) { if (curl_multi_select($mh) != -1) { do { $mrc = curl_multi_exec($mh, $active); } while ($mrc == CURLM_CALL_MULTI_PERFORM); } } //Get your data and close the handles. foreach($threads as $key=>$value) { $results[$key] = curl_multi_getcontent(${'ch' . $key}); curl_multi_remove_handle($mh, ${'ch' . $key}); } //Close the multi handle exec. curl_multi_close($mh); //Limits to one group of threads. //break; } return $results; } $urls = array( 'https://en.wikipedia.org/wiki/Wiki' ); $results = multi_thread_curl($urls, 1); print_r($results); 
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.