sending batch requests

Question

I have a daemon that does the following

retrieves site members from a mysql database (I used LIMIT 1000 to retrieve 1000 rows at a time)
send information about these members to a third party server
flag each member as having been processed
Sleep for 2 seconds
Retrieve the next batch of 1000 "unprocessed" members and send to third party server. and so on.

I am wondering whether a php daemon (I am using the system Daemon library), is the best way to accomplish this task delineated above.

I am worried of wasting too much memory (as PHP is known for that)

I am also worried about sending multiple requests to third party server, because on a high traffic day, there can be a lot of nonreceipts.

Is there a tool other than daemon I can use to accomplish this task? What methods can I implement to make this efficient considering there is a possibility of having to process over 100K rows in the mysql table, and the task is time sensitive. Also, at what point should I consider adding more servers?

Thanks!

So your system can generate 1000 new entries every 2 seconds? Consider that batch loading is much more efficient (I think that is available with mysql). just ftp a file 1x per hour. No respectable database should break a sweat loading 100K rows. Good luck. — shellter
– shellter, Commented Apr 1, 2012 at 23:06

Baduca Alexandru · Accepted Answer · 2012-04-01 21:23:35Z

A cron should be a very good option for doing a sync job with a third party server. Consider the following 'improvments':

1) A lock file to prevent multiple jobs from starting in parallel and taking extra resources from other processes you have running. And also to avoid duplicate processing of data.

2) If you don't have already implement an 'information update' and 'sync time' check on your side. For example if user A hasn't suffered any changes since he was sync you don't sync him again.

3) Consider how often you need data to be sync and if it doesn't have to be real time factor that into the selection query. Combined with user/time distribution and other factors you migth end up having periods of time when your script doesn't sync that many accounts.

4) Do your own memory cleanup unsetting variables, unlinking files and even reusing the same variables so you don't have garbage variables that are a 1 time use only inside the scripts. Carefull with this as it might lead to obfuscating the code.

Also consider using smaller datasets when you send them to php for processing. Databases love big datasets, php doesn't.

Alex Amiryan · Accepted Answer · 2012-04-01 21:21:35Z

I would suggest you using Perl, as it is more memory and performance efficient and it has more features for integrating with system and running as daemon.

And now about when it's time for adding more servers. I am assuming that third party server has enough resources for processing many records. So if you are running out of resources on your side I would suggest using MySQL replication to replicate your DBs to other server(s) and running above mentioned daemon there.

Collectives™ on Stack Overflow

sending batch requests

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related