Am I just a lazy programmer for suggesting that as the background process knows where it is up to then it could report to the database once per loop cycle?

The reason I ask that is that I once wrote a background process whose job was importing records from other databases. The database, table needed were noted in a table called command_queue and row ID on the old system added to command_queue which the background task would start a new transaction, lock the row representing the command being acted upon, add it's system given ID number (which it got by registering on another table) then start working on and then update the new table adding the old ID to the new row data and then drop the command_queue row as complete and commit the transaction.

That way I could have multiple threads running at once, could cope with a power cut (should it ever happen) and recover from any level of catastrophe. Likewise I could monopolise the box with lots of threads if there was a lot of work to do and especially when we went home and the servers were idle.

In code igniter I had a view that would output the number of commands left to run how many were being processed and what size the tables in the database were that got a mention in the command list. (There was also a stop command for each worker thread).

Using some javascript goodness the view looked quite impressive with speedometers, pie-charts and so forth. Managers liked to see that sort of thing.

I didn't have to figure out what they were doing as I had the process tell me in great detail. Although I could go look as I used (detached) screen instances for each process and could see the output. It helped that I wanted to be the first coder to write a project that even with thousands of users a second there would be a zero crisis, zero data loss, zero disruption ever.

Am I just a lazy programmer for suggesting that as the background process knows where it is up to then it could report to the database once per loop cycle?

The reason I ask that is that I once wrote a background process whose job was importing records from other databases. The database, table needed were noted in a table called command_queue and row ID on the old system added to command_queue which the background task would start a new transaction, lock the row representing the command being acted upon, add it's system given ID number (which it got by registering on another table) then start working on and then update the new table adding the old ID to the new row data and then drop the command_queue row as complete and commit the transaction.

That way I could have multiple threads running at once, could cope with a power cut (should it ever happen) and recover from any level of catastrophe. Likewise I could monopolise the box with lots of threads if there was a lot of work to do and especially when we went home and the servers were idle.

In code igniter I had a view that would output the number of commands left to run how many were being processed and what size the tables in the database were that got a mention in the command list. (There was also a stop command for each worker thread).

Using some javascript goodness the view looked quite impressive with speedometers, pie-charts and so forth. Managers liked to see that sort of thing.

I didn't have to figure out what they were doing as I had the process tell me in great detail. Although I could go look as I used (detached) screen instances for each process and could see the output. It helped that I wanted to be the first coder to write a project that even with thousands of users a second there would be a zero crisis, zero data loss, zero disruption ever.

Edit If don't like using screen and you compiled PHP with --enable-pcntl then you can use pcntl_fork to create the background processes and setproctitle to note progress and then when you view the process list there is your process renaming itself to give you a clue.