Run multiple functions every second, write result to file

Question

I'm trying to run three functions (each can take up to 1 second to execute) every second. I'd then like to store the output from each function, and write them to separate files.

At the moment I'm using Timers for my delay handling. (I could subclass Thread, but that's getting a bit complicated for this simple script)

def main: for i in range(3): set_up_function(i) t = Timer(1, run_function, [i]) t.start() time.sleep(100) # Without this, main thread exits def run_function(i): t = Timer(1, run_function, [i]) t.start() print function_with_delay(i)

What's the best way to handle the output from function_with_delay? Append the result to a global list for each function?

Then I could put something like this at the end of my main function:

... while True: time.sleep(30) # or in a try/except with a loop of 1 second sleeps so I can interrupt for i in range(3): save_to_disk(data[i])

Thoughts?

Edit: Added my own answer as a possibility

srgerg · Accepted Answer · 2012-01-16 04:16:18Z

6

I believe the python Queue module is designed for precisely this sort of scenario. You could do something like this, for example:

def main(): q = Queue.Queue() for i in range(3): t = threading.Timer(1, run_function, [q, i]) t.start() while True: item = q.get() save_to_disk(item) q.task_done() def run_function(q, i): t = threading.Timer(1, run_function, [q, i]) t.start() q.put(function_with_delay(i))

edited Jan 16, 2012 at 4:16

answered Jan 16, 2012 at 3:59

srgerg

19.4k4 gold badges59 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user244343 Over a year ago

+1 This works because the output is to separate files. If it was non-trivial and involved a combination of outputs then you would have to wait for all of the queued items to finish before writing. Make sure to call task_done on the items you get as well.

Alex L Over a year ago

Great idea! Queue is thread-safe and is much nicer than list.append(data)

srgerg Over a year ago

I added the call to task_done as suggested by darvids0n. Thanks.

score 1 · Accepted Answer · 2012-01-16 03:58:39Z

I would say store a list of lists (bool, str), where bool is whether the function has finished running and str is the output. Each function locks the list with a mutex to append output (or if you don't care about thread safety omit this). Then, have a simple polling loop checking if all the bool values are True, and if so then do your save_to_disk calls.

Community · Accepted Answer · 2017-05-23 10:24:23Z

Another alternative would be to implement a class (taken from this answer) that uses threading.Lock(). This has the advantage of being able to wait on the ItemStore, and save_to_disk can use getAll, rather than polling the queue. (More efficient for large data sets?)

This is particularly suited to writing at a set time interval (ie every 30 seconds), rather than once per second.

class ItemStore(object): def __init__(self): self.lock = threading.Lock() self.items = [] def add(self, item): with self.lock: self.items.append(item) def getAll(self): with self.lock: items, self.items = self.items, [] return items

Collectives™ on Stack Overflow

Run multiple functions every second, write result to file

3 Answers 3

3 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Linked

Related