1

Let two process function1, function2 are running at the same time. function1// continuously appending the list function2// take that the list from function1 and get all the data from the list and copy to another list, flush the original list and process that copied list.

sample code: list_p =[] def function1(data): list_p.append(data) def function2(list_p): list_q = list_p.copy() list_p.flush() x= process(list_q) return x while True: //coming data continously function1(coming data)

So, how to work with both function1 and function2 at a time so that I can get the data from function1 and flush it (after flushing start appending the index in function1 from 0) Also, at the same time list could be appending in function1.

At the same time, function1 could be appending the list and function 2 could be processing the new list, after finishing function2's process, It again takes all the data in the original list that was appending while function2 was processing.

continue..

Improve this question
11
  • Please read about Python GIL. Python is probably not the best language to do this task. If you have to use Python, maybe don't do it in threads, do it in loop or use persistent storage between functions. Commented Nov 27, 2019 at 8:59
  • 1
    You could use Queue class for it, it supports multithreading: docs.python.org/3/library/queue.html You can also google for "Producer consumer problem" Commented Nov 27, 2019 at 9:34
  • If you don't use Queues ar any other synchronisation mechanism, then your code would always use 100% CPU, as function2 executes continiously even if list_p is empty. With Queues function 2 would be sleeping until there is really data to process. As @Alex mentioned: performance wise multithreading with might not necessarily increase your performance, but this depends on where incoming data is coming from and what functions process() is calling. Commented Nov 27, 2019 at 9:42
  • here is an example with a Thread class: techmonger.github.io/55/producer-consumer-python Commented Nov 27, 2019 at 9:48
  • @Alex Can you please explain or share any link related to "persistent storage between functions"? Commented Nov 27, 2019 at 9:54

1 Answer 1

Reset to default
1

Here is an example using Threading. In place of data stream I used input function in producer. (It's based on https://techmonger.github.io/55/producer-consumer-python/.)

from threading import Thread from queue import Queue q = Queue() final_results = [] def producer(): while True: i = int(input('Give me some number: ')) # here you should get data from data stream q.put(i) def consumer(): while True: number = q.get() result = number**2 final_results.append(result) print(final_results) q.task_done() t = Thread(target=consumer) t.daemon = True t.start() producer()
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you so much, @marke; This is really helpful. You saved my day :P

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.