python multi-processing on writing file and return value

Question

After reading data from database, I want to return data and write it to a csv file. How to use multi-processing to do it?

def get_data(): data = get_data_from_database() #a dataframe data.to_csv('data.csv', index=False) #step 1: write to csv file return data #step 2: return data

How to use multi-processing to do step 1 and step 2?

There is no place for what we commonly call multi-processing here. Multi-processing (resp. multi-threading) is normally executing same task by multiple processes (resp. threads). Here you could use a pipeline: process_reading_database -> process_writing_csv_and_passing data -> process_using_data — Serge Ballesta
– Serge Ballesta, Commented Dec 14, 2017 at 8:39
Thanks. As it takes time to write the csv file, I do not want to wait for the process of writing file to finish and return. I want to concurrently writing file and return the data, not sequential processes. — Chan
– Chan, Commented Dec 14, 2017 at 8:45

Hamid Sajjadi · Accepted Answer · 2017-12-14 09:06:30Z

You mean multi-threading? If yes, then if you want to write to the file somewhere else you should use locks and it can become a pain in the code!

But if you are not worried about that you can do something like this

import threading def thread_write(data): data.to_csv('data.csv', index=False) # step 1: write to csv file def get_data(): data = get_data_from_database() #a dataframe t = threading.Thread(target=thread_write, args=(data,)) #pass your function as target of thread, and it's input variables as a tuple to args t.start() return data # step 2: return data

Hafsa Elif Özçiftci · Accepted Answer · 2017-12-14 08:31:32Z

Try this;

def get_data(): data = [] data = get_data_from_database() #a dataframe if len(data) >0: for i in range (len(data)): data[i].to_csv('data.csv', index=False) #step 1: write to csv file return data #step 2: return data

Thanks. I want to to write csv file and return data simultaneously. How to do it?

Collectives™ on Stack Overflow

python multi-processing on writing file and return value

2 Answers 2

1 Comment

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Related