Linked Questions

I am working with DNA sequences of length 25 (see examples below). I have a list of 230,000 and need to look for each sequence in the entire genome (toxoplasma gondii parasite). I am not sure how ...

I'm processing large CSV files (on the order of several GBs with 10M lines) using a Python script. The files have different row lengths, and cannot be loaded fully into memory for analysis. Each ...

I'm writing a multiprocessing program to handle a large .CSV file in parallel, using Windows. I found this excellent example for a similar problem. When running it under Windows, I receive an error ...

I am trying to implement a multiprocessing approach for reading and comparing two csv files. To get me started, I started with the code example from embarassingly parallel problems, which sums ...

Scenario: In my application, there are 3 processes which are copying documents on a shared drive in their respective folders. As soon as any document is copied on shared drive (by any process), ...

I'm using python to write an ideal gas simulator, and right now the collision detection is the most intensive part of the program. At the moment though, I'm only using one of my 8 cores. (I'm using ...

I've never worked with multiprocessing before so bear with me if I'm asking a basic question. This answer provided a very nice processing class that I adapted to my needs and it works very well. I'm ...

I have a simple python function performing itertools product function. As seen below. def cart(n, seq): import itertools b = 8 while b < n: n = n - 1 for p in itertools....

Hi I have just started programming in python and I am trying to use subprocess.Popen to run multiple instances of a program that i compile using "make". But before i do a "make", I have to do some ...

I am coding a little web scraper where I would like to implement multiprocessing / multi-threading. I have written my function webScraper() which receives a String with a website URL as input, ...

My question is inspired by a comment on the solving embarassingly parallel problem with multiprocessing post. I am asking about the general case where python multiprocessing is used to (1) read data ...

I have a simulation running inside a class "Simulation", and a class "DataRecorder" in charge of saving data on disk (after several manipulations). Here is a simplified mockup: class DataRecorder(...

I have created a script that : Imports a list of IP's from .txt ( around 5K ) Connects to a REST API and performs a query based on the IP ( web logs for each IP) Data is returned from the API and ...

I quite often write simple optimization routines that look something like this: def createinstance(n): while(True): #create some instance called instance yield instance loopno = ...

I have seen other related questions (like this one) but none of them actually answers my questions, so here it goes: I have an obviously embarassingly parallel task to perform, my own rolled version ...