I was tasked with making a program that uploads a .csv to a NoSQL cluster. We were doing 2-5 gig files, and it was taking 4-8 hours. Well when we hit 17 gig files that was just to slow.

When I remade my program I realized working in batch made is much much faster. I could get a 17 gig file done in 6 hours. So then I decided to make a consumer-producer multithreading structure. This caused it to be just as slow as before. Although my program is fast and working great, I want to know why the producer-consumer construct was slower than a batch produce, batch consume method.

As compaired to

The bottom half are infinite loops, but I just left it like that for a speed test.

I was tasked with making a program that uploads a .csv to a NoSQL cluster. The files are larger (typically 2-17GB). My program works in batch mode and can process a 17GB file in 6 hours. 

I decided to make a consumer-producer multithreading structure. This caused it to be significantly slower. I want to know why the producer-consumer construct was slower than a batch produce, batch consume method.

As compared to

The bottom half are infinite loops for a speed test.

EDIT: The bottom half are infinite loops, but i just left it like that for speed test.

The bottom half are infinite loops, but I just left it like that for a speed test.

I was tasked with making a program that uploads a csv to a nosql cluster. We were doing 2-5 gig files, and it was taking 4-8 hours. Well when we hit 17 gig files that was just to slow.

When I remade my program I realized working in batch made is much much faster. I could get a 17 gig file done in 6 hours. So then i decided to make a consumer-producer multithreading structure. This caused it to be just as slow as before. Although my program is fast and working great, I want to know why the producer-consumer construct was slower than a batch produce, batch consume method.

I was tasked with making a program that uploads a .csv to a NoSQL cluster. We were doing 2-5 gig files, and it was taking 4-8 hours. Well when we hit 17 gig files that was just to slow.

When I remade my program I realized working in batch made is much much faster. I could get a 17 gig file done in 6 hours. So then I decided to make a consumer-producer multithreading structure. This caused it to be just as slow as before. Although my program is fast and working great, I want to know why the producer-consumer construct was slower than a batch produce, batch consume method.