Clear cache or memory in Python

Question

I have a really large table need to calculate (10 billion + rows). It's too long to calculate all of them at one time. Therefore, I create a list of break points, and I calculate and store at the end of each step. However, each step costs longer time. I think the reason is the memory or cache, do you have any idea how to deal with in this situation or just how to clear cache or memory?

I reuse the variable for output in the loop, so the variable will not become bigger and bigger.

Thank you so much.

Cache is irrelevant, no software outside maybe the OS directly messes with that (cache is hardware-driven, not software). — Delioth
– Delioth, Commented Apr 17, 2017 at 21:20
How does the mentioned large table implemented? Is it a database table or a dictionary? — Imre Piller
– Imre Piller, Commented Apr 17, 2017 at 21:20
It's retrieved from SQL server, and transformed to pandas.DataFrame, And then calculated and stored into SQL server. @ImrePiller — Carl Zheng
– Carl Zheng, Commented Apr 17, 2017 at 21:24

Camilo · Accepted Answer · 2021-09-23 14:53:54Z

8

I use to solve this issue using a line that reset the variable at the end of the process, cleaning the cache:

MyVariable = None

edited Sep 23, 2021 at 14:53

answered Feb 21, 2019 at 13:46

Camilo

4376 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user30850 Over a year ago

How is that different from del MyVariable in terms of cleaning the cache?

Collectives™ on Stack Overflow

Clear cache or memory in Python

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related