Why does python seem to allocate more memory than sys.getsizeof accounts for?

Question

Example:

import sys class Test(): def __init__(self): self.a = 'a' self.b = 'b' self.c = 'c' self.d = 'd' self.e = 'e' if __name__ == '__main__': test = [Test() for i in range(100000)] print(sys.getsizeof(test))

In windows task manager: I am getting a jump of ~20 MB when creating a list of 100000 vs 10.

Using sys.getsizeoff(): For a list of 100000, I get 412,236 bytes; for a list of 10, I get 100 bytes.

This seems hugely disproportionate. Why is this happening?

You can use iterators (xrange in this case) to save memory — neoascetic
– neoascetic, Commented Jul 31, 2012 at 23:02
@Daniel DiPaolo: Windows task manager defines it as: Memory (Private Working Virtual Set). (And its more than my internet explorer is using right now!) — Jeff
– Jeff, Commented Jul 31, 2012 at 23:06
sys.getsizeof returns a shallow size: it doesn't include the objects contained in the list. — interjay
– interjay, Commented Jul 31, 2012 at 23:10
@DanielDiPaolo: If you're not sure, read the documentation. The size includes the memory allocated by the list itself (which contains pointers) but not the objects pointed to. — interjay
– interjay, Commented Jul 31, 2012 at 23:13

Community · Accepted Answer · 2017-05-23 10:33:29Z

The memory assigned is not disproportional; you are creating 100,000 objects! As you can see, they take up roughly 34 megabytes of space:

>>> sys.getsizeof(Test())+sys.getsizeof(Test().__dict__) 344 >>> (sys.getsizeof(Test())+sys.getsizeof(Test().__dict__)) * 1000000 / 10**6 34.4 #megabytes

You can get a minor improvement with __slots__, but you will still need about 20MB of memory to store those 100,000 objects.

>>> sys.getsizeof(Test2())+sys.getsizeof(Test2().__slots__) 200 >>> sys.getsizeof(Test2())+sys.getsizeof(Test2().__slots__) * 1000000 / 10**6 20.0 #megabytes

(With credit to mensi's answer, sys.getsizeof is not taking into account references. You can autocomplete to see most of the attributes of an object.)

See SO answer: Usage of __slots__? http://docs.python.org/release/2.5.2/ref/slots.html

To use __slots__:

class Test2(): __slots__ = ['a','b','c','d','e'] def __init__(self): ...

This has nothing to do with slots, it has to do with the memory usage of the Python interpreter when allocating objects. Slots are a way to reduce that usage, but he's not using them here.
@ninjagecko: I tried adding the __slots__ line of code into the class definition, but it seems to make little if any difference in memory usage. (Py 2.7)
@Jeff: are you measuring this with the same thing you used to measure "20MB", or with sys.getsizeof?
@ninjagecko: I stand corrected as far as sys.getsizeof is concerned. But it does not resolve the issue of how much ram it is gobbling, according to task manager (which remains about the same with or without __slots__)
@Jeff: there's nothing weird going on here: it's math. Your 100000 objects take up in total roughly 20 megabytes of space. See mensi's answer or this revised answer. (Daniel DiPaolo is incorrect.) You should edit your answer's title to be "sys.getsizeof does adequately account for most objects"; the memory assigned is not disproportional.

mensi · Accepted Answer · 2012-07-31 23:09:19Z

Every instance references a dict for it's __dict__ which is 272 bytes on my machine for your example. Multiply that by 100'000.

Collectives™ on Stack Overflow

Why does python seem to allocate more memory than sys.getsizeof accounts for?

2 Answers 2

6 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Linked

Related