25

I am new to Go and trying to figure out how it manages memory consumption.

I have trouble with memory in one of my test projects. I don't understand why Go uses more and more memory (never freeing it) when my program runs for a long time.

I am running the test case provided below. After the first allocation, program uses nearly 350 MB of memory (according to ActivityMonitor). Then I try to free it and ActivityMonitor shows that memory consumption doubles. Why?

I am running this code on OS X using Go 1.0.3.

What is wrong with this code? And what is the right way to manage large variables in Go programs?

I had another memory-management-related problem when implementing an algorithm that uses a lot of time and memory; after running it for some time it throws an "out of memory" exception.

package main import ("fmt" "time" ) func main() { fmt.Println("getting memory") tmp := make([]uint32, 100000000) for kk, _ := range tmp { tmp[kk] = 0 } time.Sleep(5 * time.Second) fmt.Println("returning memory") tmp = make([]uint32, 1) tmp = nil time.Sleep(5 * time.Second) fmt.Println("getting memory") tmp = make([]uint32, 100000000) for kk, _ := range tmp { tmp[kk] = 0 } time.Sleep(5 * time.Second) fmt.Println("returning memory") tmp = make([]uint32, 1) tmp = nil time.Sleep(5 * time.Second) return }
Improve this question
2
  • 1
    Here's a link to some good article about memory management in Go: lwn.net/Articles/428100 Commented Jan 30, 2013 at 14:28
  • 1
    "I am running this code on OSx, go1.0.3." If you need to do something memory intensive using Go tip (what will become 1.1) is highly recommended. I was leery at first, but after a couple of the Go developers recommended it, it as been more stable than 1.0.3 for me, esp. in regards to memory use. Commented Feb 2, 2013 at 6:21

2 Answers 2

Reset to default
36

Currently, go uses a mark-and-sweep garbage collector, which in general does not define when the object is thrown away.

However, if you look closely, there is a go routine called sysmon which essentially runs as long as your program does and calls the GC periodically:

// forcegcperiod is the maximum time in nanoseconds between garbage // collections. If we go this long without a garbage collection, one // is forced to run. // // This is a variable for testing purposes. It normally doesn't change. var forcegcperiod int64 = 2 * 60 * 1e9 (...) // If a heap span goes unused for 5 minutes after a garbage collection, // we hand it back to the operating system. scavengelimit := int64(5 * 60 * 1e9)

forcegcperiod determines the period after which the GC is called by force. scavengelimit determines when spans are returned to the operating system. Spans are a number of memory pages which can hold several objects. They're kept for scavengelimit time and are freed if no object is on them and scavengelimit is exceeded.

Further down in the code you can see that there is a trace option. You can use this to see, whenever the scavenger thinks he needs to clean up:

$ GOGCTRACE=1 go run gc.go gc1(1): 0+0+0 ms 0 -> 0 MB 423 -> 350 (424-74) objects 0 handoff gc2(1): 0+0+0 ms 1 -> 0 MB 2664 -> 1437 (2880-1443) objects 0 handoff gc3(1): 0+0+0 ms 1 -> 0 MB 4117 -> 2213 (5712-3499) objects 0 handoff gc4(1): 0+0+0 ms 2 -> 1 MB 3128 -> 2257 (6761-4504) objects 0 handoff gc5(1): 0+0+0 ms 2 -> 0 MB 8892 -> 2531 (13734-11203) objects 0 handoff gc6(1): 0+0+0 ms 1 -> 1 MB 8715 -> 2689 (20173-17484) objects 0 handoff gc7(1): 0+0+0 ms 2 -> 1 MB 5231 -> 2406 (22878-20472) objects 0 handoff gc1(1): 0+0+0 ms 0 -> 0 MB 172 -> 137 (173-36) objects 0 handoff getting memory gc2(1): 0+0+0 ms 381 -> 381 MB 203 -> 202 (248-46) objects 0 handoff returning memory getting memory returning memory

As you can see, no gc invoke is done between getting and returning. However, if you change the delay from 5 seconds to 3 minutes (more than the 2 minutes from forcegcperiod), the objects are removed by the gc:

returning memory scvg0: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB) scvg0: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB) scvg1: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB) scvg1: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB) gc9(1): 1+0+0 ms 1 -> 1 MB 4485 -> 2562 (26531-23969) objects 0 handoff gc10(1): 1+0+0 ms 1 -> 1 MB 2563 -> 2561 (26532-23971) objects 0 handoff scvg2: GC forced // forcegc (2 minutes) exceeded scvg2: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB) gc3(1): 0+0+0 ms 381 -> 381 MB 206 -> 206 (252-46) objects 0 handoff scvg2: GC forced scvg2: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB) getting memory

The memory is still not freed, but the GC marked the memory region as unused. Freeing will begin when the used span is unused and older than limit. From scavenger code:

if(s->unusedsince != 0 && (now - s->unusedsince) > limit) { // ... runtime·SysUnused((void*)(s->start << PageShift), s->npages << PageShift); }

This behavior may of course change over time, but I hope you now get a bit of a feel when objects are thrown away by force and when not.

As pointed out by zupa, releasing objects may not return the memory to the operating system, so on certain systems you may not see a change in memory usage. This seems to be the case for Plan 9 and Windows according to this thread on golang-nuts.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

thank you for explanation, as I understood, GC is not very complete now
I have written program which uses a lot of memory, while running with go1.0.3 on OSX program did panic at the middle of the computations after next attempt to alloc more memory (near 1.5Gb). I have optimised heavily to reuse as much objects and structures as I can, but still had memory exception. Then I cloned last release of go and had build it from source. By running with latest version of go my code consumes as much memory as it needs (2.2Gb) and finishes computations successfully.
I think I ran into this kind of issue code.google.com/p/go/issues/…
Glad it worked with tip. The GC seems to be quite incomplete, yes. If you're interested, you can look at the scanblock function of the GC to see, how it finds references.
On Windows as of Go 1.1, garbage collected memory is NOT returned to the OS. Man that cost me a day to hunt down. groups.google.com/forum/#!topic/golang-nuts/vfmd6zaRQVs
|
15

To eventually (force) collect unused memory you must call runtime.GC().

variable = nil may make things unreachable and thus eligible for collection, but it per se doesn't free anything.

Improve this answer

2 Comments

This is no longer correct, you have to use FreeOSMemory to return the memory to the OS.
@OneOfOne any idea why runtime.GC() does not work anymore? FreeOSMemory calls freeOSMemory which is Implemented in package runtime, what does that mean?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.