How to track memory usage accurately?

Question

I am trying to build a small tool that will allow me to run a program and track memory usage through Go. I am using r.exec = exec.Command(r.Command, r.CommandArgs...) to run the command, and runtime.MemStats to track memory usage (in a separate go routine):

func monitorRuntime() { m := &runtime.MemStats{} f, err := os.Create(fmt.Sprintf("mmem_%s.csv", getFileTimeStamp())) if err != nil { panic(err) } f.WriteString("Time;Allocated;Total Allocated; System Memory;Num Gc;Heap Allocated;Heap System;Heap Objects;Heap Released;\n") for { runtime.ReadMemStats(m) f.WriteString(fmt.Sprintf("%s;%d;%d;%d;%d;%d;%d;%d;%d;\n", getTimeStamp(), m.Alloc, m.TotalAlloc, m.Sys, m.NumGC, m.HeapAlloc, m.HeapSys, m.HeapObjects, m.HeapReleased)) time.Sleep(5 * time.Second) } }

When I tested my code with simple program that just sits there (for about 12 hours), I noticed that Go is constantly allocating more memory: System Memory Heap Allocation

I did a few more tests such as running the monitorRuntime() function without any other code, or using pprof, such as:

package main import ( "net/http" _ "net/http/pprof" ) func main() { http.ListenAndServe(":8080", nil) }

Yet I still noticed that memory allocation keeps going up just like in the graphs.

How can I accurately track memory usage of the program I want to run through Go? I know one way, which I used in the past, is to use /proc/$PID/statm, but that file doesn't exist in every operating system (such as MacOS or Windows)

Note that every step of that for loop is going to allocate some memory (mostly because of WriteString, iirc), and that memory will persist until garbage collection. Saw the same thing in my own loop, which allocated a hundred or so bytes per loop iteration until garbage collection. It's not a leak, it's just normal allocation. — Kaedys
– Kaedys, Commented Jan 31, 2017 at 16:00
@VaD: ReadMemStats is as accurate as you can get, as that shows all allocations made by the Go runtime. Maybe track the output of gctrace to see how memory is being collected on your system. — Mr_Pink
– Mr_Pink, Commented Jan 31, 2017 at 16:37
@Kaedys I did some research before asking that question, and I arrived to the same conclusion. Even though it occurs in simpler programs such as writing the output to stdout instead of a file, or using pprof. But I still cannot figure out how I can get accurate memory usage tracking of the command I run through Go (using exec.Command), I want to be able to tell if that command is leaking memory. — VaD
– VaD, Commented Jan 31, 2017 at 16:39

Garet Halliday · Accepted Answer · 2022-09-12 23:52:21Z

There isn't a way in standard Go to get the memory usage of a program called from exec.Command. runtime.ReadMemStats only returns memory tracked by the go runtime (which, in this case, is only the file handling and sprintf).

Your best bet would be to execute platform specific commands to get memory usage.

Bill Zelenko · Accepted Answer · 2018-10-31 15:40:29Z

0

On Linux (RedHat) the following will show memory usage: ps -No pid,comm,size,vsize,args:90

answered Oct 31, 2018 at 15:40

Bill Zelenko

2,8881 gold badge22 silver badges27 bronze badges

1 Comment

Myonaiz Mar 17 at 13:35

Unfortunately, this will not be accurate. it will report only the memory hogged by the runtime and not the actual memory used by the application. When an application frees memory or GC frees memory, the runtime will not necessarily return it back to the OS.

Isaac Weingarten · Accepted Answer · 2023-11-22 22:05:23Z

check out this library https://github.com/shirou/gopsutil/

there example is

 package main import ( "fmt" "github.com/shirou/gopsutil/v3/mem" // "github.com/shirou/gopsutil/mem" // to use v2 ) func main() { v, _ := mem.VirtualMemory() // almost every return value is a struct fmt.Printf("Total: %v, Free:%v, UsedPercent:%f%%\n", v.Total, v.Free, v.UsedPercent) // convert to JSON. String() is also implemented fmt.Println(v) }

this example is managing it self maybe you can modify it, to manage the specific app you started with your app

Collectives™ on Stack Overflow

How to track memory usage accurately?

3 Answers 3

Comments

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Linked

Related