2

I'm looking for option something similar to -Xmx in Java, that is to assign maximum runtime memory that my Go application can utilise. Was checking the runtime , but not entirely if that is the way to go.

I tried setting something like this with func SetMaxStack(), (likely very stupid)

debug.SetMaxStack(5000000000) // bytes model.ExcelCreator()

The reason why I am looking to do this is because currently there is ample amount of RAM available but the application won't consume more than 4-6% , I might be wrong here but it could be forcing GC to happen much faster than needed leading to performance issue.

What I'm doing

Getting large dataset from RDBMS system , processing it to write out in excel.

Another reason why I am looking for such an option is to limit the maximum usage of RAM on the server where it will be ultimately deployed.

Any hints on this would greatly appreciated.

Improve this question
7
  • 1
    What is the problem you're trying to solve? It sounds like your program just isn't using as much memory as you expect--but why is that problematic? Commented Oct 1, 2018 at 14:20
  • Go will use the amount of memory it needs, what are you expected to take up the memory if the program doesn't need it? Commented Oct 1, 2018 at 14:21
  • 1
    Have it piecemeal. Start with disabling Excel-everything first—that is, just crunch the data and throw the results away (not literally—make sure you, say, fmt.Print() them into io/ioutil.Discard for the compiler to not optimize your code away completely), and measure how it goes. If it's slow, profile, or at least come up with a simple MCVE for us to squint at your code. Commented Oct 1, 2018 at 14:44
  • 1
    @JimB As mentioned in my question, and i may be wrong about it, but i think that since go is not really using much memory and likely trying to work with limited (if there is a default memory consumption defined) memory and calling GC too often . Commented Oct 1, 2018 at 14:44
  • 1
    So, in the end, this question appears as a classical case of the XY Problem. @sup, please consider first breaking your problem down into logical pieces—fetching data from the DBMS, processing it, making an Excel document of it,—then profile, then try to come up with different question(s) with the concrete problems you will have identified by then. Commented Oct 1, 2018 at 14:46

1 Answer 1

Reset to default
4

The current stable Go (1.10) has only a single knob which may be used to trade memory for lower CPU usage by the garbage collection the Go runtime performs. This knob is called GOGC, and its description reads

The GOGC variable sets the initial garbage collection target percentage. A collection is triggered when the ratio of freshly allocated data to live data remaining after the previous collection reaches this percentage. The default is GOGC=100. Setting GOGC=off disables the garbage collector entirely. The runtime/debug package's SetGCPercent function allows changing this percentage at run time. See https://golang.org/pkg/runtime/debug/#SetGCPercent.

So basically setting it to 200 would supposedly double the amount of memory the Go runtime of your running process may use.

Having said that I'd note that the Go runtime actually tries to adjust the behaviour of its garbage collector to the workload of your running program and the CPU processing power at hand. I mean, that normally there's nothing wrong with your program not consuming lots of RAM—if the collector happens to sweep the garbage fast enough without hampering the performance in a significant way, I see no reason to worry about: the Go's GC is one of the points of the most intense fine-tuning in the runtime, and works very good in fact.

Hence you may try to take another route:

  • Profile memory allocations of your program. Analyze the profile and try to figure out where the hot spots are, and whether (and how) they can be optimized.

    You might start here and continue with the gazillion other intros to this stuff.

  • Optimize. Typically this amounts to making certain buffers reusable across different calls to the same function(s) consuming them, preallocating slices instead of growing them gradually, using sync.Pool where deemed useful etc.

    Such measures may actually increase the memory truly used (that is, by live objects—as opposed to garbage) but it may lower the pressure on the GC.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

-Xmx isn't really related to GC though, it's just the max heap size, which of course you can't really adjust in Go.
@JimB, sure, but a more relevant knob is likely to appear—see the sort-of announce and the relevant issue. It won't be about limiting the heap usage (which is useless anyway—as any seasoned Java developer knows from hard-won experience) but rather about providing feedback to the running program about reaching the memory limits.
@kostix thank you for the explanation, this defintely helps in furthering my understanding. Will be working on what you suggested / commented above.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.