3

I am considering implementing a memory caching daemon in Go. It has potential of getting some serious memory utilization (say, Terabyte). Fragmenting into separate heaps is not a good option, I want it all in one memory space. Does anyone have experience running Go with such huge memory sizes? Is GC going to perform acceptably?

Improve this question
5
  • Given the requirements you list, I would be very tempted to write such a program in a non-GCed language. Good cache expiry is hard enough without worrying about GC as well. Commented Apr 14, 2014 at 18:54
  • Also, if your needs are simple you may not have to implement anything yourself; groupcache (github.com/golang/groupcache) may do just what you need. Commented Apr 14, 2014 at 18:57
  • @Evan I thought about groupcache; I'm pretty sure it is not what I need because I plan to put quite a bit of processing logic into my "cache" (such as aggregation). Problem is, there are not many non-GC'ed languages I can force myself to use. The thought of doing this in C++ terrifies me. Commented Apr 14, 2014 at 19:02
  • 2
    If it's a terabyte of largeish blocks, then you might choose to use mmap for the raw data and garbage collection only for the metadata. That's basically what groupcache does. If you are going to have a terabyte of 16-byte blocks, that's a different story. Commented Apr 16, 2014 at 2:01
  • @RussCox I'm thinking relatively large blocks (say, thousands for 10Mb chunks). mmap sounds interesting, but I don't see any references to mmap usage in github.com/golang/groupcache ? Commented Apr 19, 2014 at 20:55

1 Answer 1

Reset to default
3

I am trying to do the same but the only projects that gave me a good performance to cache data was the binary tree https://github.com/stathat/treap m which supported more than 1 millons of nodes on memory in one machine Ubuntu 12.0.4 LTS with 8 GB memory. Furthermore, it was fast loading and searching data.

Other projects that I tested was LMDB but not support many nodes on memory, kv, go-cache and goleveldb but no one was as faster to recovery data from memory that treap.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.