9

When I run this and then watch the memory consumption of my ruby process in OSX Activity Monitor, the memory increases at about 3 MB/s.

If I remove the transaction it about halves the memory consumption but still, the memory footprint keeps going up. I have an issue on my production app where Heroku kills the process because of its memory consumption.

Is there a way of doing the below, in a way that won't increase memory? If I comment out the .save line then it's okay but of course this isn't a solution.

ActiveRecord::Base.transaction do 10000000.times do |time| puts "---- #{time} ----" a = Activity.new(:name => "#{time} Activity") a.save!(:validate => false) a = nil end end

I am running this using delayed_job.

Improve this question
4
  • Finetune your garbage collector but it's a tricky job... Commented May 5, 2012 at 14:08
  • 1
    @Morgz What versions of Ruby and Rails are you using? Commented May 5, 2012 at 14:13
  • just curious, what happens if you put it into an array instead and then remove the item from the array? Commented May 5, 2012 at 14:35
  • @K2xL I'd assume as similar memory usage pattern would be observed, albeit with a slowdown in execution time due to minor overhead involved with resizing the array. Commented May 5, 2012 at 14:42

2 Answers 2

Reset to default
5

The a = nil line is unnecessary and you can remove that.

You're creating a lot of objects every time you loop - two strings, two hashes, and an Activity object so I'm not surprised you're experiencing high memory usage, especially as you're looping 10 million times! There doesn't appear to be a more memory efficient way to write this code.

The only way I can think of to reduce memory usage is to manually start the garbage collector every x number of iterations. Chances are Ruby's GC isn't being aggressive enough. You don't, however, want to invoke it every iteration as this will radically slow your code. Maybe you could use every 100 iterations as a starting point and go from there. You'll have to profile and test what is most effective.

The documentation for the GC is here.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

based on my knowledge, you can only "suggest" Ruby to run GC. but it still might work
@K2xL I'm not so sure about that. You can force GC by calling GC.start. I would love to be shown otherwise though. There's nothing in the documentation to indicate that calling GC.start is advisory.
Hey Matt. Interestingly.. calling GC.start within the loops keeps memory from spiralling out of control, but yes it does effect performance a lot! So I thought I'd try invoking it every 1000 times. GC.start if time%1000 == 0 - This doesn't work and memory spirals up and out of control! So at moment only option is to GC every times :-/
An answer to this building on what Matt has suggested is to do this: GC.start if time%5 == 0 It runs the collector every 5 times. This helps reduce my bloat sufficiently but is in no way a good enough answer!
@Morgz You're doing something relatively extreme in an interpreted language. This comes with the territory. You might want to look at GC.malloc_allocated_size and only call it when the allocated size is excessive. You could also probably tweak things a little further by disabling GC before the beginning of the loop and re-enabling it afterwards. This will probably show some performance improvement, but remember that immediately before calling GC.start you'll have to re-enable GC, disabling it as soon as the call has returned. If this doesn't make sense I'll write a code sample.
1

I know this is an ancient issue, but I have to suggest another radical approach:

ActiveRecord::Base.transaction do 10000000.times do |time| puts "---- #{time} ----" sql = <<SQL INSERT INTO activities ("name") VALUES ("#{time}") SQL Activity.connection.execute(sql) end end

The point is that if the insert is that simple, and you're already skipping any ActiveModel validation, there's no reason to instantiate an activerecord object in the first place. Normally it wouldn't hurt, but since it is hurting you in this case, I think you'll use a lot less memory this way.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.