20

Having some problems with httpd (Apache/2.2.29) memory usage.

Over time, memory usage in the httpd processes creep up until it's eventually at 100%.

Last time I restarted httpd was about 24 hours ago. Output from free -m is:

[ec2-user@www ~]$ free -m total used free shared buffers cached Mem: 1655 1415 239 0 202 424 -/+ buffers/cache: 788 866 Swap: 1023 4 1019

To prove that it's definitely httpd, I restarted httpd and ran free -m again:

[ec2-user@www ~]$ sudo service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [ec2-user@www ~]$ free -m total used free shared buffers cached Mem: 1655 760 894 0 202 360 -/+ buffers/cache: 197 1457 Swap: 1023 4 1019

So, restarting Apache takes free memory from 239 Mb to 894 Mb - which seems like a big leap.

I've been going through the list of currently enabled Apache modules (there's quite a lot) and disabled/removed mod_wsgi and mod_perl (neither of which are required for this server, which is running a PHP-based web application - Magento, specifically).

Based on https://servercheck.in/blog/3-small-tweaks-make-apache-fly, I've run ps aux | grep 'httpd' | awk '{print $6/1024 " MB";}' and get the following output:

[root@www ~]# ps aux | grep 'httpd' | awk '{print $6/1024 " MB";}' 15.1328 MB 118.09 MB 127.449 MB 129.059 MB 117.734 MB 113.824 MB 125.062 MB 123.922 MB 119.855 MB 108.066 MB 136.23 MB 114.031 MB 113.27 MB 110.695 MB 102.113 MB 113.234 MB 186.816 MB 118.602 MB 0.835938 MB

Running the other suggested diagnosis tool for MaxClients which is ps aux | grep 'httpd' | awk '{print $6/1024;}' | awk '{avg += ($1 - avg) / NR;} END {print avg " MB";}' returns the following:

[root@www ~]# ps aux | grep 'httpd' | awk '{print $6/1024;}' | awk '{avg += ($1 - avg) / NR;} END {print avg " MB";}' 110.212 MB

This server (Amazon AWS m1.small instance) has 1.7 Gb of RAM. So, therefore:

Any further pointers/suggestions on how best to tweak the httpd settings or how to diagnose what exactly might be causing this?

Improve this question
5
  • Note that to check memory usage by processes you need to look at the -/+ buffers/cache line; however in this case the change is comparable. Depending on how apache has been tuned, it may start with just a couple of processes ready for handling requests; after a period of many concurrent requests there will probably more processes forked to handle the load. The number of idle processes can also be tuned. So if apache grows to use too much memory, you need to do some tuning. Commented Feb 27, 2015 at 10:28
  • I'm OK at Apache server configuration but haven't done a huge amount in terms of performance tuning for Apache in the past. Any suggestions on which directives I should be looking at or guides? Is there a configuration setting I can enable to tell Apache to kill off forked processes after a certain amount of time? (It seems that Apache doesn't do this by default for some reason.) Commented Feb 27, 2015 at 23:51
  • This is the exact reason I've moved away from Apache... try Nginx or even Lighttpd. Commented Feb 28, 2015 at 7:46
  • Interesting. Heard good things about nginx so maybe it's time we give it a try, but honestly, I've been using Apache for years and never had any major problems. Magento really chews memory though. Commented Feb 28, 2015 at 10:23
  • 1
    Note that your script to calculate the average memory usage includes the low memory usage of the grep httpd-process (at least for me). To fix that i would recommend to exclude that line from the calculation like this: ps aux | grep 'httpd' | grep -v grep | awk '{print $6/1024;}' | awk '{avg += ($1 - avg) / NR;} END {print avg " MB";}' Commented Apr 4, 2019 at 8:50

3 Answers 3

Reset to default
14

Here's what I've done to 'solve' it:

  1. Set MaxClients 7 (based on (1740.8Mb Memory on server - 900Mb for MySQL + other stuff) / 111Mb average usage per httpd process = 7.5747747747747747747747747747748)

Therefore:

<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 7 MaxRequestsPerChild 4000 </IfModule>

  1. Disable all Apache modules except for authz_host_module, log_config_module, expires_module, deflate_module, setenvif_module, mime_module, autoindex_module, negotiation_module, dir_module, alias_module, rewrite_module, php5_module

  2. Remove the mod_ssl package since the client isn't using https:// whatsoever.

I'll report back once this new configuration has been running a while to see if this solves it.

Some inspiration here was borrowed from: http://www.activoinc.com/blog/2009/08/31/performance-optimized-httpd-conf-for-magento-ecommerce/ and http://www.activoinc.com/downloads/httpd.conf-magento

Improve this answer
2
  • Above configuration has definitely helped massively. Commented Feb 28, 2015 at 10:24
  • 1
    Couple days later, 'memory leak' problems in httpd seem to be gone completely. Commented Mar 4, 2015 at 6:07
3

I'm afraid option MaxRequestsPerChild helped in your case, as it's enables process recycling after defined number of requests, so memory leak is there, but not visible anymore.

Additionally: MaxClients = ServerLimit * ThreadsPerChild

In your case if you need only 7 concurrent users (MaxClients=7) it's totally enough with 2 process (just in case if one will fail to minimize downtime), so config can be:

<IfModule prefork.c> StartServers 2 MinSpareServers 2 MaxSpareServers 20 ServerLimit 2 MaxClients 8 ThreadsPerChild 4 MaxRequestsPerChild 4000 </IfModule>

I use MaxClients 8, just to make more equal request distribution between 2 processes.

Improve this answer
1
  • In my case, I did not need this configuration if I wait a bit less than 24 hours, the memory will be freed with the default Apache configuration (checked with htop command after viewing analytics "Visits Over Time" for several years in Piwik / Matomo). But do you know where the memory leak could come from? Apache server or the PHP code running with mod_php? I guess it is better to fix the memory leak than trying to hide it with a specific configuration. Commented Apr 18, 2020 at 8:56
1

Memory computation

The other answers already tackle how to set the values. And for it you need to compute the memory usage of different elements of your system.

I want to tackle memory computation which can help many.

  • ps show processes only and is a good tool to use if you want to use RSS as a value
    • RSS (Resident Set Size) over report memory, due to adding the whole shared memory (shared libraries) and repeatedly through all processes.
  • htop or top by default show not only all processes but all threads of a given entity

    To show process only press H to toggle between showing threads and not showing them

Smem

For a better tool for computing memory usage, You can use smem

apt install smem

it will give pss value which it's better to rely on:

Provided values

  • RSS (Resident Set Size)
    • ram memory allocated to process Including shared memory (repetitive) shared libraries allocated once, used by many processes RSS over report memory usage
  • PSS (Proportional Set Size)
    • divide the shared memory by percentage through all processes
    • 4 processes using same shared lib, they wil be computed with 25% each from the shared memory
    • in contrast to RSS, which for each process it compute 100% of that share memory
  • USS (Unique Set Size)
  • USS and PSS are terms and concepts that were proposed by Matt Mackall, the author of smem.

Some options

  • k : memory in mb
  • t: show total memory in the bottom 🔥

Filtering

  • process
  • mapping (libraries)
  • user

Getting apache2, mysql, php-fpm, nginx memory usage

Memory for apache2 process with total computation as well

smem -tk -P apache2

Same but only selecting pss column

smem -tk -c pss -P apache2

Showing pss total only

smem -tk -c pss -P apache2 | tail -n 1

Change apache2 with any of the other processes.

Example

enter image description here

You can see each process with it's child processes consumption.

PSS vs RSS

If we want to tune to the max of the server capabilities. PSS may be a better value to use.

Otherwise, we can use RSS as it over-reports memory. And that's can be safer. And then one can tune better under stress testing. Through experimentation.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.