A little while ago, we ran into memory problems on mahara.org. It turned out to be due to the GD library having issues with large (as in height and width, not file size) images.
What we discovered is that the PHP memory limit (which is set to a fairly low value) only applies to actual PHP code, not C libraries like GD that are called from PHP. It's not obvious what PHP libraries are implemented as external C calls, which fall outside of the control of the interpreter, but anything that sounds like it's using some other library is probably not in PHP and is worth looking at.
To put a cap on the memory usage of Apache, we set process limits for the main Apache process and all of its children using ulimit.
Unfortunately, the limit we really wanted to change (resident memory or "
-m") isn't implemented in the Linux kernel. So what we settled on was to limit the total virtual memory that an Apache process (or sub-process) can consume using "
On a Debian box, this can be done by adding this to the bottom of
ulimit -v 1048576
for a limit of 1GB of virtual memory.
You can ensure that it works by setting it first to a very low value and then loading one of your PHP pages and seeing it die with some kind of malloc error.
I'm curious to know what other people do to prevent runaway Apache processes.