I got a page last night about the load on one of my customer's servers being pretty high. I jumped on to find some random processes that were running as the apache user. Finding these processes isn't too difficult as long as you know what user apache typically runs as and where to find the PID of the real parent apache process. Here's an example:
$ ps -aef | grep apache | grep -v `cat /var/run/httpd.pid`
I worked a problem for a customer pretty-much all day today. After apache started, the site would run fine for a while until a large number of keep-alive connections would establish and some large-scale streams would start. A quick look at mod_status would show some 1800 concurrent apache connections. The crazy thing about it all is that all of them were in "W-state" or "Sending Reply".
I woke up this morning to a screen full of IM's from a customer saying their site had went down over night. He suggested that within 5 minutes of my being awake the site would be down again. We do offering monitoring services to our clients, but I realy didn't have time to get him setup with a full-scale solution, so I baked up something really quick that I could run in a window in the upper right corner of my screen to tell me if things went south on his site:
$ while [ true ]; do
perl -MLWP::Simple -e '
I was challenged with an emergency yesterday:
The customer's site was down, my cell phone was buzzing every 15 seconds, Trillian was chiming new IM's every 5 seconds from the frantic customer, my Inbox was full of emails from this poor guy explaining that he had been experiencing random problems throughout the day. After digging around a little, I found that MySQL was dying, but he couldn't restart it.
