Apache performance degrades dramatically above ~256 simultaneous requests

view story

http://serverfault.com – I'm running a relatively low-traffic site that experiences a large spike in visitors once a week following a site update. During this spike, site performance is extremely poor compared to the rest of the week. The actually load on the servers remains very low, reliably under 10% CPU and under 30% RAM (the hardware should be complete overkill for what we're actually doing), but for some reason Apache seems to be unable to cope with the quantity of requests. We are running apache 2.2.3 on RHEL 5.7, kernel 2.6.18-274.7.1.el5, x86_64. Attempting to reproduce this behavior during off hours with ab (HowTos)