Get rid of mod_php, replace it with a nice build of php-fpm (included in 5.3 actually) tied to nginx via fastcgi. Bam. Nginx will handle all static content SUPER fast while keeping an incredibly small memory footprint, while passing dynamic requests to the PHP side of things via fastcgi.

Example of one of my nginx configuration files, for a Wordpress site: http://paste.pocoo.org/show/256982/

This will only do so much, but I suspect it will solve a great deal of your problems. The final bit of advice that I can give is to look into any caching modules for Drupal. Simple disk caching will do the trick, just something to prevent lots of DB reads. I'm unfamiliar with Drupal, but I'd imagine there are loads of modules out there that can assist with caching.