1. Database, database, database
2. Database
3. It's probably your database unless you're doing something CPU or disk intensive, for example resizing and prettifying pictures, rendering a 3d image, etc. Realize that these days looping a few thousand times is trivial.
4. It's your database
Starting with
fault tolerance, speed, unit testing, code abstraction, etc. is starting at the wrong point. 90% of performance problems are at the database level.
What's most likely wrong with performance in a webapp:
1. No/bad indexes/missing foreign keys on your db
2. Stupid joins
3. You're doing stupidly complicated things in an ORM
4. You're doing complicated things in the DB instead of loading a large chunk into memory and doing calculations or aggregations in code. Simple DB queries are a lot faster than you think they are. Complicated ones are a lot longer than you think they'll be. There are weird gotchas in DBs like using a function in a where or select clause will cause a massive performance hit.
5. You're not caching into memory or something like memcached things that changes infrequently but are queried regularly. Memcached is actually overkill most of the time. Actually think about how much memory storing X would take compared to how much memory your machine has. Be surprised at how insignificant it is these days.
6. 1-5 are especially true if you're using Mysql - It's great and all, but all the other big DBs piss all over it for out of the box performance. You have to give it some love. I expect some dissent here. They're wrong. MS SQL is a shit ton better at handling a poorly designed db/db queries out of the box than mysql. I can't stress this enough.
tl;dr Start looking at your database performance before anything else.