My favorite anecdote: asking tptacek to help me me squeeze a few megs out of Redis to avoid having to bump my VPS up a tier. That would have been, heck if I know, $BOATLOADS / hr in engineer costs to avoid $30 per month on my Slicehost bill. He restored me to sanity fairly quickly.
The real question is which is more valuable to your business.
Consider these two examples:
Let's say I'm a bootstrapper, and have a startup that earns $3k monthly. My customers are happy, and my business is growing, but I'm having some serious scaling problems.
Scenario One: I've outgrown the small Linode that I started on. If doubling the cost of my Linode allows me to service 2X the number of customers, I do it, because $3k in revenue is way higher than the cost.
Scenario Two: I've maxed out the biggest Linode, and the next step up is a cluster of dedicated nodes at Rackspace for $4k/month, which will allow me to scale to 2X my current volume. At this point, I fix the code, because my earnings will drop even if I get twice the number of customers.