It's really hard to keep moving all the way up and down the stack.The main advantage, to me, is being able to collapse layers of abstraction. We've done that at one of the startup's I CTO for.
On the backend, we eliminated the entire multi-tier stack we've come to know and love from the late 90s. The database, application server, caching server, and authentication server were collapsed down into a single executable.
We dropped TCP, and went to UDP + NaCl-based crypto. This in turn changed how we did session management and logins (we only target mobile devices), and allowed us to gain further performance by bypassing the Linux kernel entirely, and talking directly to the Ethernet hardware. Our wire-to-wire latency for a UDP packet (without any processing, just heading through the LuaJIT app framework) is ~26 nanoseconds. We measure the entire request/response time in less than 10,000 nanoseconds for updates, and frequently, less than 3000 nanoseconds for reads.
Nanoseconds! Today's hardware is crazy efficient, but yesterdays software architectures waste it all.
For caching, we employ a database library (lmdb) that uses the file system and memory mapping for all data, a lot like Varnish Cache does. We can service reads without a single malloc() call. No more memcached.
Finally, the entire system is built around an event streaming approach (see Disruptor/LMAX). For that to work, we wrote our own cross-platform Core Data alternative on the client, and for this particular application (a social network), the end result is that the user experiences ZERO network latency for everything but search.
Everything is local by the time the user is made aware of it. For updates like making a post, we make it happen instantly on the client, speculatively, and eventually the update makes it's way to the server and back, and then on to everyone else. The user doesn't even need a live network connection to post, and "offline" usage works as expected.
And also (since this is a speciality of mine), our object graph syncing is multi-device from day one. Lots of projects (e.g. Vesper) struggle with that one. We didn't bolt it on, it's a fundamental property of our data model and overall system.
After collapsing the crypto, database, and authentication, we were able to introduce object capabilities that requires zero memory lookups, instead of the usual RBAC, which in my experience is difficult to implement efficiently and hard for people to manage.
And because we have our own network protocol, and are targeting native clients we control, we can easily do client-side load balancing. We use consistent hashing to contact read-only servers (that also handle crypto), mostly so we can spread out the bandwidth without having a crazy expensive load balancer. Thanks to our custom database/app server, we can service all writes on a single machine, although like LMAX, it's designed to run three in parallel, discarding the results of two of them at any given time.
Point is, there are HUGE gains to be made when you (a) understand the whole stack, (b) have the skills and understanding to rewrite any part as you need to, and (c) have the guts to ignore 15 years of received wisdom on how to scale an app and are willing to collapse layers down in pursuit of speed and simplicity.
It may be harder, but the payoff is enormous. I did all of the above myself in just over four months, including building a cross-platform app framework for iOS (porting to Android in March), and the app itself.