It does run on Debian, I think you can just ssh in an do everything you want. By the way, on DD-WRT/OpenWRT/Tomato you can install pretty much what you want by using OptWare/EntWare.
The problem is usually not the software, but the hardware. Router are still slow (for example, I tried once to use transmission on my puny e3200, and the speed was capped at ~350kbps by the CPU) and they don't have lots of connectivity (on high-end routers you get maybe 2 USB 3.0 ports. The WRT1900AC is in fact the first router with an eSata port, and the first one with decent speed over USB).
If you want to run anything somewhat "heavy", you're far better with having a different box, or using a custom pfsense/smoothwall/m0n0wall/vyatta box.
It's both. The hardware is slow and the software (in the cases you mentioned, the Linux network stack) is slow and inefficient.
The ERL uses hardware acceleration to avoid all this. It's capable of routing 1M PPS and the latest beta build does this with PPPoE as well. (In other words, it'll support Google Fiber connections well, for the lucky few.)