We're talking about systems designed for throughput. For processing huge amounts of data in overnight batch processes. For five nines uptime. And for still being compatible with the code you wrote 30 years ago that still manages millions of customers.
It IS incredible - but modern Computers are even more incredible.
The reason why these are kept around is that the custom software that runs on them often implements very complex business rules that are otherwise undocumented (the code is the documentation). If you're talking about a financial institution, the mainframe IS the business.
As far as reliability, these things are rock-solid and proven, but mainstream distributed systems design has pretty recently surpassed the mainframe model IMHO. It's relatively simple (compared to just a few years ago) to implement clustered data processing using MapReduce, replicate the data across multiple datacenters, and not care about individual node failure.
You have every piece of I/O handled by its own subsystem; in the old days when they had lots of green screens they had FEPs, front end processors, that took care of all the busywork for displaying and accepting data - the main CPU never saw any of that. Same thing with networking, with disk I/O, you name it.