I think you're underestimating the amount of work done in hospitals that can be scheduled ahead of time - there really is a ton of it, especially for older folks. Not all hospital visits are to the ER.
Also, single purpose hospitals would be a bad idea - what if you're delivering a baby and the mother starts bleeding profusely? They'd all need to have their own ER (and the attendant facilities). You'd really just have specialized hospitals, which already exist.