I think there's a few different things to think about here. Does EVERYTHING need to be on-chain? No, clearly not. but for the stuff that does it's been found that you can scale by having multiple chains. I mostly use AVAX, it's like ETH but uses a DAG for consensus, which means finality is WAY faster (seconds). It also has a concept called subnets, so scale is accomplished by having multiple blockchains which can interoperate with each other. Not all validators validate all subnets, but all subnets have multiple validators. Your need for level of security is going to depend on your application. Defi has a high need for security, gaming has a low need. The idea of 1 chain for everything is kind of obsolete.
They don't have to, but they can if they want to.