I'd be surprised if search engines don't penalize websites that 404 previously crawled pages. They'd want to avoid the risk of a page disappearing and causing a poor user experience.
The web today has 6 billion users instead of maybe 50mm in the golden age you're reminiscing about.
Whether that is done on purpose, or is an unintended side effect of other ranking decisions, is up for debate.
these two things are not remotely the same. if you are redesigning your website then it's your responsibility to make the old links redirect to the new pages. if you don't do that, it's a broken redesign.