I'd say the reason why following spec on status codes has fallen out of favor is because browsers don't do anything interesting or informative with them for most users.

On APIs, I'd say it's a sin to not return proper HTTP headers, but when the end user on a web site sees a File Not Found that's 200 Status OK versus File Not Found that's 404, it doesn't really matter.

And while it matters for things like the Googlebot, over the years developers stopped caring, because delivering an attractive and descriptive status page was more important than delivering an error status and not rendering a page at all.

I'm not saying it's right, but it was largely a ornamental step in the early days of the Web, so I understand why it disappeared. If browsers proactively provided data to users about HTTP status codes, I think they'd be adhered to much more.