To get a website up and running, most of what I listed falls indeed in the 20% category (80/20).
But if you care about accessibility, usability and SEO you should probably care about HTML issues like pagespeed and validation.
Web Content Accessibility Guidelines 1.0
Create documents that validate to published formal
grammars.
http://www.useit.com/alertbox/response-times.html
Users still hate slow sites and don't hesitate telling us.
Google Webmaster Guidelines (Design & Content)
Check for broken links and correct HTML.
So perhaps it is the "shit nobody, but front-end engineers, should care about" category.