Being "semantic" (your quotes) is not an abstract concept or dogma. This discussion has been exhausted as far back as 2005. A more practical explanation:
HTML is meant to be parsed and interpreted by machines too, with no visual representation. Tags are meta-data about their content. A 'table' element says "I contain tabular data", and user agents will make lots of assumptions based on that. One of the consequences of this is you'll be giving bad data to search engine bots, that's why it's bad for SEO.
HTML is also meant to be used by non-visual interpreters, like screen readers. Assistive technology is not only for the legally blind; visual impairment affects 20-30% of the world's population. We don't see high user %s because they just most won't bother, it's too hard, sites' HTML is broken, it is easier to have somebody look it up for you.
Going on a little more over semantics, there are rules for what elements can contain each other, so that a structure can be inferred based on headings and hierarchy. By not following the rules you are wreaking havoc on this, and preventing aforementioned bots and screen readers from providing meaningful navigation and context for your page.
We were supposed to have a semantic web by now, where every piece of information was annotated with meaningful meta-data, allowing for context-aware applications (add this commenter on facebook; where can I buy this product?; add this event to my calendar). Turns out it's damn hard to get everyone to use markup correctly, or even agree to common formats.