Yes you are supposed to cater to the 2%. Wolfram Alpha says 2% of the US population is 6.39 million people. That's how many potential people from just one country you are treating badly. And as for screen reading users, that's just another 2% isn't it (or another "tiny" number). Why do you care about them? Well, it is the law, but, it's also just sensible.
Also, people browse with poor Internet connections, and even if they have JS enabled, the JS might time-out before it downloads, and so they'll just have to see your content without it.
I browse without JS on most sites (and I need a good reason to enable it) for at least the following reasons:
1. It's safer for me.
2. I've lived and will live in places with really shit Internet connections.
3. It also conveniently gets rid of a bunch of crap (flashing stuff, ads, flashing ads, etc.) without me having to do anything specifically.
4. It blocks some tracking across the Internet (I also use RequestPolicy to really drive that message home).
Other reasons as well. Edit: including it's too often used for unnecessary fan-inducing overly-CPU using shit.
So you don't want me as a potential visitor? A tech-savey person? I'm not your target audience? That's OK, there's loads of other sites on the web, and I'm sure that yours is crap anyway. After all, if it wasn't crap, you'd have made it work without JS.
Edit:
And if you are stupid enough to not be able to make your site work without JS, then at least put a noscript message explaining why your site is special enough to require JS, and why I shouldn't just close the tab and go to the next site. If you explain that it is necessary to make the froggles boggle, I'll think about it more than if you just say "you need to enable JavaScript or upgrade your browser". No, fuck you, I don't need to enable JS, you need to make your site work without it. Displaying text and images in no way requires JS, unless you're a real thicko who shouldn't be allowed to have any commit privileges.