Do accessibility.
Seriously, don't worry a single bit about SEO. Yet, if you make it so that your site's content, navigation and URLs are screen-reader friendly, then you've probably made a site that has solid SEO and is accessible to people with vision problems.
Yet, just doing SEO doesn't make for a very accessible site all the time.
Once this is done, make good content, be real and responsive humans to your customers and you're on your way to a good business model and website.
"Don't do marketing." "Just make a good product and people will find it."
I think this is the downfall of many great products. The current startup world has an idealistic belief that better products always win and if you build something people want, they'll find it, when in fact, this has never been true. The "best" politicians consistently lose to those better at marketing and branding. The "best" ideas in organizations often fall to those trumpeted with more style or by those with more clout. Many great startups shut down because they couldn't find a repeatable, low-cost way to acquire customers (and SEO is exactly this).
More here: http://www.seomoz.org/blog/i-disagree-with-fred-marketing-is...
And here: http://hackersandfounders.tv/RDmt/rand-fishkin-inbound-marke...
Too many posts about SEO get everyone obsessing over marginal website tweaks/details when a single backlink to their webite doesn't even exist outside of their HN account.
The only problem is that it isn't currently this way (despite what Google and Matt Cutt's might say). It is naive to think that this approach is going work.
Furthermore, you're competitors are going to do accessibility + SEO, and they are going to trounce you. randfish said this better than I could, so I'll leave it at that.
You can get away with what you are suggesting if you have staggeringly awesome content, but even then - why risk it. If you integrate SEO into your culture from the start, and have the right process then it doesn't have to be that hard.
e.g. I like going on motorbike holidays, like "Long Way Round" (but not as extreme) that's called "motorbike touring". Not motorbike holidays, or trips, or travelling, but touring. One step of SEO is to see what people are searching for, which will tell you what your customers call your service.