It looks like OSM has two builtin bike routing algorithms you can use, and at least for a few routes I tried they both seem significantly worse than what Google Maps does (Big fat disclaimer: I work at Google, although not on anything related to this).

The OSM directions for biking to my office involve going through a decommissioned oil field (fenced off, so the road isn't actually passable) and a swamp. What's weird is that it looks like OSM knows the road is closed but still tries to route me through it. Google's directions for the same destination aren't perfect either but at least they'd get me to my destination (they include a short jaunt down an eroding hillside where I have to walk my bike because the sand is so loose).

That cycle.travel site looks really promising. They give the same route as Google Maps, but do a much better job explaining that part of it is an unpaved dirt trail.