A training set like Google's Recaptcha data would be useful. Maybe Project Gutenberg, Wikipedia and other open source projects should start an open Recaptcha-like service to collect such data based on scanned documents/books/etc.
One needs a real world training set, a lot of variation, like house numbers and scanned books with labeled datasets.