Won't machine learning largely solve this problem? Fine, don't provide an API, but I can extract a useful JSON document from your HTML representation.