I agonize about this every day, since a large part of my job is aggregating data from many sites that seem hell-bent on not letting anyone access it without going one-form-at-a-time through their crap UI.

The thing is, we would gladly pay these companies for an API or even just a periodic data-dump of what we need. We've even offered to some of them to write and maintain the API for them. They're not interested, for various industry-specific reasons.

I often wonder how much developer time and money are wasted in total between them blocking and devs working around their blocks.