Do you think PadMapper (if they were to overtake CL) or AirBnb or Google or anyone else will let anyone crawl their data and disrupt their revenue stream without putting up a fight? Of course not.
The cognitive dissonance here is simply caused by people thinking CL is a non-profit, which he has shown it is not.
I'm not an MBA, but why the hell not? They entice people to become regular users with all the free stuff, building a network, allowing them to make an enormous profit off of the few products they do sell.
If they started charging for other things besides job ads (and the other 1 or 2 things they charge for), they could very well start losing money because of people refusing to pay and going elsewhere.
The money left on the table builds value by:
1. Leaving very little profit margin for any potential competitor.
2. Ensuring a continued large network effect, CL's principle source of value.
I suppose you could argue that if padmapper found a new awesome data source to crawl & became so big they no longer needed craigslist and people stop bothering to check CL for rental postings THEN padmapper could be affecting their revenue stream.