Sure enough, after a few hindered of requests my crawler got banned.
Doing a simple search at Google, I can see from the results I get that CL is happily allowing Google bots.
I strongly believe that this discrimination against small players is highly unethical, but my question is, is it breaking any laws or maybe anti-monopolistic rules?
That said, you've already given the solution to your problem yourself. Instead of crawling craigslist, crawl google instead.
CL finally backed down when the EFF told them to, which says a lot because the EFF is practically owned by CL.
Sorry?
I'm totally floored by that, did I miss something?
I know that CL donated some sum of money to the EFF but that was a settlement.
Only until it became a high profile issue with 3taps did the EFF speak out against CL.
That was about a year after I contacted them and they only did after 3taps started getting press.
To me it was pretty clear that they didn't want to bite the hand that fed them, until they had to or else lose all credibility.
I don't think it is as simple. Wouldn't selectively banning competitors of Yahoo search, break some competition/anti-monopoly rules/laws?
I got around IP restrictions by essential having SETI like software running on my client's computers.
They will threaten you saying you're committing a felony by violating the CFAA. They are insane, but have deep pockets.
After I shut down, others stood up against CL's ridiculous claims and lost. Search for 3taps.com and padmapper.com to learn more.
I wonder how can they claim that what you did was illegal or against their TOS, but what search engines are doing is not.
I guess I am lucky I didn't spend too much time on this. Thanks again!