Cleaning up bad bots
blog.cloudflare.com
blog.cloudflare.com
- Harrassment with an extra verification step every god damn time you log in into a website you've been a paying customer at for years because some rando risk model thinks you're a bad guy
- Harrassment by reCAPTCHA to respond to US-centric image challenges when it decides it's up for some sadistic fun (Mark all images with a store front or street light? Sure, let me just Google how they are "supposed" to look, because I sure as hell ain't seeing any I'm familiar with here.)
- Blanket IP range bans which serve you a default 503 error page and call it a day, with ripple effects throghout tons of unrelated websites. I always know when my ISP's IP range is added to some new blacklist.
- Harrassment by Cloudflare "verifying my browser", presumably burning my CPU cycles so they can be sure that my browser, which has been hitting their IP ranges for years with not as much as a cookie wipe, has not suddenly turned into a bot
- "Your name is invalid": no, your regex is
- "Our fetishism for credit prevents us from accepting your debit card payment unless you submit a scan of your passport signed in blood"
- (NEW) CloudFlare's anti-bot measures which will surely not misfire because no one thought to test their shiny new model on the traffic patterns of some culture with < 50 million Internet users
Apologize for the off-topic rant, but you wouldn't believe how using the Internet for basic things has gotten difficult in the past few years over here (and from what I hear, in many other "forgotten" geographies). No one cares. Long live colonialism!
For comparison purposes, the dispute rate for US-origin payments (US card, US origin IP, no evidence of proxy use) was under 1%.
If I couldn't do things like "submit a scan of your passport", I would probably just decline any cards from high-risk countries.
What would you suggest I do instead?
In the EU one thing we are doing is basically enabling 2FA for card purchases aka "Strong Customer Authentication".
Now I'm not 100% happy with the use of SMS as a 2FA but what will happen is if the bank or site ask for extra verification of a purchase the issuing bank will prompt the user to auth the transaction via another means (Either a SMS Code they enter into the page (think verify by visa but using a one time code that gets delivered to the customer with the amount they are approving), a Rich Communication Services with a Approve / Deny prompt or a push notification to the users banking app on their smart device.
Will it defect card fraud completely? Nope, crooks are smart and will find ways around it like they did with chip and pin. But it will help (as long as we don't keep pushing back the rollout date - https://www.businessinsider.com/uk-delays-strong-customer-au...).
Agreed. There are basically zero digital payment systems in the US that aren't a complete joke, and all of our institutions think the correct answer to that problem is to add more captchas.
Imagine an alternate reality where we discovered that HTTP traffic was trivially monster-in-the-middled, and instead of moving to HTTPS, we all said, "well, we'll just do device fingerprinting to validate the request came from the right person." This is the current digital payment system.
Instead of moving to 2FA for each purchase or overhauling ACH, or changing payments to be a push system instead of a pull system, our solution to credit card fraud is that:
A) payments will take a long time to resolve,
B) customers will be required to be vetted using invasive, privacy-incompatible tools that don't work particularly well, and
C) whenever something goes wrong, the merchant will take the financial hit instead of the payment processor that let the fraud happen in the first place.
So many of our payment/privacy/fraud problems boil down to the fact that we are putting band-aides on a fundamentally broken system.
The situation with poorer countries is completely clear. As people have less and less honest options of providing a dignified living for themselves and their family, their moral principles give way and they turn to crime. Maslow hierarchy of needs and all that. In my opinion, the blame for a lot of these situations is the aggressive exploitative treatment and bullying of "small guys" by "big guys" on the global scene. But that's a separate political discussion, and a long one at that.
One thing that could help, but I doubt it's under your direct control, is taking advantage of systems such as 3-D Secure, which is SMS-based OTP for card payments. My card has this and I'm always happy to verify my payment this way, but payment processors are free to ignore these capabilities and just charge the card. (Conspiracy theory: this is because it hurts the conversion rates, as it adds another step to the funnel.) I think that a payment verified via 3-D Secure would have a much lower risk associated with it.
But overall, I don't think I can offer any suggestions on how to make the system better. I ranted due to a perceived injustice, but that didn't mean I have ways ready to make the situation better. I think that change must come from the powers that be - and they're unlikely to suddenly start giving a shit, so there goes that.
In any case, I thank you for considering my perspective and offering to help move the status quo. Rest assured, my friend, you are not part of the problem.
> In any case, I thank you for considering my perspective and offering to help move the status quo. Rest assured, my friend, you are not part of the problem.
I disagree with basically all of this. A lot of people here are part of the problem. They are part of "the powers that be". Not that they want to keep anyone down or anything, it's just that they have businesses to run and will do reasonable accommodations but no more.
Since I'm sure everyone here wants the problem to be solved, if it can be done easily enough, there is a good hope of solving it with some smart technologies.
3d secure and looking at cookie reputation are both good ideas. If they are not used as much as you think they should be, we gotta ask why that is? Is it because it's too hard to implement? People haven't heard of it? Can library for doing it easily be made? Can it be pushed on people through add it as default, or including it in code samples?
I understand your frustration, but the problem isn't that it's far away geographically, it's that your country is allowing criminals to use your IPs to do bad stuff.
My country has such a poor median standard of living that things like computer fraud (or crime, generally) become attractive even to reasonably honest people as they become desperate in finding ways to provide a dignified living for themselves and their families.
Due to largely the same root causes, its government has insufficient infrastructure, bandwidth, and expertise to deal with cases of computer fraud, especially with big tech corporations usually refusing to cooperate/communicate with our officials.
One of the contributors to this situation is a military operation perpetrated by a well-known and powerful transatlantic military alliance as recently as 20 years ago; it resulted in $5 billion of direct economic damage (a big deal when that's your entire GDP) and countless billions of indirect economic loss which rippled throughout the years in the aftermath (not to mention the hundreds of civillian lives lost).
I'm trying not to drag politics and emotions into this too much, and my country surely has many issues unrelated to anyone outside, but I'm expressing my grievances as a common person who, like anyone else, did not choose their birthplace, and does in no way identify with the policies and actions of their government, but suffers greatly in many ways due to them + the constant bullying and pressure by the "colonial forces", which is what I was primarily referring to.
respond to US-centric image challenges
> - Identification of well known legitimate bots;
What about non-well-known legitimate bots? If I run my own web crawler, am I at risk of falling into the tarpit (and having my IP address reported)?
Use CommonCrawl https://commoncrawl.org/
Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
A big part of the net neutrality fight was "allowing the little guy a chance to play with the big boys".
Part of the service says:- for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline... If the infrastructure provider hosting the bot is part of the Bandwidth Alliance, we’ll share the bot’s IP address so they can shutdown the bot completely. The Bandwidth Alliance allows us to reduce transit costs with partners and, with this launch, also helps us work together with them to make the Internet safer for legitimate users.
My reading of that is if CF decide your IP is bad, they can leverage the providers Bandwidth Alliance status to shutdown the providers customer. What if CF's systems misfire? Will there be a grace period? Will there be an appeals process? Will CF compensate anyone effected and have their hosting withdrawn from a misfire?
No one likes bad bots, but I'm feeling more and more uneasy allowing CF decide which is which.
They bring real visitors and customers.
Random bots running on AWS or TOR bring me headaches and bigger bills.
if this is to be believed [0] less then 50% of google searches result in a click though to the site, so one could argue that because of things like snippets in search results Google don't always drive traffic to your site.
[0] https://sparktoro.com/blog/less-than-half-of-google-searches...
Respecting robots.txt, using a well-known ua, coming from publicly declared, google owned IP block.
So if you're a small business that wants to build the next Google, you can't? Because obviously, your User Agent won't be well known when you start.
I would love for them to prove me wrong though and be open about such things.
I'm pretty sympathetic to this line of thought, so don't take this as 'you're wrong', but I notice that we would never apply this logic to laws, or environmental standards, or contracts.
You'd never hear someone say, "if we have a clearly defined tax code, that will just make it easier for people to find loopholes."
Where I do hear this argument come up is explicitly in contexts of moderation and abuse policies. And it's something that sounds very reasonable, but it's hard for me to get away from the fact that in most other contexts it sounds problematic to me.
Maybe part of the problem is Cloudflare's scale? Maybe the reason it feels bad to have a police officer pull me over because, "we think you're going too fast" instead of "you went over a posted limit", is because that's a critical infrastructure that I can't avoid.
Cloudflare is a private company, and even if it wasn't, I don't know if it would be big enough presence that I would worry about their policies. But when I hear things like this:
> Our goal is nothing short of making it no longer viable to run a malicious bot on the Internet. And we think, with our scale, we can do exactly that.
That maybe shifts the situation a tiny bit farther away from "moderation policy on a personal blog" towards "policeman pulling me over because I broke a law I didn't know existed."
I dunno. I'm not 100% sure how to feel about it. I do think that Cloudflare should be able to filter traffic however they see fit, but that doesn't mean that every strategy they choose is inherently good, or that it might not be problematic for them to lack transparency about their standards.
Also for fairness - Does Akamai publicize the internals of BotManager? Imperva gives you much transparency into how DistillNetworks works?
Then fraud detection system cannot work ethically.
Can't become a well-known UA if your shutdown before you can make a name for yourself. And a Google IP block doesn't mean legitimate.
I'm not trying to say that Google is bad, but why can't OP's crawler be legitimate?
It does. 2-way DNS lookup is how you verify legit indexers [0] (as ua itself is meaningless).
But what are CF (and others like akamai) using to determine you are a bad bots? My objection is more the threat they will use their position to kick bad bots offline.
Now don't get me wrong, I had bad bots as much as the next person. But what about misfires? They will happen. I hope CF have a plan in place to deal with such cases.
Personally, due to family issues I've been working remote-remote (I'm a remote worker anyways, but I've been working remotely away from my normal office). As such I've been using a 4G Mobile broadband connection and the amount of CloudFlare "We are just checking you are not a bot" hoops I've had to jump though recently has opened my eyes on how much of the net is actually behind CF's network because CF still associate "IP as Identity" and my 4G connection is CG-NAT'ed.
I just fear at times we forget that there is the rest of the world out there, not just our little bubble and the choices we make have a huge impact on the usability of the net for others.
EDIT: LOL.. Just checked your profile. The Reg is one of the sites that doesn't like my 4G connection and as such I visit less. What are the chances. (Though you only make me sit there and wait to access the content....)
Sorry if that's not the answer you wanted, but that's how I feel after seeing how the other side of the net is treated. IF the urge takes me, I'll help, but I dunno when that urge will take me though.
<EDIT>Fun Fact: My public IP address appears to have changed in the time it took me to write this comment. Atleast to the reg, How my ISP handles routing is another black box upon itself.</EDIT>
Don't worry, their proprietary AI will figure it all out.
> Identification of well known legitimate bots;
> Hand written rules for simple bots that, however simple, get used day in, day out;
> Our Bot Activity Detector model that spots the behavior of bots based on past traffic and blocks them; and
> Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.
In my dream world, Cloudflare would make this a two way street: you get protection against malicious bots, but you have to allow equitable treatment of all well behaved search engines (i.e. you can use robots.txt to ban lesser search engines, but doing that requires you to ban Google as well).
Disclaimer: I have no current economic interest, but kinda want to run a personal search engine.
>The targets are goods of generally of limited supply and high in demand and in value. Think sneakers, concert tickets, airline seats, and popular short run Broadway musicals. Bot operators who are able to purchase those items at retail can charge massive premiums in aftermarket sales. When the operator identifies a target site, such as an ecommerce retailer, and a specific item, such as a new pair of sneakers going on sale, they can purchase time on the new Residential Proxy as a Service market to gain access to end user machines and (relatively) clean IPs from which to launch their attack.
They then go on to spout some economic nonsense about how such bots are harmful. Actually, resellers make the market more efficient, and cloudflare is doing a disservice by lumping legitimate bots in with malicious ones like their credential stuffing example.
If the manufacturer of an item increases its price and thus their profit from it, then in principle the increased profit can drive them to manufacture more of the item, avoiding deadweight loss due to insufficient supply. However, that doesn’t apply if a reseller is the one making the profits! And in any case, some of the concrete examples cited (tickets to concerts and Broadway musicals) have an essentially fixed supply; it’s hard or impossible to simply increase production.
Separately, moving goods toward those willing to pay more for them is to some extent equivalent to moving them toward those who care more about them, which could be said to improve efficiency in terms of how much personal happiness is gained per unit. However, it also moves the goods toward those who are simply wealthier. And there are other possible proxies for personal interest besides price, ones that aren’t as affected by wealth. One of them is a person’s willingness to research in advance when certain tickets will go on sale, and be online at that time to snag them before they sell out – in other words, the system that bots are used to subvert. This system has its own biases (favoring those who are organized, have more flexible schedules, or are simply in a more favorable timezone), but it‘s still considered to be reasonably fair, at least when it’s not being subverted.
Scalping unambiguously increases Kaldor-Hicks efficiency under a simple model. You can argue that efficiency isn't the best way to be fair, and it's fairer to someone to not have a ticket due to random luck rather than being priced out, perhaps. But scalping increases efficiency, which is all I claimed.
I don’t see how that follows. If a consumer can’t buy a ticket from the original seller due to a shortage, there’s no guarantee that the original sale price was at (or even near) the maximum price that that particular consumer was willing to pay. Therefore, that consumer might also be willing to buy from a scalper. It would be in the opposite case, if there wasn’t a shortage, that someone who doesn’t buy could be assumed to be unwilling to pay the price.
In practice, I’d expect a decent fraction of those consumers to be near their maximum price, since if not, we could conclude that the original seller irrationally set their price far too low. (But the original price must have been set somewhat too low if a significant shortage exists.) Still, depending on the markup charged by the scalper, I wouldn’t be surprised if a significant fraction of consumers actually would be willing to pay it.
As a practical example, according to an article from last year about Hamilton tickets [1], among seats originally sold for $69-$179, a few were on sale for >$10,000 each, but the average resale price was $412. $10,000 will obviously price out most consumers, but going from ~$100 to ~$400 might not, considering the extremely high desirability of Hamilton seats.
> Scalping unambiguously increases Kaldor-Hicks efficiency under a simple model. You can argue that efficiency isn't the best way to be fair, and it's fairer to someone to not have a ticket due to random luck rather than being priced out, perhaps. But scalping increases efficiency, which is all I claimed.
If I understand correctly, it increases efficiency if the scalper could have hypothetically purchased their ticket from someone who was deprived the opportunity to buy from the original seller, instead of from the original seller directly, and still made a profit by reselling it.
But that doesn’t seem true. There’s no reason to think that person who “won the lottery” would be willing to sell the ticket for the same price they bought it for. Instead they would likely sell it for a similar price to what the scalper wants to charge. At best the scalper can gain some value on the margin by taking on the risk that the ticket won’t sell, but not much value.
[1] https://www.seattletimes.com/entertainment/theater/how-did-h...
As long as the ticket moves to someone with a higher willingness-to-pay, Kaldor-Hicks efficiency is increased.
Since the ticket holder without scalpers isn't the same as the one with scalpers, they have a lower willingness-to-pay than the price the scalper sells at. Therefore, they'd be willing to sell to the actual ticket holder at that price.
The only difference between this scenario and the one with a scalper is some money that the scalper has instead of the original ticket holder, and transfers don't factor into Kaldor-Hicks so this is irrelevant.
Which, as I said before, is only some of the time, but even under that assumption...
> Since the ticket holder without scalpers isn't the same as the one with scalpers, they have a lower willingness-to-pay than the price the scalper sells at. Therefore, they'd be willing to sell to the actual ticket holder at that price.
Then why don't they? Anyone who buys a ticket intending to go to the play is free to change their mind and resell it. Modern online services make this quite easy. But most people don't resell their tickets.
There are a variety of possible explanations. One is to invoke arguably irrational (but still real) behavior like loss aversion. Another has to do with the inconvenience of rescheduling or the risk of not selling, but that doesn't apply to the Kaldor-Hicks calculation since being deprived of a ticket (as opposed to selling it) doesn't create that inconvenience or risk. But here is one possible situation where even an economically rational actor would prefer to have a ticket than to be awarded a market price:
The value of the ticket to them may be higher than the scalpers' market price. This can be true even if the holder was not originally willing to pay that much, because the holder effectively gained net worth as soon as they "won the lottery" for the ticket, so they have a different amount of spare money to work with. And if this person took the time to buy tickets as soon as they were available, they probably care more about the play than the average person in the secondary market, so they would put a higher value on it relative to their wealth.
This is balanced by the fact that other people in the secondary market will likely be wealthier than them. Indeed, in the case of those rare orders-of-magnitude price hikes, most holders probably would be willing to sell if they could be sure the sale would actually go through. But most scalping margins are much more moderate and may not be enough to get the price over the value to the original holder.
The only scenario where it's not is if the demand curve is perfectly inelastic. If the price is set at $100, and selling it for $400 instead doesn't raise the average willingness to pay of buyers, then there has to be exactly the same demand at $100 as at $400, which is extremely unlikely.
>Then why don't they? Anyone who buys a ticket intending to go to the play is free to change their mind and resell it. Modern online services make this quite easy. But most people don't resell their tickets.
We're talking about a hypothetical scenario without scalpers. In that scenario, it's difficult to impossible to resell tickets.
>But here is one possible situation where even an economically rational actor would prefer to have a ticket than to be awarded a market price:
Yes, positing an income effect from winning the ticket lottery can produce a model where this result no longer holds. But income effects that strong are insanely rare. To reframe this, you're suggesting if they had found $400 on the street that morning, the rational thing to do given their preferences is to spend $400 on a ticket, but they shouldn't otherwise. You're depicting someone so poor that an additional $400 represents a significant wealth difference, but also where it makes sense to spend $400 on a ticket rather than on other things. In the real world, I think we'd call someone like that irrational.
It might be plausible in other contexts, but it doesn't seem reasonable in the ticket context.
I'm glad to see that if my services are behind cloudflare, I could just turn something on and let _them_ deal with it.
"The wait is over. Our product and engineering teams have been working round-the-clock to build a new experience that makes the Internet faster and safer for everyone. If I were to say anything more, it would surely give it away so I’ll leave it at that — you’ll just need to tune in to the blog on Wednesday to find out."
Steamspy.com seems to trigger one of these basically every time when loaded with a fresh cookie.
They could also be doing things like serving fake streaming audio that never ends, or anything that might make it seem like the web page is just a huge page that needs time to load.
I'm not sure what they do for the non-JS use case. They sit in the request path like a CDN though so maybe they just return an error or deliberately slow response times?
My question is, what if you are starting an alternative search engine or something legit?
Edit: my point is, the rules than make a legit or not bot, crawler, scrapper etc. are not clear at all.
Nice UX, fast, free. Nice domain service if you transfer to them. Fast DNS management. DOS/bot mitigation. Caching. Quick SSL, and affordable upgrade options. 2FA via TOTP.
With all the networking/domain stuff momentum build, it would be nice to be able to spin up servers for apps/db.
Main language at cloudflare is golang? Rust? Any python over there?
Request: Allow changing the super administrator for cloudflare account more easily. At least for early-stage accounts.
What's that mean, 'kicked offline'?
Isn't scraping 'legal'?
Buying products at the price offered is perfectly legitimate, regardless of all the scaremongering to the contrary.
In my experience a lot of bots seem to be running on hacked servers or through hacked/insecure proxies. I'd imagine tracking down the owner or someone upstream of those boxes could be effective in taking them offline.
When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.
There's no way you're in this conversation without being aware that scalping is a controversial practice at best.
https://theconversation.com/the-economics-of-ticket-scalping...
That doesn't make them right.
It doesn't matter if it's legal, it matters if the website owner doesn't want x doing y on their site. A bot consistently not abiding by owners' intent is inheritly malicious.
The LinkedIn case shows that it's not illegal in some cases - would you classify the behavior of the defendant in that case as malicious?
Have you run this by an economist? It's pretty basic economics that "scalping" increases consumer welfare, despite your cursory claim to the contrary.
If the IPs are part of a botnet, that's one thing. But the biggest residential IP network is luminati, which does have consent for their IPs.
Citations needed.
Here's some reading info and links for you to illustrate that you're comments are ignorant of several dimensions of the discussion: https://economics.stackexchange.com/questions/6576/is-scalpi...
> And the unwitting users who are part of the botnet have their resources, such as their home broadband connection, used without their consent or knowledge.
Perhaps I missed something, but doesn't this mean that a lot of homes and IoT devices that have been compromised will have increased CPU usage as a result of CF implementing the bad bot behaviour envisaged? In other words, the botnet owners won't care in the slightest about your response, but a lot of homes will suddenly get shoddy performance as their router grinds to a halt.
(Yes, of course it'd be best if every home and every IoT device were secure... but that's unrealistically optimistic)
Assuming you're building on the LI<>HiQ case here, the ruling would only be applicable to a subset of cases (public, user-generated content, no authorization...).
Even before the ruling is overturned, we can't say scrapping is legal without applying some qualifying conditions.