Hacker News Reader

This is not a good way to do this. Tor exit nodes are public and may be marked for special behavior by the marketplace you are surveying. There is no reason to believe you are getting good information this way.

The right way to do this would be through a VPN/tor + Residential proxy to hide your intentions from everyone involved.

> There is no reason to believe you are getting good information this way.

Spot checks checked out. And it was a perfectly fine way to do it.

You are correct that Tor exit nodes often get special handling (at the moment, including by Cloudflare, and by Google Recaptcha). And the idea of poisoning of data is starting to propagate, due do anti-AI-scraper sentiment.

Poisoning the data has been around for at least a decade now. I used to work for a very large pricing analytics company that would track product pricing for Fortune 500 manufacturers.

We found on several occasions that some shady retailers would find the CIDR of the manufacturers corp networks and comply with the MAP policies on pricing if traffic came from them. Then when our bot went through with obviously generic AWS / Proxy ips we would see a much lower price that broke their agreement. That one was a fun realization for the manufacturers as to the level of shadiness some retailers would go through.

Next time you find yourself in this situation, a $5 VPN subscription (Mullvad, etc.) gets you the same result without the IP being an obvious Tor exit node. Faster, too, in latency, bandwidth, and the time it takes to change locations. You only care about the VPN part for this, not the onion part.

Worth mentioning a $5 cloud instance and installing Algo VPN on it gets you the same thing without having to trust a 3rd party VPN provider (only a generic VM provider such as AWS). It’s always worth minimizing companies you deal with if you already use AWS, GCP, etc.

It is not. VPN Providers have server all over the world. How many cloud instances in the different part of the world you can buy with $5?

A lot of, a t3.micro is free, I guess Amazon know who you are and which IP you were using at a certain time though.

Also, public cloud machines come with IPs of well known ranges. It may be easy to spot them if required.

On the other side, this is the approach that I'm currently using, without any problem for my particular case.

AWS offers all the regions I’d care about, but I do concede many-exit scenarios are not what I was referring to. I’ve been more than satisfied with my personal use, but have never required that feature.

This is much worse than using Mullvad if you need a lot of locations, which it sounds like OP did. There are far more Mullvad exit nodes than AWS regions, and it takes mere seconds to flip between them. VPN providers have exit nodes in every major city, but there are only a handful of AWS regions.

I'm not sure what trust you need--HTTPS gives you the confidence that your connection hasn't been read or modified as it passed through the VPN provider, and frankly, I trust Mullvad more than I trust AWS.

Yes if you need many-exit, DIY is going to be more painful. I’ll disagree about the trust point, though. It’s about relationships. I want as few corporate relationships as possible. If I already deal with Amazon and AWS and I can get this additional service from them, I will gladly do it over starting a new corporate relationship (iff my trust is never violated, in which case I will then rehome everything).

Are Mullvad IPs somehow non-obvious? I assume all mainstream VPNs are detected by their IPs and slapped with captchas at best (and blocked outright at worst).

Your assumption is incorrect.

I would disagree. One of the problems of operating a VPN is that it is somewhat like operating a tor exit node: bad traffic is absolutely going to exit your infra. This means if you're renting VPSes or whatever you need a tolerant hoster.

For this reason a large part of Mullvad's infrastructure is hosted by M247 Ltd. If you simply block that ASN you will block quite a large part of Mullvad. You can block the ASN by using one of the myriad services that allow you to query all the IP blocks assigned to an ASN.

It's also possible to simply enumerate all their servers. They have an API.

You might not get it all, but you can block a significant percent.

The fact you could do it, doesn't necessarily mean that you WOULD do it. In my experience, the kind of mitigation you're talking about is seldom enforced (except Cloudflare, perhaps, and pretty much anybody serious knows how to get around THAT.)

this ASN is also popular among most VPN providers so you block this ASN and you could possibly remove a lot of abuse

I've done it at work, this is on a blacklist of ASNs that require "extra" authentication

Yeah. That’s a weird parent comment. Mullvad publishes all their “exit node” ips. Easy to detect and block.

https://mullvad.net/en/servers

Respectfully, have you tried browsing the web on Tor and then on Mullvad? Almost everything blocks Tor and almost nothing blocks Mullvad. My comment was not "avoid using detectable exit nodes", it was "avoid using Tor exit nodes." This is still my recommendation. This is coming from personal experience with the services in question.

No, any decent anti-abuse system is going to know where the VPN exit nodes are too.

The difference is that the ratio of abusive to legitimate traffic tends to be much lures with VPNs. Because a VPN isn’t 100% anonymous and just shifts the trust boundary to the VPN provider, most hard-core criminals won’t use them. So your traffic tends to be mixed in with folks who just want to pirate movies, watch Netflix shows that are only available in other countries, avoid geo-pricing surcharges, or who just care about privacy in the abstract. And then they use their VPN for all their web browsing, since they already paid for it. Tor is used by all sorts of folks who are trading kiddie porn, hacking into systems, scraping large quantities of data, buying & selling drugs, and doing other outright criminal things. Most abuse-detection systems care a lot about the legit:malicious ratio. An IP that is 95% legit but 5% malicious traffic will usually go through, as long as the malicious traffic doesn’t crash the system. An IP that is 80% malicious usually won’t.

When I browse through Tor, big websites like Reddit, Google, or anything protected by climate Cloudflare will 429 me, even if I’m just innocuously viewing a page or making a search in a regular browser. In a VPN, it almost always goes through, except for rare transient moments when somebody else is running a scraper through that endpoint.

Counterintuitively, it can often be better to use a system that is less secure, because lots of other people use the system with less security and so your actions get lost in theirs. Systems that are more secure are used by more hardcore criminals, and so they have more eyes on them. In these days of stochastic guilt, it’s often better to be doing naughty things in a law-abiding population than to be a legit user in a criminal population.

The $5 will only give you the equivalent of 1 exit node.

Incorrect - you've obviously never used one of these services. You get N concurrent connections to your choice of exit nodes. Sometimes N is even infinity.

Sites know about them though. If you need to be even sneakier there's residential proxy networks.

> selecting appropriate Tor exit nodes for each regional site

So, a proxy? Onion routing doesn't really play a role for this use case.

> So, a proxy? Onion routing doesn't really play a role for this use case.

The onion routing obscured our identity from the "proxy" exit nodes.

Separately, Tor was also a convenient way to get a lot of arbitrary country-specific "proxies", without dealing with the sometimes sketchy businesses that are behind residential IP proxies.

(Counterfeiting/graymarket operations can be organized crime. I'd rather just fire up Tor, and trust math a little, than to try to vet the legitimacy and intentions of a residential IP broker.)

Why would you need to obscure your identity from the exit nodes?

So that the exit node can't go to the site they were scraping and say "this is the person scraping your site".

But you'd have relays in between, there's no way an exit node would know who is scraping...

Right, but the question was "why would you need to obscure your identity from the exit nodes", in the context of why the person chose Tor vs. a simple proxy.

Ohh I see!

Yes, but many scrapers configure their Torrc into single-relay mode (following tutorials and open source “rotating tor proxy” repos), in which case the exit relay can easily identify who is scraping which site.

The exit nodes are free to observe and exploit the traffic that they move.

https://www.vice.com/en/article/badonion-honeypot-malicious-...

The Tor exit nodes are public.

They were concerned about the exit node identifying them, not the site identifying that a Tor exit node is connecting.

Honestly what he describes sounds like Raptor (Princeton Report, 2015)

How is this related to Princeton's Raptor, other than having the keywords "Tor" and "surveillance"?

https://www.princeton.edu/~pmittal/publications/raptor-USENI...

(Strange coincidence: We also had different key tech with the codename of Raptor, but it had nothing to do with Tor nor Web scraping. It was for discreet smartphone-based field auditing of physical product, in global physical retail and other locations. The codename was the result of a great morale-boosting impromptu brainstorming session between engineering and marketing people ("can you help think of a cool codename for this..."), and the resulting name highly apt, at least for the movie velociraptors. I built it, and, until Covid disrupted our F500 customers and investors, I was looking forward to hiring engineers to do further work on something cool-sounding like "Raptor", rather than "internal-app" or whatever first came to mind when creating the Git repo. :)

The major attack of concern described in the paper is the transparent early terminated encryption attack, and root trust signing that fall under effectively the same centralized hands at the AS level.

Where an AS level entity MITMs all outbound connections from a region in automated fashion for collection, before that traffic ever makes it to TOR or its destination.

It works for TOR, TLS, pretty much any protocol out there where key exchange or trust occurs; so long as the protocol is known and has distinct classifiable characteristics allowing computation to automatically do this.

There have been instances where public certs issued by a CA with the same domain names, but are issued from a root CA that is other than the legitimate site's root CA which are used for attacks. CT logs don't stop this either.

There is a lot of ephemeral content, and private information that can be both collected, and injected on a targeted basis if one has access to such junctions which the industry (Telecom) has proven time and again that they can't secure following basic practice; largely because mandates to backwards compatibility at the regulatory level.

Social credit, where invisible factors people don't control force those same people into poverty through targeted denial of service (communications for job hunting/social contacts), zersetzung, etc; that all would be a breeze to set up without any external indicator, or remedy using that attack.

What the target sees vs what everyone else sees would be quite different, and of course there would be people that gaslight and torture on top of it all (as a natural psychological defense mechanism of denial).

Compromised communications under such type of attacks are madness inducing.

HEH

I'm letting my imagination fill in the color on the specifics here and I'm working up a little grin.

A hat tip to you

Is it that difficult for a business to colo in various locations? Not rhetorical, I've never thought about it.

exit node is not really a colo.

though there are commercial residential-proxy services available

What was the scraper gathering specifically?

Listings of items for sale (for ~100 brands), and how that changed over time. With the marketplace having a pretty rich schema to reconstruct from their server-side rendering.

One of the purposes was cold sales outreaches to an exec at a brand, maybe something like, "Here's a report about graymarket/counterfeit of your brand online, using data you probably haven't seen before; we have a solution we'd like to tell you about".

If I could wager a guess, it sounds like the startup was in the business of scraping Amazon.

No. And when people share info on HN, I don't like to see speculation in the comments about things they obviously intentionally didn't say (assuming that they seem to be speaking in good faith). That person, and other people who see the dynamic, presumably are less likely to share in the future.

Thank you for pointing that out. Really appreciate you sharing.

To the parent, please do not try to lure info out of people it is just not cool online or in real life when people obviously are being generic for a reason.

I feel there is a level of irony in you being bothered about people interacting with content you've shared in a way you don't like when said content is a story about you interacting with other's content in a way they've explicitly put up barriers to try and stop you from doing that.

Who said the site put up barriers?

I think you have a valid general question (and you'll note I said "appropriate kind, IMHO" at the top of the original comment, acknowledging others might disagree that it was appropriate), but I'd like to contrast two distinct situations:

* A collegial forum, where people might go to share information, sometimes with discretion about what can and can't be said (or just comfort levels).

* A large corporation that was profiting off of illegal businesses (e.g., contract-violating, IP-violating, defrauding buyers, possibly fencing), and we wanted to gather evidence of that on behalf of some of the harmed parties, to try to stop it. And we did that in a technologically gentle, non-disruptive way. And (as I mentioned in the original comment) we had a conscious policy to immediately cease if we were ever told to.

Did you know if you violated any ToS with your software? If yes, why did you feel compelled to continue?

No.

Ok, with the phrasing used it looked much more sus than it is then :)

I am not a lawyer but you should know that in most/many/all cases, website ToS have no legal effect whatsoever.

They amount to the same thing as stating "we don't like if you do X and we'll probably ban you" but written in legalese so people take it more seriously.

Get the advice of an actual lawyer before relying on this of course.

You won't be able to scrape Amazon using Tor.