Hacker News Reader

Comment by incolumitas | Hacker News Reader

Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/?

Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.

But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!

Some questions:

- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?

- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?

- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?

View on HN

https://incolumitas.com/

This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-lik...

I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones.

For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply.

By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today.

https://github.com/ipinfo/mmdbctl

I'm very curious why you'd do VPN/proxy detection...

But at a previous company I worked at that ran a very large chunk of the internet, we did indexing of nearly the entire internet (even large portions of the dark web) approximately every two weeks. There were about 500 servers doing that non-stop. So, I think it is relatively reasonable if you have 600 servers to do that.

In the business of media streaming, rightholder will require that you check for vpn and proxies in addition to countries when deciding if a given viewer will be able to stream a given media.

Does that actually work? That could explain an issue with a particular streaming service I use. There are currently some ongoing routing issues in BGP land and my ISP. When trying to stream, it says I’m using a proxy, so due to the incredible route my packets are taking, that might be it. What’s funny is that the only way to watch this service is to use a vpn right now.

of course it doesnt work but they gotta try clutching pearls and applying whatever pressure they can think of on these fronts

Why is this getting downvoted? It seems to me that a lot of the media-focused anti-piracy tooling is essentially a performance of toughness to make rightsholder execs comfortable. Everybody accepts you can't stop piracy entirely, and nobody's willing to say, "Fuck it, we'll compete on convenience and strong consumer relationships," so we all put up with this weird middle ground of performative DRM and the like. With only the rare occasional bit of honesty, as from Weird Al: https://sfba.social/@williampietri/110906012997848549

This is correct. Imagine in the days of yore, some two decades and change ago, when I was charged with implementing putting some music reserves "online" for streaming ...

[Harp music, progressive diagonal wave distortions through the viewport ...]

We had two layers of passwords (one to get to the webpage for the class, one when actually streaming via the client, which was RealPlayer) as well as an IP range restriction to campus (you live off campus? So sorry) because our lawyers were worried about what the RIAA's lawyers would find sufficient in the wake of a bunch of Napster-baited lawsuits launched at universities. The material itself was largely limited to snippets.

I wanted to say, "Calm down, have a martini or something. College students are just not going to go wild to download 128 kbps segments of old classical music," but alas I was not in charge.

I love that HLS+AES "encryption" is still in use, thanks to Apple.

They probably just keep a list of known VPN server IP’s.

Routing should not impact the detection, it's usually based on maxmind's anonymous/datacenter database using your IP. Accuracy won't be 100% of course but you have to show compliance.

I doubt it. According to that database my ip is in a totally different country but I'm served the correct content. Despite my efforts to fix this for years...

You can guess pretty well how IP's are related by BGP announcements, so as long as a few per block and if small, ASN. You can use that logic.

Reply on news.ycombinator.com