Hacker News Reader

Comment by BeetleB | Hacker News Reader

Wow, but I wonder if there's more to it.

I've not been able to access web.archive.org from my work computer - I always get the 429 error.

But I then pull out my phone and can access it just fine. All along I was assuming my company was blocking it. Still weird that it happens every time from my work PC and never from my home one.

View on HN

I assume that's because the IP range of your company network overlaps with a range used by some scrapers, and if it doesn't happen on your phone even in the corporate network, then IA probably checks some extra signals like the user agent in addition to the IP

No - my phone is not connected to work's WiFi.

Wonder who the bad actors in my company are...

Doesn't have to be someone at your work, it could be a block on a whole ISP network or at least an IP subnetwork that is shared between many clients

You can try emailing the address mentioned in their post so they adjust their filters to match just the bot networks more precisely

You should file a support ticket with your manager and the IT security or support desk. Show them the evidence of 429s that are blocking your assigned tasks during working hours. Also include the screenshots and files that you downloaded on your personal device in order to access your work-related materials. Be sure and thank them for adequately configuring the MDM on your personal mobile device so that you could do these work-related tasks. You should definitely also file an expense report to request reimbursement of your personal mobile bill, any data charges incurred, and the hours of networking or collaborating with external colleagues, while you were working on these work-related projects with your personal device.

Not sure if you're posting as a joke or sarcasm but accessing archive.org is not relevant to my work.

Well I'm not gonna lie: the comment was generated by a neural net. In decades to come, scraping will liberate all forum comments from context, hypertext, and subtext alike.And our immortal human souls shall

It might just be your corporate VPN and whatever ASN it’s being routed through.

There are definitely factors beyond IP being used. A week ago I found that all requests from Chrome-like browsers got a 429 across more than a half dozen networks and several machines, while Firefox reliably worked. I assume this was an overzealous policy on UAs.

In my work, it's failing on both Firefox and Chrome.

Are there any decent/reputable residential proxy companies? I’m pretty sure I’ll end up needing one occasionally, for those days when my residential IP has a poor reputation score.

No, they are all grey-market. Some have more professional-looking websites, but they're all using the same proxies. This should not prevent you from using them.

Your workplace is probably redirecting traffic through a datacenter IP range. Especially if they have their own datacenters like google, microsoft, oracle, amazon, etc.

Try making a vpn via digital ocean for example and you'll see similar patterns.

They seem to be aggressively blocking IPv6 source IPs. I ran into this problem over the past month traveling. I got nothing but 429 errors until I switched on my VPN (which is IPv4 only) and magically the Wayback machine worked again. The lack of transparency on the part of IA is very frustrating.

Could be due to some scrapers from either your work ISP block, or the larger block which lends IPs to multiple workplaces.

My buddy said he could not access it even from a residential IP, it was blacklisted for some reason.

Reply on news.ycombinator.com