So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
Merely for your consideration, they actually do a great job of indicating in the response how many more requests per "window" the current authentication is allowed, and a header containing the epoch at which time the window will reset: https://docs.github.com/en/rest/overview/resources-in-the-re...
I would suspect, all things being equal, that politely spaced requests are better than "as fast as computers can go" but I was just trying to point out that one need not bend oppressively over the other direction when the site is kind enough to tell the caller about the boundaries in a machine-readable format
Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in the company was just pulling all our data into Algolia to improve search. They were clearly not very courteous!
So on the other end (building APIs) I certainly do pay attention to traffic and have Grafana alerts set up around it.
In general I would rate limit by IP anything connected to the internet.
Can you provide an actual example? I see it come up a lot in these conversations, but I'm really skeptical that anyone actually does this analysis. It seems like rate analysis (either requests or bandwidth) would achieve the same result in a far simpler manner, so I suspect that is what actually happens.
I am sorry, admins! I won't do it next time.