It does break it, from experience authorizing the pulls with a bot user fixes it.

In the case were the build happens from a github action there are standard builtin credential (workflow permissions).

https://docs.github.com/en/rest/using-the-rest-api/rate-limi...

Can't count the times a "nuget restore" in our CI fails with 401, just to succeed on a 2nd attempt a few seconds later. Seems like the IP range is somehow flagged, so there's definetly a downside to it.