...but not to download. You can clone a repo and download release artifacts without a PAT. That's only necessary for interacting with the API for actions that need authentication, which would be anything involving mutating a repository.

GitHub access tokens are a bit of a nightmare since you can't limit the permissions for a token. Only workaround I've found is to create another GitHub user for an access token and restrict that user's access.