Couldn't you use fine grained github tokens for some of what you're accomplishing with the github/network filtering?
There are going to be cases where you want to white list an org or a repo for read access that is not under your control and Github filtering will be a simple way to do that.
Github can be a source of hostile code, prompt injections, and exfiltration- you may want to lock down using repos that aren't yours.
The tool inherited this feature from the prior implementation and its something I am still exploring.