You could handle even the large numbers of IPs that access github using a Bloom filter (http://en.wikipedia.org/wiki/Bloom_filter).

It will let a small percentage of non-whitelisted IPs in, but would filter the majority of them out.