This is default behaviour for OAuth apps in github. I will have to migrate to Github Apps for more fine-tuned access.