Thank you for reading and commenting. It is great to understand different people's perspectives.

From my experience the main downside of keeping the "guardrails on", people are less likely to learn about the importance of git global ignore and also with open source projects there can be a green square chasing game (which we also need to educate people on) - just thinking about this, more Pull Requests will come in with unwanted files though, which will be painful for contributors and maintainers, but could be a good lesson?