Always using containers make it harder for you to tell when you're making your setup brittle. If your environment always is exactly the same, how will you notice when you introduce dependencies on particular quirks with that environment? If your developers use different operating systems, different compilers, etc., you have a better shot at noticing undesirable coupling between the system and its environment.