I think that's fine. 12-factor is a set of guidelines about what you do in production. There's a whole universe of things that are smart ideas for development and bad ideas for production. Leaving debug symbols in your binaries, leaving ports for connecting a debugger open on your containers, etc.
You just set the variables at startup in prod, and in the file on development (probably mostly because you want to change them all the time).
There's an enormous difference between "I'm going to use a dev-friendly way to manage and swap out all the different configurations that I might want to plug into the app while I'm testing it, but still plug them in using the same mechanism", which is basically what you get with dotenv files, and, "I'm going to configure with environment variables in production and JSON files in development," which is another thing I often see people doing.
... your secrets get checked into git.