Doing Python Configuration Right (2019)
whalesalad.com
whalesalad.com
Specific to the article, my way of handling the issue of "turn everything into an environment variable" is to rely on the cascading feature of Gila/Viper and store sane defaults in code that only are overridden when need be. This allows for default values to be set, overridden by config files provided, and further overridden either by code or by ENV variables.
This would allow us to instead of having config written in python (something I usually try to stay away from) we could have it written in something like YAML or TOML instead and loaded in on the fly in each environment, while still allowing for overrides in either the code or by ENV variables.
I recently (last few months) have switched all of my python projects over to using Gila and I have been super impressed by how well the language itself lends to building dockerized 12factor apps when using tools made for that purpose - which is a sentiment I would not have had last year at this time.
https://gitlab.com/dashwav/gila
https://gila.readthedocs.io/en/latest/index.html
I appreciate any feedback either the author or HN in general have as well.
I do a lot of Django, and have found the `django-configurations` package to be pretty good. The thing it's missing that Gila provides is the config file layer, but that's also not _typically_ how Django projects are configured, config is generally done in Python rather than config files.
I would love to hear any feedback should you decide to use Gila in one of your projects!
If I'm just focusing on how to make my code easier to read by my fellow developers, I would want to see something like this be based off of a dictionary-style API, that could be imported and used like so:
import gila as config
bucket_name = config["bucket_name"]
- Do all the consumers of gila config use the singleton class instance and just call gila.get() everywhere?
You can either use the singleton pattern (which is the recommended way of utilizing the library) or you can assign the Gila object to a local variable and ship it around your code (this would allow you to have two separate configurations at the same time). You could see that here: https://gitlab.com/dashwav/gila/-/blob/develop/examples/mult...
I also think striving for readability is the way to go when building code in general, and that is actually one of the reasons I opted to keep the `.get()` syntax! This may come down to personal opinions but I think that when the behavior of a library is fundamentally different than that of a base type like dict it should be explicit that a library is being used.
While dictionaries also have a `.get()` syntax I think that seeing `config['value']` would lead a developer to make the assumption that config is simply a dictionary, and this might lead to erroneous code whereas `gila.get("value")` (or even `config.get('value')`) will make it more obvious that this is not a dictionary but actually a library that is doing more than a dictionary under the hood.
As a final point this syntax keeps it closer to the Viper go library that I was inspired by and therefore eases the transition between the two should anyone have familiarity with either library.
config.__getitem__ = lambda x: config.get(x)One reason that 12 factor apps use environment variables to store configuration is because then you have a single source of truth on how your application is configured.
Using the principles of the 12 factor app, someone who had no hand in writing the code doesn't have to wrap their heads around whatever convoluted logical path your application takes to figuring out which Redis instance it will connect to, especially during an outage in which their priority is to fix your service as quickly as possible. They can in an emergency SSH into whatever machine is hosting your application or exec into the docker container and inspect the environment to have an almost complete picture of how your application is configured.
The "globals().update(vars(module))" line actually increased my blood pressure. Please show some consideration to your teammates that help you keep your applications running.
Have had nothing but bad experiences keeping apps running that used dotenv in node land and viper in go land. The 12 factor app people did not speak lightly.
This approach is not mutually exclusive with 12-factor. It is complementary to 12-factor.
> They can in an emergency SSH into whatever machine is hosting your application or exec into the docker container and inspect the environment to have an almost complete picture of how your application is configured.
Totally agree with this too. I build apps with ops in mind and this specific realtime debug ability.
> The "globals().update(vars(module))" line actually increased my blood pressure.
Being a little dramatic here, aren't we? Please elaborate on what is wrong here. I have been writing Python professionally for approaching 12 years, so I am very interested. It is certainly not something you would see every day, but this is not an area of the codebase you visit on a daily basis. Also, as a sidenote, I would strongly encourage you to look under the hood at some of your favorite/popular Python libraries. You'll vomit out your nose.
> Please show some consideration to your teammates that help you keep your applications running.
I forklifted FarmLogs from Heroku to a homebrew Kube cluster in 2016, led operations and support with that tool (40 microservices around the time I left) all while continuing to be an IC and ultimately became the CTO. I have an extreme amount of consideration for others when building software (almost to a fault). Clearly I have some work to do on my writing.
> Have had nothing but bad experiences keeping apps running that used dotenv in node land and viper in go land.
Yes. IMHO viper and dotenv strictly hobby tools that should be used for projects that involve a single developer and a single environment. I agree whole heartedly that they encourage bad practices and make things tough to debug and find.
I cannot tell you how many times I had to matrix my way into a running container and try to figure out what the fuck was going wrong. Let me tell you ... when you have a microservice that depends on 50 environment variables to boot you are not going to be happy. If you can shove as much of that INTO the application as possible (without handicapping yourself or your fellow team mates in the production environments, losing flexibility in deployment target, etc..) why would you not want to do it? That is ultimately what I want people to gain from this.
>> The "globals().update(vars(module))" line actually increased my blood pressure.
> Being a little dramatic here, aren't we? Please elaborate on what is wrong here.
and
>> Please show some consideration to your teammates that help you keep your applications running.
The reason I don't like this is that it first dynamically imports the right module and then implicitly sets global variables using the variables from that module using vars. This means that someone couldn't use simple greps or such searches to trace the logic through which a parameter was set (a go-to tool when debugging code someone else put into a docker container).
- - -
> Clearly I have some work to do on my writing.
Your writing is good. I disagree with your approach to solving the problem. Willing to accept that my position is wrong, but am reluctant to do so since it's built on bitter experience.
Funny to see this stated explicitly in this way. In my opinion, this is one of Python's biggest flaws (I'm a big fan of everything but the module system). Paths to files on disk should be treated as string literals, not magic unquoted strings that look like they're language keywords.
You can easily end up in situations where a directory in your project's tree has a name conflict with some library, and this causes issues, which is mind-bogglingly bad design (incidentally: also not a one-to-one map). If you don't live and breathe Python, and accidentally put a hyphen in a filename, God help you.
They are not new but implicit namespace packages are new-ish. Before 3.3 it was more complicated to set up, now you just omit __init__.py from `foo`.
I don't see any reason to use this if `foo` is shipped as a single package.
If the implementation is easy to explain, it may be a good idea.
What would you do instead?
> from './app/utils.py' import numbers
> Regarding hyphens and such in package names pretty much every language has restrictions on identifier names.
This particular restriction is an artifact of the bad design of treating paths on disk as special language tokens instead of string literals.
Python doesn't treat 'paths on disk' as special language tokens, python chooses to handle certain tokens by possibly accessing files on disk. Classes in java (lacking 'modules') resolve to equally named files during compilation, as it goes in go, node, haskell, and pretty much everything I can think of with the sole exception of C/C++ preprocessor driven multi-file development (which is even more filesystem-coupled than the former as you literally type filenames).
The solution with multiple files actually goes against the 12fapp
"Sometimes apps batch config into named groups (often called “environments”) named after specific deploys, such as the development, test, and production environments in Rails. This method does not scale cleanly"
If you plotted those axis' on a four quadrant grid you'd be able to draw a little dot somewhere on it for your particular app. Your app is not my app. My apps are not always the same, either. My personal blog for instance has a far simpler configuration than the tools I build for my clients.
"In a cloud native world, code should not be aware of the environment."
This is a dogmatic fallacy. There are tons of reasons code should be aware of its environment. The problem is that most people are unable to agree on what "aware of its environment" actually means. I do not agree that my code should not know what the staging or production environment is. This is literal devops! Developing and building your application/system with the logistics of shipping it in mind.
When the dust has settled from configuration arguments the most important thing you can do when designing your application is to first replace the word "application" with the word "system" in your mind and begin to think about it as a system. The system contains subsystems: config, persistence, presentation, blob storage, logging, etc... and all of these are self-contained and isolated units that start and stop for their own unique reasons during the app/system lifecycle. If you design things to allow for this decoupling, the right approach for configuration will manifest itself.
Sure, by default Django may create a single settings.py, but common practice is to split that into a settings package, containing a base settings module for common settings, and other files for different scenarios, say "development" and "production". Either / all of them can load secrets etc. from the environment, so the "production" settings file is probably better thought of as "deployment" settings, when supporting multiple deployed environments e.g. staging and production.
Then again it will depend greatly on the surface area of your application. A small app will do fine with a single configuration file or the DJANGO_ approach, but a larger application deserves its own config.
Your configuration paradigm should be: simple. Environment variables are probably preferable because they're widely supported by different tools (on account of being a "12 factor app" thing for whatever that's worth), but I'd say a file is also just fine, so long as it's easy to figure out where it's being loaded from (I don't want to see any "configuration is merged from files found at the following five paths and then overridden with environment variables..."). The chief concern in operations is being able to quickly and easily determine the set of configuration in use, and strangely enough, developers appreciate this too.
The overhead saved by having some "dev/prod" switch or override scheme is really not overhead at all, if you set one environment variable in your dev/test/prod infrastructure you can set ten.
This stems from not having one person or one team design the entire configuration system that will support the entire application.
All the mayhem you are referring to is certainly possible. I have experienced it too. But I would posit that this is not due to having 'named environments' but rather just a poor implementation with no leadership.
Another thing that I did not put in the post but that I tend to add-in to this concept is a 'dump my env' function. I will occassionally bundle this up and write it out when certain relevant issues or crashes happen.
def get_snapshot():
"""
Provides a snapshot of the current system configuration.
"""
return filter_dict(vars(module), exclude_predicate)
many people are pretty good at getting test coverage for their code nowadays. But the equally important test coverage for config variations remains a common blind spot.
It has various options for cascading configuration sources, such that it makes it relatively straightforward to build a hybrid approach in the event that you're attempting to migrate from a file-based config system to a more modern 12-factor approach.
At one point I needed to write a custom config source for Amazon Secrets Manager, and it proved quite easy to implement. Perhaps I should open source that…
In my experience the productivity you gain from such abstractions is negligible compared to the complexity they add to your code.
Also checking for environments is done weirdly, why not use
if 'ENV' not in os.environ:
instead of
ENV = os.environ.get("ENV", None)
if ENV is None:
Also it's a bit better from the performance perspective (but not that it also matters in this case).
Then why mention it?
Now, it’s not really an issue here with environment variables. But in general, doing assignment and then walking through the cases is safer.
Why do you say that? It looks like it can handle any number of environments to me.
I was recently onboarded into a project which is using Python in a server-less setting, and when my colleague walked me through the concept of setting up a virtual environment for each one of my python projects I could not really believe it. It seems like you practically need to containerize a python code-base to make it repeatable.
While I get that you may have been frustrated with a Python setup, concluding that it's not suitable for "serious" production software isn't a proportionate response, and the evidence of many companies using it for "serious" production systems suggests that it is in fact fine. Examples include Instagram and Dropbox.
> It seems like you practically need to containerize a python code-base to make it repeatable.
This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary. Arguably even for anything that doesn't produce completely static binaries.
Yes Python has a learning curve for production use, all languages and ecosystems do. In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. Python is not out of the ordinary, it's not the worst. It's not the best either, it's fairly reasonable for its age, complexity, and typical uses.
It's not so much that I was frustrated by this, it's more the case that given the other ecosystems I have worked with, I was kind of shocked that people actually manage to ship projects with this type of tooling.
> This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary
I don't agree. I think npm does an excellent job of making js packages repeatable for example.
> In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages.
To be honest, that's a big part of the reason I'm not a fan of interpreted languages in general in production environments. I think something like Python is fine for a setting like a notebook where you only have to worry about your own workstation, and you just want to iterate quickly without a ton of constraints on performance or robustness, but for running code in an un-supervised environment? I would much rather have a compiler checking every possible issue ahead of time rather than relying on testing and monitoring at runtime.
And Python seems particularly bad, because if I have to use a virtual environment to get the same code to run consistently on my workstation and my colleagues, that just gives me no confidence that I know how it's going to behave in some cloud execution environment. I'm not saying Python is "the worst" but basically all the properties of Python as a language ecosystem seem to run counter to what you want in order to produce robust, reliable production software.
The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.
Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task.
In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies.
Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing.
If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.
Versus, one of Python's bigger challenges with environment management is that you need to come up with an environment isolation mechanism that won't also interfere with the operation of the OS itself, or cut you off from being able to use userland tools that rely on it.
Which. . . Python being an essential part of virtually every production system out there being one of the root causes behind a situation that makes people feel skittish about its appropriateness for use in production is a fun thought. It's a particularly delightfully Unixy brand of ouroboros.
It seems crazy to me that the precise Python version isn't defined in the Python file itself, and the interpreter is responsible for sorting out compatibility issues.
Node I'd say is about as complicated. You still need to care about the Node version, about as much as the Python version. a node_modules directory is pretty similar to a virtualenv, it's not portable, maybe a little more reproducible by default, but no better than a pip-tools managed requirements file in my experience. Package upgrades are a little trickier than with pip-tools. You still need to care about things like what version of OpenSSL is on your machine, or do you have libpq installed, etc. One problem with JS is whether to compile and if so, how, what do you run, etc. You probably don't want to ship a node_modules to production with all the random package build scripts that it includes, so there's some additional complexity there introducing a build process.
But there are a lot of other languages out there besides C++ and Python.
Not really, no. Only if the compiler, compiler version, compilation flags, kernel version, platform, platform version, cpu version, directory structure, shared object versions, locale, environment (LD_PRELOAD, LD_LIBRARY_PATH, TZ), config in pwd, .local, and /etc or maybe $XDG_*, "~/Library/Application Support/" are all the same.
I think they're referring to unpredictable behavior under future versions of language VMs and garbage collectors vs. statically compiled binaries with either no GC or a statically linked GC library.
Most of the time, all of that machinery going on behind the scenes is very convenient, but coding strictly to POSIX and the C/C++ standard libraries / STL and keeping your compilation artifacts gives you a lot more stability than coding to some version of the Java standard library (or other language VM) and keeping around your JVM installer with hopes that the installer will agree to install and run on future systems.
I remember being bitten by testing our product against the earliest JRE we supported and also the latest JRE sun had released at the time. I was aware that reading environment variables was deprecated in the earliest JRE I tested with, and I knew it had been un-deprecated in the latest JRE I tested, and the methods I was using were present in all of the JRE versions in between. What I had missed was that there was at least one JRE version in the middle where the deprecated environment variable reading methods actually threw runtime errors.
I've also run into issues in the past with the version of MSVC I was required to use having an installer that depended upon some ancient version of the CLR which would no longer install on my machine. There were some workarounds, but I needed to talk to several of our helpdesk people before I found someone who knew the required magic incantations. Had the MSVC installer been coded strictly to C++ stdlib, STL, Win32 and MFC, it would have run just fine on my modern desktop.
C and C++ have a a fairly similar problem with dependency hell due to shared libraries being installed in a single system-wide location. They have their own hacks for getting dealing with it. I don't know which way of doing it is technically superior, all I know is none of them offer a developer experience that I'd choose for myself if I were designing a new programming language in this day and age.
FWIW, I give Python some credit for coming up with a solution that doesn't rely on containerization, and therefore will work consistently on many OSes, and not just Linux. Containerization is an option, too. It's really down to whether you need a system that works for one language on many OSes, or one that works for many languages on one OS.
Or, if you'd rather keep it really simple, there are zipapps, which are Python's answer to fat binaries. Similar to uberjars in Java. Or to good ol' static compilation in most the languages I like to use even when someone isn't paying me to do it.
(That said, there is one thing about environment management in Python that absolutely drives me up the wall: There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Which sucks for everyone, and makes learning the ropes without a good mentor way more of a chore than it needs to be. That's not too far off from C, either, though.)
Yeah I guess this is really my main issue. Like it seems like every Python project needs to come with a README to explain which tools are required, and maybe even which order they have to be applied to get the desired result. It just feels like an almost absurd step back when compared to other language ecosystems where these are well-solved problems.
Generally, newer projects are settling around Poetry.
As a user it's possible to put up with this, even enjoy it, when there are a handful of tools using it. A container running your webserver and its dependencies like database and caching for instance, that's reasonable.
Most serious production workflows are containerized these days -- are you sure that this supports your conclusion?
It's not that Python is a "bad language", but all these things taken together just basically work against everything I am trying to achieve in a solid production environment.
I do have if DEBUG: based switch all along my settings file, and enforce DEBUG=True in manage.py at the same time: in manage.py set os.environ['DEBUG'] = '1', and in settings.py DEBUG = bool(os.getenv('DEBUG', False)).
My objective is that when a new developer clones the repo, and runs ./manage.py runserver or shell, it will load with all development settings by default.
In theory this could cause the problem that running manage.py commands in production would run them with DEBUG=True. I tried that, and in practice this turned out to never be a problem. But this could certainly bite you if you're not careful.
In particular, I also spread env vars in two config files: docker-compose.persistent.yml for persistent deployments, and docker-compose.ephemeral.yml for ephemeral deployments.
To complete that, because I don't want to store secrets in the git repository, I do have things like that to load multi-line environment variables in CI:
- export $(echo $PROD_EMAIL_SETTINGS | xargs)
- export $(echo $PROD_ENV | xargs)
- export $(echo $PROD_SOMEAPI | xargs)
And if I want the production email server settings in staging and production, I can load the set of variables that configures for the production email server with that single line.
Anyway, for me the best setting profile management, is no profile management at all: local clone should have development settings by default and just work with 0 config, configuration for deployments in compose files for non-secrets, in CI vars for secrets, and I should be able to easily decide if I need any of the settings that are in production to develop something in which case I end up building a custom profile with environment variables (ie. to debug a blackbox such as a partner API) just "for the time being", for example I have dumped the blackbox request/responses and then build up my unit tests based on that.
I'm extremely happy with that trivial setup that I have in a bunch of projects now for the last years, I thought that's what everybody who's not using a profile-based system was doing, I thought that was the essence of 12factor/config, at least that's what I understood.
Larger apps that I deploy have a development.py.example file that people can copy to development.py (or fredsmith.py, madonna.py, it doesn't matter. These would not be checked-in to Git)
Then all I need to do before running anything in my project is prefix with ENV=fredsmith.
As a team we could settle on a different name for this variable. We could all have our own environments. We could even decide, as a team, to check them into Git so that we can all see each others config in case there is an issue.
Using different names for the same thing across different teams also looks like something I'd typically strive against.