A guide to organizing settings in Django | Hacker News Reader

A guide to organizing settings in Django

apibakery.com

98 pointssenko34 comments

This isn't wrong advice but their tone of authority seems a bit misplaced.

1. Make settings.py a package

2. Please do not put anything more than imports in __init__.py since you are adding logic downstream.

3. Use django environ https://github.com/joke2k/django-environ which gives you a defined schema of your environment needs with casting and default values when needed.

Now you keep your .env files, keep your setting overrides (settings/base.py, settings/local.py, etc) when needed AND you have defined a schema with non-string types for your environment variables.

From the readme you can see how you make a schema with a casted type and default value:

  env = environ.Env(
      DEBUG=(bool, False)
  )

While we started off using `django-environ` to help manage environment-based/12-factor settings, we've moved away from it in favor of django-classy-settings.

The biggest knock against django-environ is that it does not treat the `.env` syntax the same as Docker or bash -- meaning that the same environment file can't be reliably used to provide variables for both the container and Django.

django-classy-settings has been a joy to use, and its code is really simple and readable (~150 lines).

[0] https://github.com/funkybob/django-classy-settings/

Have you got any examples of where the same .env file can't be used for both Docker and Bash? This is exactly how I do it in my django projects and haven't yet run into any issues.

`django-environ` makes a fairly simple effort[1] to strip starting/ending quotes off of literals. This will set the value of MY_VAR to the string `foo` (with doublequotes removed) in django-environ:

   MY_VAR="foo"

Docker does not do any quote parsing. For this same env file, it will set the value of the variable to `"foo"` (retaining the doublequotes in the value).

Bash, of course, requires quotes if the variable contains any special bash characters (for example, literal JSON with curly brackets), but its quote handling is much more complex. django-environ doesn't interpret bash code; it just does simple quote chomping.

There's no reliable .env syntax you can use that works in all 3 of django-environ, Docker, and bash; and any variable that should start and end with quotes that are not stripped off can't be expressed in a way that both Docker and django-environ will read in the same way.

This may seem like a nit-picking edge case, but it's indicative of the design philosophy in django-environ of trying to be "helpful", but in ways which lead to subtle confusion. The way it guesses the path to your `.env` file is another example.

[1] https://github.com/joke2k/django-environ/blob/main/environ/e...

Thanks - that's useful to know.

OP here. There are different styles you could go about, all of them having in common that you wan't to pull things from .env nicely.

I personally don't prefer django-environ because it's a bit too verbose/magical for my tastes, so use python-dotenv. I would recommend against using `local.py`. It's easy to overcomplicate things here and one of the things I worry about is how easy it is to have a new person set the project up and start working on it.

So, in general, absolutely agree with you. Details (which package to use and how exactly to set up the layout) are always going to be preference-based.

Or at least until the nice folks from Black alleviate us from that pain as well :)

Sure. Claiming "too verbose/magical" for a library that adds structure to generally unstructured, smattered config data seems a bit misplaced.

You are entitled to your opinion but I don't think trying to defend your stance with homemade env parsing functions while writing an authoritative sounding blog post helps the community as a whole when there are sound libraries and patterns that solve this exact problem.

That’s how I do it. .env files are terrible if you’re using them with a language specific loader. The entire point of env is that it’s an abstraction you can manipulate externally. You can setup the env, then launch the program. It allows for runtime changes.

https://whalesalad.com/blog/doing-python-configuration-right

You can do it right without packages or external dependencies.

Scrolling through the thread, this is my favorite so far.

I like that it avoids counterproductive dependencies for loading a dotfile within the app. This is an operation that should be done at the same level as launches the Python interpreter, and Python should then complain if required envs are not set.

We just have a single settings.py and use django-environ to populate it.

    # settings.py

    from environ import Env

    ENV = Env()

    DEBUG = ENV.bool("DEBUG", True)
    SECRET_KEY = ENV.str("SECRET_KEY", "debug-secret-key")
    ALLOWED_HOSTS = ENV.list("ALLOWED_HOSTS", default=["*"])

    ...


    # Running with local .env
    $ foreman run -- python3 manage.py runserver

    # Running with prod.env
    $ foreman run -e prod.env -- python3 manage.py runserver

It's easy and scales well with a lot of settings/environments/developers.

I realise that this is probably just an example for HN so this isn't criticism of your comment! But in case anyone reading doesn't know already, `runserver` is not appropriate for running in production.

It doesn't have any security hardening, is single threaded and blocking, will reload code as it changes, and is generally very slow.

It's generally best to use something like gunicorn, uwsgi, uvicorn, etc, and also to then host behind something like nginx which is designed to be exposed to potentially malicious users.

Furthermore I would like to add that having DEBUG set to True by default could lead you to accidentally enabling Django debug mode in production which could lead to accidentally leaking secrets.

We've gone back and forth on this a number of times internally and decided for the most part to just keep Django's defaults. There are a number of different ways that having "secure by default" can break your local development experience. For instance, if you accidentally run your local Django server with DEBUG=False, SECURE_SSL_REDIRECT=True, SESSION_COOKIE_SECURE=True, and any of the SECURE_HSTS_* settings your browser will basically remember that for all of eternity and you will no longer be able to just use http://localhost:8000/ without having to go figure out how to get it to "forget" all that stuff.

So for local development we follow the principle of least surprise, and we have a fairly rigorous policy/procedure for configuring and auditing production environment settings.

With a single settings file how do you handle things that should only ever be configured in a specific environment?

Django debug toolbar for example for a development environment? It needs to be listed as an installed app.

Or something specific to tests when the CI/CD pipeline is running the test suite.

The Django Debug Toolbar has a SHOW_TOOLBAR_CALLBACK setting that you can give a dotted import path to a function that returns True/False whether it should be enabled or not. By default it just looks at settings.DEBUG.

Other configurations work the same way.

Don't do this.

Keep everything in settings.py and use environment variables (django-environ) to configure it.

In a project with tens and tens of apps and so many modules requiring configurations and easily reaching more than thousand settings key, keeping 1 settings.py has not failed me.

With the approach mentioned in the article, you'll get to a point where you'll be having so different setting module for each env.

01. local

02. testing

03. CI/CI

04. E2E

05. Docker

06. Kubernetes

07. staging

08. release

09. production

10. k8s_win.py

11. k8s_linux.py

12. k8s_mac.py

13. heroku.py

14. pie.py

15. home.py

16. vacation.py

17. ...

Soon you'll get to a place where you need to kind of merge 2 of them (settings/docker.py and settings/kubernetes.py) or worse, several of them and then hell is unleashed. You get confused, your team gets confused, the new team member is confused, no one has any idea where to look at.

Just keep it in 1 place, let your brain stay healthier and prevent early cancer.

I'd go so far as to say that configuration details for specific runtime environments should never be the concern of the application/service/program itself. It's almost always a smell to me when I see "dev" "prod" "stage" "local" in code unless it's an abstraction over the configuration for some sort of logical runtime profile.

OP here. I agree with you. Quoting from the article:

> We can now go back to only having settings.py that imports dotenv, fetches the configuration, and initializes Django settings as needed.

Can you share an example of what a single settings file looks like that configures the 1 through 17 environments you have listed?

Well, now you have the exactly equivalent problem of managing your environment variables.

What is better depends on your tooling.

The next step from this is the django-configurations package. While it's not perfect, it does help when settings get very complex.

We've got a Django site that we run in production for different partners. We've got 560 settings, and then dev/test/prod configs for each partner, plus we vary the list of INSTALLED_APPS per partner. django-configurations is pretty nice for making this all manageable, and mypy helps to make sure everything is consistent.

+1 for django-configurations, it's been invaluable for us in helping to keep settings managable as we build / test / deploy across multiple environments and modes

I use python-decouple with django and it’s solved this pretty elegantly. A dotenv file for local and env variables for production. Isn’t that the 12 factor recommendation?

https://github.com/henriquebastos/python-decouple

Seconded. I switched from django-environ to python-decouple. Now that python-decouple defers to environment variables over .env files I find that I have our DevOps people spin up containers with secrets configured using their tool de jure while I just have a local .env file. We just make sure our env.sample is up to date with respect to our .env. DevOps folk use that to define environment variables for the container.

This seems overly complex to me. They have their settings in a .env file, they have a separate program/script that loads those settings into the environment variables, then they have another program that reads them out of the environment and parses them into python objects in settings.py... so why not just put them in settings.py to begin with?

I have also never agreed with the common suggestion of putting secrets into env vars- I believe secrets belong in something like Hashicorp's Vault and stored on the running system using proper OS level access control.

Having experimented with a bunch of different ways to organize complex django settings, I think what OP suggests is a local optimum that's hard to scale and doesn't solve a bunch of important problems (eg ok, credentials are now in local.py which is not committed to git. Now how do I get it to people's dev machines / CI?)

My key observation is this: If you're actually deploying and maintaining a django app you can't escape having quite a few environment variables that control different pieces of config. Once I accept that I can't shake these off, I'd rather minimize all other complexity, including having to reason about multiple settings modules that import/override each other in creative ways.

What I've settled on is:

1. Every thing controlled by env vars, not choice of settings module

2. the one and only settings.py toggles config based on env vars, say something like this:

    STORAGE_MODE = os.environ['STORAGE_MODE']
    
    STATICFILES_STORAGE = {
       'local': '...StaticFileStorage',
       's3': '...S3Boto3Storage'
    }[STORAGE_MODE]

EDIT: wording and typos

Dynamic configuration is the first step, second step is the validation of configuration. To that aim, one can use Pydantic Settings management with extensive validation to avoid footguns down the road. If validation is configured properly, if the application starts without crashing, it is already on good tracks.

While this might be decent advice around the internal details of a Django application's settings, what's missing is a mention of who this approach is designed for. It appears geared for developers who work on the project because the internal details (module names like settings.local, settings.production) leak to the command line interface. This might be fine in some cases, but this advice is not universal. All apps have a configuration API and you might consider how that looks depending on who/what is running the application, even if it's people who work on the code.

Interesting read!

Just wanted to say though I don't think the critique of the "settings/__init__.py" approach is accurate.

You can easily access environment variables inside these files using os.environ, and it doesn't require a file locally that isn't present in the repo. As you say you just specify which settings file to use with the DJANGO_SETTINGS_MODULE env var. Am I missing something?

Thanks!

Once you switch to using environment variables, having additional specialized settings files is a complexity you don't need, in my experience. So I prefer to simplify without losing much in terms of convenience.

For newbie, I think the best way to learn how to organize a Django project is using cookiecutter-django. They follow most best practices, and you have already worked docker-compose config to quickly up and running without problems.

So, they figured out they can can use os.environ within the settings.py? Am I missing something?

OP here.

You can use whatever you want in settings.py. It's "why" that matters.

Yeah, you can boil it down to "use env", but do people know why it's usually the best approach? In my experience, many don't, hence the blog post.

I mean, if it were obvious and accepted truth, Django would be doing this by default already.

The community writ large is already using django-environ to solve the problems you faced.

https://github.com/joke2k/django-environ

Reply on news.ycombinator.com