I'll definitely be looking at further tweaking vacuuming based on this article.
I'll definitely be looking at further tweaking vacuuming based on this article.
If you're using docker/podman or docker-compose and your db size is small, a major speedup on linux is to just mount the entire data dir into memory with --tmpfs /var/lib/postgresql/data (or tmpfs: - /var/lib/postgresql/data in docker-compose)
Additionally, if you constantly reset your db in the tests, consider making a template db at the start and later just doing CREATE DATABASE ... TEMPLATE foo; to copy the pages from that template instead of running migrations that produce WAL log. In fact, consider making a db for every test suite from that template at the start - then you can run each suite in parallel (if your app's only state is the db and a single backend).
A few more application-level optimisations: if you need to clear your database between tests, the best way is to wrap each test in a transaction and roll it back (Postgres supports nested transactions with checkpoints which might help make that transparent to the application under test). The second best way is usually to use DELETE rather than TRUNCATE, the latter is much slower on small tables (but much faster on big ones). The third fastest way is to create a new DB from template, although it plays nice with parallelisation.
Just made me curious.
How is memory consumption during the tests? If you run out of memory on a tmpfs, you'll hit swap and that'll slow you down considerably.
Do you see other disk access during the test?
I asked Karen Jex this question at her Djangocon 2023 talk on obscure postgres switches.
I used this among other things to build some model integration tests which exhibit best practices: https://github.com/hitchdev/hitchstory/tree/master/examples/...
I considered using tmpfs but since I wanted to cache the entire database volume after it had been built according to the fixtures and couldnt figure out how to do that with podman and tmpfs.
I just disable autovacuum completely for test runs with autovacuum=off. Remember that it kicks in according to the percentage of data updated so with a table which starts empty you are guaranteed to see autovacuum run once you insert a few rows.
If you have a lot of updates you can also try change the table fillfactor. This makes HOT updates a bit more efficient.
https://postgresqlco.nf/doc/en/param/fsync/#:~:text=Setting%....
For situations like these, creating the database in a tmpfs volume and having enough memory on the db host would do wonders.