How I build software quickly
evanhahn.com
evanhahn.com
Here are some things I have learned:
* Learn one tool well. It is often better to use a tool that you know really well than something that on the surface seems to be more appropriate for the problem. For extremely large number of real-life problems, Django hits the sweet spot.
Several times I have started a project thinking that maybe Django is too heavy, but soon the project outgrew the initial idea. For example, I just created a status page app. It started as a single file Django app, but luckily realized soon that it makes no sense to go around Djangos limitations.
* In most applications that fit the Django model, data model is at the center of everything. Even if making a rought prototype, never postpone data model refactoring. It just becomes more and more expensive and difficult to change over time.
* Most applications don't need to be single-page apps nor require heavy frontend frameworks. Even for those that can benefit from it, traditional Django views is just fine for 80% of the pages. For the rest, consider AlpineHJS/HTMX
* Most of the time, it is easier to build the stuff yourself. Need to store and edit customers? With Django, you can develop simple a CRM app inside your app in just few hours. Integrating commercial CRM takes much more time. This applies to everything: status page, CRM, support system, sales processes, etc. as well as most Django apps/libraries.
* Always choose extremely boring technology. Just use python/Django/Postgres for everything. Forget Kubernetes, Redis, RabbitMQ, Celery, etc. Alpine/HTMX is an exception, because you can avoid much of the Javascript stack.
While I agree with you, these two are the boring tech of 2025 for me. They work extremely reliably, they have well-defined use cases where they work perfectly and we know very well where they shouldn't be used, we know their gotchas, the interest around them seems to slowly wane. Personally, I'm a huge fan of these, just because they're very stable and they do what they are supposed to do.
When you are building software really fast, you need to take the boringness to the next level and cut all non-essential components.
At fly.io, hosted Redis costs over $100/month, when one server + Postgres is less than $10/month. This tells you about the complexity.
Many people use Redis with Django, because that is "obvious choice" for Celery. But either Django built-in tasking (or backport with django-tasks) allows you to handle most of the use cases (long-running tasks, timed events) when combined with a cron such as Superchronic.
Many people consider ElasticSearch as an "obvious choice" to handle full-text search, but Postgres can handle it in most cases.
Mostly, if you have problems that require "webscale solutions", then you will have a team building a dedicated solution anyway.
1. I don't believe tenfold difference in pricing PostgreSQL vs Redis at fly.io is due to complexity, Postgres is much more complex. If I had to guess at why they priced it that way, I'd say it's because they choose more RAM-beefy machines to host their Redis instances. Other providers may have the pricing completely different.
2. It's been a decade that Lucene-based search engines are no longer the default choice. If for some reason PostresQL's FTS is not the right choice, there are many excellent alternatives like Meilisearch or Typesense.
2. It is surprisingly common to see people preferring Lucene-based search engines over Postgres. I don't know about your alternatives and not saying anything about them.
I would argue that k3s on a vm is not too much harder to setup than a normal reverse proxy setup. And the configs are a bit easier to find on a whim when coming back after months.
Obviously a full self hosted setup of k8s is pretty large task but in other forms its a competitive option in the "simple" area.
My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them.
I tried Longhorn, but even if I manage to get a volume showing up in the UI, it was never detected as healthy and after a while I refactored the workflow to use Apache Beam so that I could drop databases and volumes and run everything from within a single beefy VM.
Is it still an issue (it was a while ago TBH)?
If you are already familiar with k8s volumes and csi's its not a huge problem but if you arent its not worth learning if your goal is simple. At least in my opinion.
Yeah, that is also my opinion after that experience.
Also, it was really frustrating as the number of moving parts that have to play nice to each other is high.
Many hobby projects are just that, simple apis and can benefit from the simple declarative yamls that exist in say a sqlite db on the vm vs random config in /etc, /var and ip table scripts etc.
But trying to do anything out of that simple mold? just do it the simpler and easier traditional way.
I'd describe a really simple setup as this:
Pod: you put 1 container inside 1 pod - you can basically replace the word "container" with "pod". Let's say you have 1 backend in python and 1 frontend in React: you deploy 1 pod for your backend, and 1 pod for your frontend. The simplest way to deploy pods is not with "Pod" but with "Deployment" (1).
Service: Kubernetes really doesn't want you to connect directly to pods, it wants you to have a service in between. This is sensible because pods are ephemeral - they can be killed due to crashes, updates, etc. You define a service, point towards the pod, and all internal requests should be sent to the service, not to the pod. (2)
(1) https://kubernetes.io/docs/concepts/workloads/controllers/de...
(2) https://kubernetes.io/docs/concepts/services-networking/serv...
Note: technically, a pod is a group of containers. It can get more complex than this, but it's not unusual to have a group of 1 pod, especially in simpler setups.
In the deployments example, what gets me is they called 4 different things "nginx." It's hard to tell what does what. Like I've read 6 times what matchLabels does, and it's still not clear if that's supposed to mirror template.metadata.labels.app, or what happens if it doesn't. Sure I'd figure it out if I set up the real thing and messed with the config to see what it does, but it shouldn't take that.
Of course it's easy to follow tutorials without fully understanding how stuff works, and I did. Slightly more advanced things were a lot of trial and error. At some point I lost interest because I didn't really need this.
Keep in mind, for K8s to work you also need to containerize your application, even if it's just static files (or I guess you can break containment and do volume mounts, or some sort of static file resource if you're serving static content). For non-static applications, you still need to configure and run it, which is just harder with k8s/docker in the mix.
On the other hand if you are running thousands of containers it's hard to beat k8s, simpler solutions like ECS fall short at some point.
In my day job I work with Go and while it's fine, I end up writing 10x more code for simple API endpoints and as soon as you add query parameters for filtering, pagination, etc. etc. it gets even longer. Adding a permissions model on top does similar. Of course there's a big performance difference but largely the DB queries dominate performance, even in Python, at least for most of the things I do.
I'm writing a CRUD CLI ( https://github.com/bbkane/enventory/ ) partly to practice doing it in Go, and while I'm mostly happy with the resulting code, theres just a whole lot of it. At least it's simple enough that I can trust LLMs to ok jobs with detailed prompts (example: https://github.com/bbkane/enventory/blob/master/.github/prom... )
In Java land a very nice and much lighter weight framework is Dropwizard: https://github.com/dropwizard/dropwizard
It's basically a sort of All-Stars collection of Java libraries, nicely packaged and with some nice conventions.
Towards the more servless route route there's Micronaut: https://micronaut.io/
Personally I find Django and Rails too magical for my taste so I just use simpler libraries like Axum, but it's nice if you want all the bells and whistles. Plus, it's much faster than either Python or Ruby.
Still not the same bells and whistles as Django, but really good otherwise. The database interactions are pretty good.
as always: imho. (!)
hmmm ... i think spring-boot combined with either java or kotlin is a very good alternative to django.
even so i wouldn't compare them directly, but static typing avoids a lot of problems.
idk ... for me personally one of djangos great features is its custom db-model and relative (!) painless db schema-migration.
for spring-boot i often went with the tool flyway for handling db-migration ...
just my 0.02€
I find such a blanket opinion to be unhelpful, what's fine for writing microservices is less good for bootstrapping a whole SaaS app and I think that people get in a bit too much of an ideological tizz about it all.
It's mostly about libraries vs opinionated frameworks.
No one in their right mind would say: just use the standard library but I've seen it online. That discourse is not helping.
I think people get this miscontrued on both sides.
A set of reusable, composable libraries would be the right balance in Go. So not really a "framework" either.
I think that reflects better the actual preferred stance.
Just picking one of the examples I gave, pagination - that requires (a) query param handling (b) passing the info down into your database query (c) returning the pagination info in the response. In Django (DRF), that's all built in, you can even set the default pagination for every endpoint with a single line in your settings.py and write no more code.
In Go your equivalent would be wrangling something (either manually or using something like ShouldBindQuery in Gin) to decode the specific pagination query params and then wrangling that into your database calling code, and then wrangling the results + the pagination results info back.
Composable components therefore always leave you with more boilerplate
It also doesn't tie you to ORM usage.
You have to be responsible for "your" specific flow... meaning you can build your own defaults easily wrt parsing query parameters and whatnot (building a generic paginated query builder API?). Nothing insurmountable.
https://www.django-rest-framework.org/api-guide/pagination/#...
That's the same as for most other configurable things. And ultimately if you want to override the behaviour for some specific endpoint then you can still easily do that by just implementing your own method for it.
After working in Go now for several years, what I've found is that generally people just don't some things in their first pass because it's too much work and makes the PRs too large, and then they retrofit it afterwards. Which meant that the APIs were less consistent than the ones I worked on in Django.
Yes but...sometimes the proper level of abstraction is simply a good HTTP library. Where the API of your application is defined in terms of URLs, HTTP methods, status codes, etc., the http library from the Go standard library might be all you need (depending on other requirements, of course).
A real life example: I needed a simple proxy with well defined behavior. Writing it in Go as a web application that forwarded calls to another service based on simple logic only took a few pages of code in a single file.
I would say that.
The most important thing about standard library is its stability. You won't ever need to touch code that works with standard library. It's finished code. Other than bug fixes, of course.
Third-party libraries is a very different thing.
They gets abandoned all the time, so now you're left with burden. You either need to migrate to another library, maintain that abandoned library or live with huge chunk of code that might be vulnerable.
They gets changed often enough, as their developers probably not so careful about backwards compatibility, compared to core language developers.
Third-party library is a liability. Very rarely its source code is an ideal fit to your application. Often you'll use 10% of the library, and the rest is dead weight at best, vulnerability source at worst. Remember log4shell? Instead of using standard logging code in Java, some developer decides to pull log4j library which is very nice to use, has lots of features. It can even download and execute code behind your back, very featureful.
Of course I'm not advocating to rewrite the world. This is insane. Some problems are just too big to solve by yourself. I also should note, that different ecosystems have different approaches to the language library and overall library culture. JS is terrible, while Go is not that bad, but it's not ideal either.
But my absolutely 100% always approach is to use standard library first and foremost. I won't use third-party library just to save few lines of code or make code more "beautiful". I prefer dependency-free boring repetitive code any day.
And if I'm using third-party library, I'm very picky about its stability and transitive dependencies.
It also depends on kind of company. My experience has always been: you write some service, you throw it at production, it works for the next 20 years. So you want this code to be as self-contained as possible, to reduce "chore" time with dependency management. Perfect application is a dependency-free software which can be upgraded by changing "FROM" line in Dockerfile. It is stable enough that you can trust CI do that.
Also, by their very nature they almost require you to write a program of a minimum size even when a simpler program would do to solve your problem.
It goes like this: the framework helps with the initial progress, because framework works with the more typical use cases. This allows you to "prototype" the solution and show progress to potential users etc.
But then you have a specific use case that does not fit the model of the framework, and you will end up going around the limitions of the framework. You often end up spending more time that you saved with the initial implementation.
My experience with core Django is that it has so many hooks and ways to inherit functionality, that it is extremely rare to end up with such problems. But it still means you need to learn when to use what feature, and how to structure your solution. I had to learn this through painful experience. I am still making mistakes, such as the "single-file status page app", a mistake which cost me several hours of productive coding. That might not sound like much, but when working full-time and running several side projects, few productive hours is extremely costly.
Most third party Django apps, even the popular ones, suffer from the framework problem, however. For example, I have have had to rewrite django-ads and django-distill, because their structure was limiting me from implementing features I needed. I just ditched django-formset as well.
I believe the reason for the limitation is that these third party apps have not been exposed to enough use cases and users to overcome the framework problem.
I'm not sure I agree, sure they require you to pull in a large dependency, and that might not be good for some use cases (particularly microservices) where you're sensitive to the final size of the image but for e.g. with Django you get spawned a project with settings.py which configures your database, middleware, etc. and urls.py which configures your routes, and then you're really free to structure your models and endpoints however you like.
Hell, think twice before you consider postgres. Sqlite scales further than most people would expect it to, especially for local development / spinning up isolated CI instances. And for small apps it tends to be good enough for production too.
I'm particularly thinking of workers running tasks, here. It's possible to lock up everything with write transactions or cause a spate of unhandled SQLITE_BUSY errors in cases where Postgres would just keep chugging along.
I really don't consider it a good idea to use different databases on your production vs development vs CI instances. Just use PostgreSQL everywhere, it doesn't cost anything, scales down to almost nothing, and scales up to whatever you're likely to need in the future.
So lots of allocations/deallocations. If you're only storing a few key/value pairs long term, you won't have any issues.
Are you keeping the entire database in memory, or are you flushing to disk?
It seems wild that such a critical and pervasive piece of software would behave like that.
But as far as I can tell, it's more an OS issue than really a Sqlite issue, simply doing malloc/free in a loop results in a similar behaviour. And Sqlite does a lot of allocations. We see a similar problem on Windows, but not as pronounced, and there we can force the OS to reclaim memory.
It's probably solvable by using a custom allocator, but at this point it's no longer plug and play the way the GP meant.
Sqlite seems to be the hip new thing to use where MySQL should have been used in the first place. Sqlite is great for many things, but not for the classic CRUD web app.
But I am running two major side projects I while still working full time. One of my side projects is a site with 2M page views per month. Additionally, depending on how you count (whether apps are included in the major projects or not), I have 3-4 smaller side projects, two of which use SQLite database.
To be able to run all of this, I have an extreme cut-throat approach to technology. SQLite is almost there, but ultimately Postgres is still more mature all-around solution. As an example, I believe (have not tested this though) it is quicker to set up full-text search for Postgres using online resources. The developer experience is better, because Postgres full-text search has been around for longer and has wider adoption.
This isn't to complain about SQLite overall, it's perfect for local DBs like you'd see in app frontends or OS services.
as always: imho. (!)
while i personally really love SQLite for a lot of use-cases, i wouldn't recommend / use it "in serious production" for a django-application which does more than a simple "hello world".
why!? concurrency ... especially if your application attracts user or you just want to scale your deployment horizontally etc. ;))
so in my opinion:
* why not use sqlite for development and functionality testing
* postgresql or mariadb/mysql for (serious) production-instances :)
just my 0.02€
This would be true for any database, something read / written during a transaction would block at least that table universally until the transaction is finalised.
So I dunno why people insist on spreading so much FUD.
Doesn't that contradict "learn one tool well"?
I write every webpage in React, not because I think everything needs to be an SPA, but because enough things end up needing client-side state that I need something that can manage it well, and at that point it's easier to just do everything in React even if it initially seems like it would be too heavy, just like your example.
But let me elaborate my approach which explains this apparent contradiction. Backend is needed in any case, and for me that is almost always Django. It is extremely fast to build traditional (non-SPA) CRUD apps with Django if you have only one model / one page. I can make a new one less than an hour.
If I need more interactivity or more data on the page, I use the same approach (models, views, forms with validation) but I create partial forms and return HTML fragments, instead of full pages. AlpineJS/HTMX allows me to implement 80% with Django/python, and I can avoid implementing a REST API and JSON serialization/deserialization, all the frontend data management, and there is usually almost no need for Javascript.
Well, sort of. Much as I hate the idea, maybe Next.js for everything is the way to go, bashing out a CRUD app there is very quick.
> create partial forms and return HTML fragments, instead of full pages
I loved that approach with Wicket and made systems that way for many years, but it just isn't good enough these days, at least for a website that's going to be used in more than one location. When it takes a network roundtrip to update your UI it's always going to be noticeable.
I don't have experience with Wicket, but with HTMX, I have tried to make components very small. For example one switch button is one component. Loading that is very fast in my experience, as compared to reloading and rerendering all data with React etc.
Once you see the smoothness of fully local UI transitions compared to doing a WAN roundtrip, you can't go back, even if it's objectively not a huge difference. Things like switching between tabs, or hierarchical dropdowns where the choices in the second dropdown depend on what was chosen in the first dropdown, doing a network roundtrip to load an HTML fragment is maybe OK over a LAN but it's a nonstarter over the internet.
So your state management needs to be at least partially local to the frontend. And in my experience the only approaches available for that are a) React or b) ad-hoc hacks; some frameworks build the ad-hoc hacks into their builtin components so you get smooth local UI transitions as long as you're sticking to the happy path, but that ends up not being enough.
People are doing genuinely hugely impressive production-grade work there, and I am currently looking at a SigNoz dashboard while >200 users are frantically clicking on dozens of places in our UI (contains many reactive bits); our code is nowhere near optimal and we very rarely get >150ms response times, most are 20-50ms, and we're talking some quite tame machines (2 of them) with 8GB RAM and 2 CPU cores... and even then 90% - 99% of the time is spent waiting on the database.
I try to simplify the stack further and use SQLite with Borg for backups. Caching leverages Diskcache.
Deployment is slightly more complicated. I use containers and podman with systemd but could easily be a git pull & gunicorn restart.
My frontend practices have gone through some cycles. I found Alpine & HTMX too restrictive to my liking and instead prefer to use Typescript with django-vite integration. Yes it means using some of the frontend tooling but it means I can use TailwindCSS, React, Typescript etc if I want.
I use celery with the same code base/docker image. Just a different entry point to start a celery worker instead of a wsgi (web) worker.
Too many http requests? Add web worker instances.
Background jobs piling up? Add celery workers.
Clearly separate read endpoints from write/transactional endpoints and you can hit a slave postgres db or the master db depending on the http call.
This creates a very robust system that can scale easily from a single code base.
Out of interest, how do you run your migrations in production, deploy the service then run an ad-hoc job with the same container again? That was one thing I was never super happy with.
So new version can work with the previous version's DB schema.
Then, yes, simply run the migration in a transaction once the new code is deployed. Postgres has fully transactional DDL changes which helps.
Of course, it heavily depends on the actual change being made. Some changes will require downtime or must be avoided if it is too heavy on the db.
Another approach if the migration can be applied quickly is to just run the migrations as part of the deployment script. This will cause a downtime but can be short.
Easiest is just to do runmigrations in your docker image start commands, so DB is always migrated when the container starts.
tl;dr: It depends on the complexity of the migrations and your uptime requirements.
But then I found out that background workers are being implemented to Django.
https://github.com/django/deps/blob/main/accepted/0014-backg...
This functionality has been backported as django-tasks library. You can use the database (i.e. Postgres) as the permanent storage.
For periodic jobs and running the django-tasks, I use Superchronic, which is a Docker-compatible cron. It is compatible with cron, with the added benefit that you can run jobs with sub-minute granularity. I have had no problems with running Superchronic inside my fly.io app.
I run Superchronic and Django with honcho and Procfiles.
Maybe I'm missing it, but I don't think any of these has a nice simple built-in feature.
I love finding out when celebrities have talents elsewhere. And Wikipedia says you've had quite a bit of aviation experience as well.
Kinda makes my morning... lol
p.s. The inner city cynical kid in me is now required to throw in that I found Django disappointing even though every else seems to love it. Ok... identity restored... can now go back to shaking my fist at the sky as per usual...
Knowing a good Swiss army tool very well is a super power.
How do you do background job processing without Celery? Every app/website I ever developed required some sort of background job processing, for Python I use Celery, Rails I use Sidekiq, Node.js I use Faktory. One of the biggest drawbacks (at least until a while ago) was that setting this up had to be a hacky webhook call to your own app that allowed up to N seconds request/response.
Celery is a solution to scalability, not to any business problem in particular.
I also have multiple celery applications, but I wouldn't recommend it for smaller or simplier apps.
Can you explain what you mean?
Sure, Django,Rails,Asp MVC,etc with static pages could be fully functional for something that is 99% CRUD.
But I myself foolishly tried using static view+HTMX for something that should have been built with something more dynamic like Vue/React(perhaps Alpine?) from the start due to actual client-side complexity.
In general imho one should classify what your main interaction points are and use the correct tools for each.
- Is it static pages for masses of consumers with little to no interaction? (Ie a news-site,etc) then fast SEO-friendly pages is your nr 1 priority.
- Is it mainly massive CRUD updates that needs persistence, go with server built pages with Django, Rails, etc.
- Is it an actual client/data heavy UI (games but also scenarios where you want fast client data slicing or processes/sources that complete at different paces), use an SPA.
Ie, right tool for the right job.
But when you say you had a "static view" (in singular, not plural) with HTMX, what do you mean? I am very interested in hearing what kind of complex interaction you had where that approach failed?
With HTMX, I have found the most useful approach is to build your pages from lots of very small components. Initially, I don't worry about reuse of those components, it is more about decomposition than reuse.
Of course, if you want to do "fast client data slicing", then HTMX makes no sense, because it pushes logic to the backend and by definition your solution wants to process data in the fronted. So if you start from that premise, HTMX is obviously the wrong tool.
If you want to use HTMX, you need to rethink your solution, i.e. question whether you really need "fast client data slicing". So why bring all the data to the frontend, and not just the slice you want to visualize?
One reasonable use case is reducing server load. I have one app where data is loaded from static generated pages (periodically updated, for performance, stability and reduced server load), and sorting is done on the frontend using Javascript, based on data attributes on the DOM.
That works well until that tool stops being maintained, or is evolving in such an odd direction that it breaks your code.
Django is really an exception to something one can rely on, but reality is more like Angular -> breaking, Vue -> breaking, React -> breaking.
You clearly work on web applications of moderate scale. For example, Kubernetes and Redis can suddenly become almost necessities for a back end service once it reaches a certain scale.
As one of my side projects, I am running a website with 2M page views per month that has been implemented on Django, deployed to fly.io. Another side-project built-in tenant support, and several independent deployments as well.
Yes, those are moderate scale (for modern hardware), but in my experience very few sites really require "webscale". It is far more common to build prematurely for scalability than the opposite. If you have "webscale" problems, then you also have have a custom platform and team working on those.
The context of this discussion is how to build robust systems as fast as possible. My approach fits surprisingly large number of use cases.
Instead, it becomes "final ship" code.
I tend to write ship code from the start, but do so, in a manner that allows a lot of flexibility. I've learned to write "ship everywhere," even my test harnesses tend to be fairly robust, ship-Quality apps.
A big part of that, is very high-Quality modules. There's always stuff that we know won't change, or, if so, a change is a fairly big deal, so we sequester those parts into standalone modules, and import them as dependencies.
Here's an example of one that I just finished revamping[0]. I use it in this app[1], in the settings popover. I also have this[2] as a baseline dependency that I import into almost everything.
It can make it really fast, to develop a new application, and can keep the Quality pretty high, even when that's not a principal objective.
[0] https://github.com/RiftValleySoftware/RVS_Checkbox
[1] https://github.com/RiftValleySoftware/ambiamara
[2] https://github.com/RiftValleySoftware/RVS_Generic_Swift_Tool...
It becomes quickly very visually dominant in the source code:
> / ###################################################################################################################################### / // MARK: - PUBLIC BASE CLASS OVERRIDES - / ###################################################################################################################################### */
My comment/blank line-to-code ratio is about 50/50. Most of my comments are method/function/property headerdoc/docc labels.
Here's the cloc on the middle project:
github.com/AlDanial/cloc v 2.04 T=0.03 s (1319.9 files/s, 468842.4 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Swift 33 1737 4765 5220
-------------------------------------------------------------------------------
SUM: 33 1737 4765 5220
------------------------------------------------------------------------------- ###############################################################################
############################### LIBRARY IMPORTS ###############################
###############################################################################
import sys
from pathlib import Path
If you're building something yourself or in a small team, I absolutely agree with everything written in the post. In fact, I'd emphasize you should lean into this sort of quick and dirty development methodology in such a context, because this is the strength of small scale development. Done correctly it will have you running circles around larger operations. Bugs are almost always easy to fix later for a small team or solo dev operation as you can expect everyone involved to have a nearly perfect mental model of the entire project, and the code itself will regardless of the messes you make tend to keep relatively simple due to Conway's law.
In larger development projects, fixing bugs and especially architectural mistakes is exponentially more expensive as code understanding is piecemeal, the architecture is inevitably nightmarishly complex (Conway again), and large scale refactoring means locking down parts of the code base so that dozens to hundreds of people can't do anything (which means it basically never happens). In such a setting the overarching focus is and should be on correctness at all steps. The economies of scale will still move things forward at an acceptable pace, even if individual developers aren't particularly productive working in such a fashion.
(Sidenote: having this kind of architecture where you create layer of deps from one team to another is a bad idea from my point of view, but is still done a lot)
What you're describing is somewhere in the middle (if you imagine a logarithmic scale), it's at a point where working like a solo dev begins to break down especially over time, but not at a point where it's immediately catastrophic.
Startups sometimes work in that sort of hybrid mode where they have relatively low quality code bordering on unmaintainability, where they put off fixing its problems into the future when they've made it big.
Services that can each be maintained by a small team, with a clean, understandable API, appropriate protections for data (both security and consistency) and easily predictable cost and behavior.
Then you can compose these services to get more complex functionality.
Whenever I asked customers what they wanted from their new systems they always started by saying we want to match the existing system.
Hate to be an anecdote Andy here, but as someone who has done a lot of code review at (non-game) hackathons in the past (primarily to prevent cheating), the teams that performed the best were also usually the ones with the best code quality and often at least some rudimentary testing setup.
But, in another sense, they didn't "prioritize" good code because it isn't really a tradeoff. They're just better.
Systems like UE blueprints showcase how pointless the pursuit of clean anything is when contrasted with the resulting product experiences.
Im pretty sure the latter vastly outpeform the former under all circumstances - whether quick and dirty hackathon or ultra hardened production code.
This is the exact opposite of my experience. Every time I am playing around with something, I feel like I'm experiencing all of its good and none of its bad ... a honeymoon phase if you will.
It's not until I need to cover edge cases and prevent all invalid state and display helpful error messages to the user, and eliminate any potential side effects that I discover the "unknown unknowns".
Tools aside, I think everyone who has 10+ years can think of a time they had a prototype go well in a new problem space only to realize during the real implementation that there were still multiple unknown unknowns.
The part about distraction in code feels also very real. I am really prone to "clean up things", then realize I'm getting into a rabbit hole and my change grows to a size that my mates won't be happy reviewing. These endeavors often end with complete discard to get back on track and keep the main thing small and focused - frequent small local commits help a lot here. Sometimes I manage to salvage something and publish in a different PR when time allows it.
Business mostly wants the result fast and does not understand tradeoffs in code until the debt hits the size of a mountain that makes even trivial changes painfully slow. But it's about balance, which might be different on different projects.
Small, focused, simple changes definitely help. Although, people are not always good at slicing a larger solution into smaller chunks. I sometimes see commits that ship completely unused code unrelated to anything with a comment that this will be part of some future work...then prio shifts, people come and go, and a year later we have to throw out all of that, because it does not apply to the current state and no one knows anymore what was the plan with that.
The perpetual looming threat of layoffs, the need to deliver wins ASAP, stifles creativity, punishes experimentation, and pushes people to burnout. It forces people into groupthink about topics like AI. Nobody can say the emperor has no clothes (emperor being leadership or the topic du-jour)
Forget LLM coding, solve this problem...
Consumers that have good taste (or at least perceive differences in quality) are not numerous enough to support new products that differentiate themselves only with quality. And they (the consumers) are not successful enough in their own enterprises to pay extra for better quality products.
It's easier to find examples where people do pay for quality outside of software. Look at the spectrum in quality available for vehicles or household appliances.
So much this.
Get the data model right before you go live, and everything is so simple, get it wrong and be prepared for constant pain balancing real data, migrations, uptime and new features. Ask me how I know
I'm not convinced this was a good abstraction that really helps us be more effective.
Some omissions during initial development may have a very long tail of negative impact - obvious examples are not wiring in observability into the code from the outset, or not structuring code with easy testing being an explicit goal.
Presence of a lab notebook allows me to write better documentation faster, even if I start late, and tests allow me to verify that the design doesn't drift over time.
Starting blind-mindedly for a one-off tool written in a weekend maybe acceptable, but for anything going to live longer, building the slow foundation allows things built on this foundation to be sound, rational (for the problem at hand) and more importantly understandable/maintainable.
Also, as an unpopular opinion, design on paper first, digitize later.
> Six months ago, only I and God knew how this code worked. Now, only God knows. :)
So that's not a problem with this process itself. You're describing problems with managers, and problems with developers being unable to handle bad managers.
Even putting aside the manager's incompetence, as a developer you can mitigate this easily in many different ways, here's a few:
- Just don't show it to management
- Deliberately make it obviously broken at certain steps
- Take screen shots of it working and tell people "this is a mockup, I still have to do the hard work of wiring it up"
It's all a balancing act of needing to get feedback from shareholders and managing expectation. If your management is bad, you need to put extra work into managing expectations.
It's like the famous duck story, from Jeff Atwood (see jargon number 4), sometimes you have to manage your managers:
From launch to failure is definitely getting fast-tracked; few months ago we had yet another hospital system that just lost data; reading the code (no tests, no peer reviews; they don't use versioning) shows clear signs of LLMs; many files that do almost the same thing, many similar function names that do almost the same thing or actually the same thing, strange solutions to trivial problems, etc. The problem was that there was a script ran at startup which added an admin user, but if an admin user already exists, it truncates the table. No idea why, but it wasn't discovered earlier because after testing by the devs it was put live, devs left (contractors) and it just ran without issues until the ec2 instance needed maintenance by aws and was as such rebooted after which all users were gone. Good stuff. They paid around 150k for it; that is not a lot in our field but then to get this level of garbage is rather frightening. This was not life threatening, but I know if it was, it would not be better as you cannot make this crap up.
The big problem: the decision makers(c-suite executives) never really understood what was happening before, so you can't expect them to see the root cause of the problem we're actively creating. This means it will not get the attention and resourcing needed - plus they'll be gone and on to the next one after taking a huge payday for slashing their R&D costs.
I fear this is different from the "code slop jello poured over a bespoke marshmallow salad of a system" problem though. Mostly for the same reasons that Brooks described that make SW inherently hard 60+ years ago. It feels like the JS framework / SPA experience but with every.single. developer. and the 10x "improvement" is just speed.
Having worked on some 24-hour game jams and similar, I've found completely the opposite. It's when you're in a real hurry that you really can't afford bad code. Writing better code will make it easier to get it right, will put less pressure on my working memory, will let me try things faster and make those last-minute changes I wanted, will make adding features towards the end easier rather than harder and, crucially, will both reduce the chance that I need to do intense debugging and make debugging I need to do easier.
Working with good code just feels lighter.
The thing that breaks 24-hour projects isn't writing code too slowly, it's writing yourself into a corner or hitting some problem that derails your work, takes hours to solve or doesn't even get resolved until after the deadline.
A game jam isn't the place to try to squish all bugs, sure, but that's a question of what you're doing, not how. I still want to write good code in that situation because it makes my life easier within the time constraints, and because, even if I'm fine with some bugs, there are still lots of bugs that render a game unpleasant or unplayable.
I'll need to fix some bugs no matter what; I'd rather fix fewer, easier bugs than more, harder bugs!
The same thing applies to longer time horizons too. When you have more time you have more slack to deal with bad code, but that doesn't mean it makes any more sense to write it!
And, of course, once you get in the right habits, writing solid quality code becomes basically free... but, even if it really did meaningfully slow you down, chances are it would still be worth doing in expectation.
As the above comment says, in my experience bugs introduced from messy code are way more likely than the time savings of not cleaning up code.
The usual exception I'd make are things that like, mostly the same but not quite (e.g. a function to fade out a light over time vs a function to fade out a color over time). Often I find requirements for those diverge over time, so I'll just leave some repeated patterns across both.
Or if you need to do some pathing and your quickest solution is some breadth first search.
Perhaps it isn't "bad code", but still a bad solution that could quickly be implemented and be solved by a lot of computing power.
Of course you may use ready-to-use modules that provide such features as well..., but it may be prohibited by competion rules, I don't know...
Sucks, as that is effectively arguing for rote repetition in the tasks. And it is. But, that works. Really well.
Stated differently, show me someone that can write clean code in a hurry, and you have shown me someone that has written this before.
I think it's a misconception that writing good code must take longer than writing bad code. At least if you want it to vaguely satisfy some requirements.
I hop between projects regularly, and this has been the biggest source of inter-team conflict in my career.
Different people from different backgrounds have an assumed level of what "good enough". The person from big tech is frustrated because no one else is testing thoroughly. The person from a startup is frustrated because everyone else is moving too slow.
It would be nice if the "good enough" could be made explicit so teams are on the same page.
This doesn't mean writing tests for everything, and sometimes it means not writing tests at all, but it means that I do my best to make code "testable". It shouldn't take more time to do this, though: if you're making more classes to make it testable, you're already messing it up.
This also doesn't mean compromising in readability, but it does mean eschewing practices like "Clean Code". Functions end up being as large as they need to be. I find that a lot of people doing especially Ruby and Java tend to spend too much time here. IMO having lots of 5-line functions is totally unnecessary, so I just skip this step altogether.
It also doesn't mean compromising on abstractions. I don't even like the "rule of three" because it forces more work down the line. But since I prefer DEEP classes and SMALL interfaces, in the style of John Ousterhout, the code doesn't really take longer to write. It does require some thinking but it's nothing out of the ordinary at all. It's just things that people don't do out of inertia.
One thing I am a bit of hardliner about is scope. If the scope is too large, it's probably not prototype or MVP material, and I will fight to reduce it.
EDIT: kukkeliskuu said below "learn one tool well". This is also key. Don't go "against the grain" when writing prototypes or first passes. If you're fighting the framework, you're on the wrong path IME.
But I am also pretty disciplined on the 2nd pass in correcting all of the hacks and rewriting everything that should be rewritten.
There are two problems I have with trying to do it right the first time:
- It's hard to know the intricacies of the requirements upfront without actually implementing the thing, which results in designing an architecture with imperfect knowledge
- It's easy to get stuck in analysis paralysis
FWIW I am a huge fan of John Ousterhout. It may be my all time favorite book on software design.
So I don't really want to know the future requirements, or refactor on the 2nd pass to "match".
If some feature needs too many modifications or special cases in the current architecture, it's a round peg in a round hole. I prefer to have those places be a bit more "painful" in the code. The code doesn't have to be bad per se, but it should be clear that something different and not traditional is happening there.
1: Tacticale/shareable/paintable in a physical meeting
2: It drives home that it's a sketch, with bad customers a visible UI so easily hides the enormous amounts of complexity that can sometimes be under a "simple" UI and make it hard to educate them as to why the UI sketch one did in an hour or two then needs 1500 hours of engineering to become a functioning system.
I usually start top-down, sketching the API surface or UI scaffold before diving into real logic. Iteration drives the process: get something running, feel out the edges, and refine.
I favor MVPs that work end-to-end, to validate flow and reduce risk early. That rhythm helps me ship production-ready software quickly, especially when navigating uncertainty.
One recent WIP: https://zero-to-creator.netlify.app/. I built it for my kid, but I’m evolving it into a full-blown product by tweaking the edges as I go.
I do admit that modern frameworks also help in that regard, instead of just stitching libraries together, at least for typical webdev stuff instead of more minimalistic CLI utilities or tools. The likes of Ruby on Rails, Django, Laravel, Express and even the likes of ASP.NET or Spring Boot.
Reason (my context: small team(s) but in a somewhat complex and integrated environment):
Many times, bugs in production cost the company (IT and business areas) much much more to identify/correct/resolve then making sure the stuff works in the first place.
Incorrect transactions flowing between different systems and different areas of the business and possibly out to customers and other external partners are time consuming to correct, or require costly manual labor to work around.
Sometimes a redesign of the types you relied on becomes necessary to accommodate new stuff, but that would be true under any language; otoh, the "exploratory" part of coding feels faster and easier.
1. I use a custom code generator to generate 90%+ of the code I need from a declarative spec.
2. The remaining hand written code is mostly biz logic and it is exhaustively auto tested at the server API level.
3. I maintain and "bring along" a large library of software tools "tested in combat" I can reuse on new projects.
4. I have settled on an architecture that is simple but works and scales well for the small to large biz applications I work on.
5. I constantly work on removing complexity from the code, making it as easy to understand and modify as possible.
What programming language you like for that?
The author said LLM helps. Let's lynch him!
I recently re-made my web based notes app. Before working on this project I made a web based S3 file manager (e.g. CRUD operations in my own UI).
Instead of trying to store notes in a database or something, I just yoinked the S3 file manager code and store my notes in S3. Just a few tweaks to the UI and a few more features and now I have a notes app.
Something to keep in mind for design/development/testing.
Yessir, that is the way to speed. You're composing and climbing to higher levels of abstraction while the competition is trying to synchronize their accidental state machine network.
It helps reveal unknowns in the problem space that synthetic data might miss.
It's possible, but requires designing a safe way to run pre-production code that touches production data. Which in practice means you better be sure you're only doing reads, not writes, and running your code in the production environment with all the same controls as your production code.
I meant data heterogeneity - the variety in formats, edge cases, and data quality you encounter in production. Real user data often has inconsistencies, missing fields, unexpected formats, etc. that synthetic test data tends to miss.
This helps surface integration issues and performance bottlenecks early.
Basically there are things you can't avoid that are not necessarily fast (e.g. compilation, docker build, etc.) and things that you can actually control and optimize. Tests and integration tests are part of that. Learning how to write good effective tests that are quick to run is important. Because you might end up with hundreds of those and you'll be spending a lot of your career waiting for those to run. Over and over again.
Here's what I do:
- I run integration tests concurrently. My CPUs max out when I run my tests. My current build runs around 400 integration tests in about 35 seconds. Integration test means the tests are proper black box tests that hit a REST API with my server talking to a DB, Elasticsearch and Redis. Each test might require users/teams and some content set up. We're talking many thousands of API calls happening in about 35 seconds.
- There is no database cleanup in between tests. Database cleanup is slow. Each build starts with an ephemeral docker container. So it starts empty but by the time the build is over you have a pretty full database.
- To avoid test interaction, all data is randomized. I use a library that generates human readable names, email addresses, etc. Creating new users/teams is fast, recreating the database schema isn't. And because at any time there can be 10 separate tests running, you don't want this anyway. Some tests share the same read only test fixture and team. Recreating the same database content over and over again is stupid.
- A proper integration test is a scenario that is representative of what happens in your real system. It's not a unit test. So the more side effects, the better. Your goal is to find anything that might break when you put things together. Finding weird feature interactions, performance bottlenecks, and sources of flakiness is a goal here and not something you are trying to avoid. Real users don't use an empty system. And they won't have it exclusive to themselves either. So having dozens of tests running at the same time adds realism.
- Unit tests and integration tests have different goals. With integration tests you want to cover features, not code. Use unit tests for code coverage. The more features an integration test touches, the better. There is a combinatorial explosion of different combinations of inputs. It's mathematically impossible to test all of them with an integration test. So, instead of having more integration tests, write better scenarios for your tests. Add to them. Refine them with detail. Asserting stuff is cheap. Setting things up isn't. Make the most of what you setup.
- IMHO anything in between scenario tests and unit tests is a waste of time. I hate white box tests. Because they are expensive to run and write and yet not as valuable as a good blackbox integration test. Sometimes you have to. But these are low value, high maintenance, expensive to run tests. A proper unit tests is high value, low maintenance and very fast to run (it mocks/stubs everything it needs, there is no setup cost). A proper integration tests is high value, low maintenance, and slow to run. You justify the time investment with value. Low maintenance here means not a lot of code is needed to set things up.
- Your integration test becomes a load and stress test as well. Many teams don't bother with this. I run mine 20 times a day. Because it only takes less than a minute. Anything that increases that build time, gets identified and dealt with. My tests passing gives me a high degree of certainty that nothing important has broken.
- Most of the work creating a good test is setting up the given part of a BDD style test. Making that easy with some helper functions is key. Most of my tests require users, teams, etc. and some objects. So I have a function "createTeam" with some parameters that call all the APIs to get that done. This gets called hundreds of time in a build. It's a nice one liner that sets it up. Most of my tests read like this: create a team or teams, do some stuff, assert, do more stuff, assert, etc.
- Poll instead of sleeping. A lot of stuff happens asynchronously so there is a lot of test code that waits for shit to happen. I use kotest-assertions which has a nice "eventually" helper that takes a block and runs that until it stops throwing exceptions (or times out). It has configurable interval that it tries again that backs off with increasing sleep periods. Most things just take a second or two to happen.
- If your CPUs are not maxed out during the test, you need to be running more tests, not less. Server tests tend to be IO blocked, not CPU blocked. And your SSD is unlikely to be the bottleneck. We're talking network IO here. And it's all running on localhost. So, if your CPUs are idling, you can run more tests and can use more threads, co-routines, whatever.
- Get a decent laptop and pay for fast CI hardware. It's not worth waiting 10 minutes for something that could build in about a minute. That speedup is worth a lot. And it's less likely to break your flow state.
This stuff is a lot easier if you engineer and plan for it. Introducing concurrently running tests to a test suite that isn't ready for it can be hard. Engineering your tests to be able to support running concurrently results in better tests. So if you do this properly, you get better tests that run faster. Win win. I've been doing this for a while. I'm very picky about what is and isn't a good test. There are a lot of bad tests out there.
This is a pretty easy and natural thing to do because it's quite easy to go "I shaved 2.5 minutes off my build" whereas "I increased the maintainability and realism of our tests, adding 3 minutes to the build" is a much more nebulous and hard thing to justify even when it does save you time in the long run.
As Drucker says, what gets "measured gets managed" <- quantifiable metrics get more attention even when they're less important.
>A proper unit tests is high value, low maintenance and very fast to run (it mocks/stubs everything it needs, there is no setup cost).
^^ this is a case in point, mocks and stubs do make fast running test code but they commensurately decrease the realism of that test and increase maintenance overhead. Even in unit tests I've shifted to writing almost zero mocks and stubs and using only fakes.
I've had good luck writing what I call "end to end unit tests" where the I/O boundary is faked while everything underneath it is tested as is, but even this model falls over when the I/O boundary you're faking is large and complex.
In database heavy applications, for instance, so much of the logic will be in this layer that a unit test will demand massive amounts of mocks/stubs and commensurate maintenance and still tell you almost nothing about what broke or what works.
A slow test will be something you avoid running when you should be. When it takes 20 minutes to validate a change you did it gets tempting to skip it or you post pone doing it. Or you'll do it and get side tracked by something else. The ability to run high quality integration tests quickly is a super power. We're used to these things running slowly but my point is that you can engineer it such that it's fast and that's worth doing.
IMHO a key mistake is treating integration tests as unit tests. Which then dictates you do silly things like running expensive cleanup logic and isolating tests for each other and giving them their own virgin system to run against. That actually makes your tests less valuable and more tedious to run.
The real world is messy. A good integration test benefits from the noise created by lots of other tests running. It's the closest you can get to a real running live system without using the actual live running system and testing in production. Real users will never see a virgin system and they won't be alone in the system. It's OK for there to be data in the database. You can isolate through other means: give tests their own users. Randomize key things so they don't clash with other tests, etc. This results in better tests that actually run faster.
I love my unit tests as well. But I don't unit test things that I cover with an integration test anyway. I reserve those for things that are proper units that I can easily test in isolation. Anything with complicated logic, regular expressions, or algorithms basically. Testing that with an integration tests is counter productive because your goal is to test the logic and you probably want to do that with lots of different inputs. And then you mock/fake anything that just gets in the way of testing that.
But unit testing APIs is silly if you are in any case writing proper full blown integration / scenario tests that use those APIs. I don't need to unit test my database layer with an in memory database either. If it's at all important, that functionality will be used as part of my integration tests triggering logic that needs a database. And it will run on a proper database. And I can use my API from the outside to evaluate the output and assert everything is as it should be without poking around in that database. This adds more API calls and realism to the test and ensures I don't need to make assumptions about the implementation. Which then means I can change the implementation and validate that it didn't break anything important.
Lightwork