Hacker News Reader

Comment by kukkeliskuu | Hacker News Reader

In recent years, I have learned how to build sufficiently robust systems fast.

Here are some things I have learned:

* Learn one tool well. It is often better to use a tool that you know really well than something that on the surface seems to be more appropriate for the problem. For extremely large number of real-life problems, Django hits the sweet spot.

Several times I have started a project thinking that maybe Django is too heavy, but soon the project outgrew the initial idea. For example, I just created a status page app. It started as a single file Django app, but luckily realized soon that it makes no sense to go around Djangos limitations.

* In most applications that fit the Django model, data model is at the center of everything. Even if making a rought prototype, never postpone data model refactoring. It just becomes more and more expensive and difficult to change over time.

* Most applications don't need to be single-page apps nor require heavy frontend frameworks. Even for those that can benefit from it, traditional Django views is just fine for 80% of the pages. For the rest, consider AlpineHJS/HTMX

* Most of the time, it is easier to build the stuff yourself. Need to store and edit customers? With Django, you can develop simple a CRM app inside your app in just few hours. Integrating commercial CRM takes much more time. This applies to everything: status page, CRM, support system, sales processes, etc. as well as most Django apps/libraries.

* Always choose extremely boring technology. Just use python/Django/Postgres for everything. Forget Kubernetes, Redis, RabbitMQ, Celery, etc. Alpine/HTMX is an exception, because you can avoid much of the Javascript stack.

View on HN

> Forget Kubernetes, Redis

While I agree with you, these two are the boring tech of 2025 for me. They work extremely reliably, they have well-defined use cases where they work perfectly and we know very well where they shouldn't be used, we know their gotchas, the interest around them seems to slowly wane. Personally, I'm a huge fan of these, just because they're very stable and they do what they are supposed to do.

I am not going to argue with you about specific technology or use cases. But these are complex pieces of software, and the fact that you consider them to be "boring technology" is the reason why I was talking about "extremely boring technology".

When you are building software really fast, you need to take the boringness to the next level and cut all non-essential components.

At fly.io, hosted Redis costs over $100/month, when one server + Postgres is less than $10/month. This tells you about the complexity.

Many people use Redis with Django, because that is "obvious choice" for Celery. But either Django built-in tasking (or backport with django-tasks) allows you to handle most of the use cases (long-running tasks, timed events) when combined with a cron such as Superchronic.

Many people consider ElasticSearch as an "obvious choice" to handle full-text search, but Postgres can handle it in most cases.

Mostly, if you have problems that require "webscale solutions", then you will have a team building a dedicated solution anyway.

There are a couple of points I'd disagree here.

1. I don't believe tenfold difference in pricing PostgreSQL vs Redis at fly.io is due to complexity, Postgres is much more complex. If I had to guess at why they priced it that way, I'd say it's because they choose more RAM-beefy machines to host their Redis instances. Other providers may have the pricing completely different.

2. It's been a decade that Lucene-based search engines are no longer the default choice. If for some reason PostresQL's FTS is not the right choice, there are many excellent alternatives like Meilisearch or Typesense.

1. fly.io sells redis hosted by upstash. But you are correct in the sense that default fly.io PostgreSQL was unmanaged and probably subsidized, and recent PostgreSQL is probably more expensive. But while I think you cannot directly say that tenfold increase in price means tenfold increase in complexity, I am just saying that it is complex.

2. It is surprisingly common to see people preferring Lucene-based search engines over Postgres. I don't know about your alternatives and not saying anything about them.

A lot of the conversation about "simplicity" with kubernetes ignores that there are MANY different distributions of k8s.

I would argue that k3s on a vm is not too much harder to setup than a normal reverse proxy setup. And the configs are a bit easier to find on a whim when coming back after months.

Obviously a full self hosted setup of k8s is pretty large task but in other forms its a competitive option in the "simple" area.

Last time I tried what stumped me was getting "ReadWriteMany" volumes working.

My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them.

I tried Longhorn, but even if I manage to get a volume showing up in the UI, it was never detected as healthy and after a while I refactored the workflow to use Apache Beam so that I could drop databases and volumes and run everything from within a single beefy VM.

Is it still an issue (it was a while ago TBH)?

Ya this kind of usecase is for sure straying into the "this is more of a pain than its worth" with regards to k8s. Esp when you can just do basic nfs mounts in nix anyways.

If you are already familiar with k8s volumes and csi's its not a huge problem but if you arent its not worth learning if your goal is simple. At least in my opinion.

Thank you for taking the time to confirm this.

Yeah, that is also my opinion after that experience.

Also, it was really frustrating as the number of moving parts that have to play nice to each other is high.

K8s and its derivatives ultimately were designed to declaratively orchestrate stateless apis/middlewares. It of course can quite competently do many other things. But it does what it was BUILT to do originally the best and the simplest.

Many hobby projects are just that, simple apis and can benefit from the simple declarative yamls that exist in say a sqlite db on the vm vs random config in /etc, /var and ip table scripts etc.

But trying to do anything out of that simple mold? just do it the simpler and easier traditional way.

Use a bucket for this. Or if you’re in a home lab use a boring old NFS server outside k8s

Thanks. The Apache beam code in the end doesn't need a bucket but the HDD attached to the VM was more than enough to store everything and then moving things out of it as the final step.

K8s is chock full of complexity though. Complexity that the vast majority of applications do not, and will not need

I only used it once forever ago, and all I remember is being very confused about pod vs service and using a lot of magic strings to get load-balancing to work. Sure I could figure it out, but didn't want to.

Kubernetes is really complex but I'm surprised by this - for a simple setup, I think those 2 resources are not that difficult.

I'd describe a really simple setup as this:

Pod: you put 1 container inside 1 pod - you can basically replace the word "container" with "pod". Let's say you have 1 backend in python and 1 frontend in React: you deploy 1 pod for your backend, and 1 pod for your frontend. The simplest way to deploy pods is not with "Pod" but with "Deployment" (1).

Service: Kubernetes really doesn't want you to connect directly to pods, it wants you to have a service in between. This is sensible because pods are ephemeral - they can be killed due to crashes, updates, etc. You define a service, point towards the pod, and all internal requests should be sent to the service, not to the pod. (2)

(1) https://kubernetes.io/docs/concepts/workloads/controllers/de...

(2) https://kubernetes.io/docs/concepts/services-networking/serv...

Note: technically, a pod is a group of containers. It can get more complex than this, but it's not unusual to have a group of 1 pod, especially in simpler setups.

The thing about managing pods using deployments rings a bell, but most official Kubernetes tutorials/examples (at least back then) were declaring "kind: Pod" directly which gave mixed signals, maybe because deployment was a newer feature. Per the service doc you linked, services connect directly to pods rather than deployments, using a seemingly magic "kubernetes.io/service-name" label.

In the deployments example, what gets me is they called 4 different things "nginx." It's hard to tell what does what. Like I've read 6 times what matchLabels does, and it's still not clear if that's supposed to mirror template.metadata.labels.app, or what happens if it doesn't. Sure I'd figure it out if I set up the real thing and messed with the config to see what it does, but it shouldn't take that.

Of course it's easy to follow tutorials without fully understanding how stuff works, and I did. Slightly more advanced things were a lot of trial and error. At some point I lost interest because I didn't really need this.

You can treat Pod as a container (or set of containers in some cases) and Service as its IP address (whether visible from inside the cluster, outside etc. - that's why you have different service types). Starting with simple analogies is very helpful in understanding complex structures in general.

I mean a basic k3s install + a barebones deployment combined with a nodeport service is about as trivial as you can get. its not THAT much more "lines of config (60-70 max)" then an nginx conf + "git clone + other stuff" deploy script.

I dunno, a sane nginx config is like 30 lines max for a basic service. You can mange it entirely with git (`git config receive.denyCurrentBranch updateInstead`) and some post-receive / push-to-checkout hooks.

Keep in mind, for K8s to work you also need to containerize your application, even if it's just static files (or I guess you can break containment and do volume mounts, or some sort of static file resource if you're serving static content). For non-static applications, you still need to configure and run it, which is just harder with k8s/docker in the mix.

Yeah but what's the point of running k8s if your app isn't even containerized, why not use a simple VM for that, it's cheap and easy to manage.

On the other hand if you are running thousands of containers it's hard to beat k8s, simpler solutions like ECS fall short at some point.

Agree. Maybe it should be more like "forget learning Kubernetes, Redis or anything you don't already" know since you don't want to bifurcate time between learning curve and product.

I also like Django a lot. I can get a working project up and running trivially fast.

In my day job I work with Go and while it's fine, I end up writing 10x more code for simple API endpoints and as soon as you add query parameters for filtering, pagination, etc. etc. it gets even longer. Adding a permissions model on top does similar. Of course there's a big performance difference but largely the DB queries dominate performance, even in Python, at least for most of the things I do.

Yes I really wish for something like Django for a statically typed language (maybe Spring? Haven't tried it).

I'm writing a CRUD CLI ( https://github.com/bbkane/enventory/ ) partly to practice doing it in Go, and while I'm mostly happy with the resulting code, theres just a whole lot of it. At least it's simple enough that I can trust LLMs to ok jobs with detailed prompts (example: https://github.com/bbkane/enventory/blob/master/.github/prom... )

You mention Django, but these days, are you using the full Django experience or are you mostly writing REST APIs?

In Java land a very nice and much lighter weight framework is Dropwizard: https://github.com/dropwizard/dropwizard

It's basically a sort of All-Stars collection of Java libraries, nicely packaged and with some nice conventions.

Towards the more servless route route there's Micronaut: https://micronaut.io/

I haven't tried Dropwizard. Is it a batteries included as Spring Boot? Eg: How is authentication support? Do we need a lot of boilerplate to implement let's say an OAuth2 resource server?

I used it a long time ago. As far as I know, it's an abandoned project now, very few people use it in the Java community. Go for one of the more popular ones: Spring, Micronaut, Helidon or Quarkus.

There are new frameworks all the time, but only very few establish a community that allows the framework to survive on the longer term. When you choose something that has been around for a long time, the probability for this is much higher, than for something that looks promising. Another benefit of the established frameworks is the amount of experience from various corner cases that has been coded in them. New frameworks often appear to handle simple use cases more conveniently, but that is only because they ignore all the complexity that the established solution has painfully learned throughout the years. You get that "for free" when you choose such a framework.

Hm I can guarantee that the frameworks I listed above will be well maintained for a long time and are much more “battle tested” than Dropwizard itself (though some of the libs that it includes are actually independent and used in other frameworks, those are sure to survive long time). You make it sound like Dropwizard is the more established framework when in reality it’s rather the opposite.

Yup, exactly, and Dropwizard is basically an opinionated bundle of Java "All-Stars" libraries. It's not like the "glue" needs a ton of updates and everything else is already well maintained at the source.

https://loco.rs, it's like Ruby on Rails, but for Rust (as their website says).

Personally I find Django and Rails too magical for my taste so I just use simpler libraries like Axum, but it's nice if you want all the bells and whistles. Plus, it's much faster than either Python or Ruby.

Springboot is pretty good, if java/kotlin are your lang of preference.

Still not the same bells and whistles as Django, but really good otherwise. The database interactions are pretty good.

hello,

as always: imho. (!)

hmmm ... i think spring-boot combined with either java or kotlin is a very good alternative to django.

even so i wouldn't compare them directly, but static typing avoids a lot of problems.

idk ... for me personally one of djangos great features is its custom db-model and relative (!) painless db schema-migration.

for spring-boot i often went with the tool flyway for handling db-migration ...

just my 0.02€

oh that's interesting. Is that due to missing libraries in Go? That could be a nice open source project if so.

There's an almost pathological resistance to using anything that might be described as a 'framework' in the Go community in the name of 'simplicity'.

I find such a blanket opinion to be unhelpful, what's fine for writing microservices is less good for bootstrapping a whole SaaS app and I think that people get in a bit too much of an ideological tizz about it all.

I don't think anyone would advise to do everything from scratch all the time.

It's mostly about libraries vs opinionated frameworks.

No one in their right mind would say: just use the standard library but I've seen it online. That discourse is not helping.

I think people get this miscontrued on both sides.

A set of reusable, composable libraries would be the right balance in Go. So not really a "framework" either.

I think that reflects better the actual preferred stance.

It's always going to be more work with composable libraries since they don't 'flow'.

Just picking one of the examples I gave, pagination - that requires (a) query param handling (b) passing the info down into your database query (c) returning the pagination info in the response. In Django (DRF), that's all built in, you can even set the default pagination for every endpoint with a single line in your settings.py and write no more code.

In Go your equivalent would be wrangling something (either manually or using something like ShouldBindQuery in Gin) to decode the specific pagination query params and then wrangling that into your database calling code, and then wrangling the results + the pagination results info back.

Composable components therefore always leave you with more boilerplate

I guess there are pros and cons. Pros of composable libraries is that you can more easily build upon them for your specific use case.

It also doesn't tie you to ORM usage.

You have to be responsible for "your" specific flow... meaning you can build your own defaults easily wrt parsing query parameters and whatnot (building a generic paginated query builder API?). Nothing insurmountable.

That's fine as long as you never need to deviate from the default behavior of the framework.

Fortunately the framework is pretty configurable on all this sort of stuff! Just to hit pagination again since it's the example I've used in other comments, in DRF you implement a custom pagination class:

https://www.django-rest-framework.org/api-guide/pagination/#...

That's the same as for most other configurable things. And ultimately if you want to override the behaviour for some specific endpoint then you can still easily do that by just implementing your own method for it.

After working in Go now for several years, what I've found is that generally people just don't some things in their first pass because it's too much work and makes the PRs too large, and then they retrofit it afterwards. Which meant that the APIs were less consistent than the ones I worked on in Django.

> No one in their right mind would say: just use the standard library but I've seen it online.

Yes but...sometimes the proper level of abstraction is simply a good HTTP library. Where the API of your application is defined in terms of URLs, HTTP methods, status codes, etc., the http library from the Go standard library might be all you need (depending on other requirements, of course).

A real life example: I needed a simple proxy with well defined behavior. Writing it in Go as a web application that forwarded calls to another service based on simple logic only took a few pages of code in a single file.

I don't dispute that. But in general you need sessions, you need a few middleware to, for example, handle csrf or gzip compression of the response, auth, etc. Telling people to juste use the standard library doesn't help. Of course it is available. But often there is a need for more.

Look, I'm not for a moment suggesting that a proxy application would be a good fit for Django (or even Python). I'm talking about 'building a full stack web application in a reasonable amount of time, with auth, with a database, etc. etc.'

> No one in their right mind would say: just use the standard library but I've seen it online. That discourse is not helping.

I would say that.

The most important thing about standard library is its stability. You won't ever need to touch code that works with standard library. It's finished code. Other than bug fixes, of course.

Third-party libraries is a very different thing.

They gets abandoned all the time, so now you're left with burden. You either need to migrate to another library, maintain that abandoned library or live with huge chunk of code that might be vulnerable.

They gets changed often enough, as their developers probably not so careful about backwards compatibility, compared to core language developers.

Third-party library is a liability. Very rarely its source code is an ideal fit to your application. Often you'll use 10% of the library, and the rest is dead weight at best, vulnerability source at worst. Remember log4shell? Instead of using standard logging code in Java, some developer decides to pull log4j library which is very nice to use, has lots of features. It can even download and execute code behind your back, very featureful.

Of course I'm not advocating to rewrite the world. This is insane. Some problems are just too big to solve by yourself. I also should note, that different ecosystems have different approaches to the language library and overall library culture. JS is terrible, while Go is not that bad, but it's not ideal either.

But my absolutely 100% always approach is to use standard library first and foremost. I won't use third-party library just to save few lines of code or make code more "beautiful". I prefer dependency-free boring repetitive code any day.

And if I'm using third-party library, I'm very picky about its stability and transitive dependencies.

It also depends on kind of company. My experience has always been: you write some service, you throw it at production, it works for the next 20 years. So you want this code to be as self-contained as possible, to reduce "chore" time with dependency management. Perfect application is a dependency-free software which can be upgraded by changing "FROM" line in Dockerfile. It is stable enough that you can trust CI do that.

I don't think that everyone is capable of or should be implementing csrf protection or cors handling. While the standard library is an awesome starting point, telling people that it is sufficient is not going to convince them.

I wrote CORS handling in one project, it's like 10-20 lines of code. Very simple and well documented feature.

Good for you but most people don't even know what a preflight request is so I would maintain my position that it is not obvious.

I mean, I'm keen on a small number of dependencies too, but the smaller the scope of a package you go, I find the more likely they are to be abandoned. The Go JWT library has been forked twice because it became abandoned by the original authors, just to give an example.

In my experience frameworks (in contrast to "libraries") add complexity making your application harder to understand and debug once you get past the initial prototype.

Also, by their very nature they almost require you to write a program of a minimum size even when a simpler program would do to solve your problem.

This has been my experience as well with almost all frameworks.

It goes like this: the framework helps with the initial progress, because framework works with the more typical use cases. This allows you to "prototype" the solution and show progress to potential users etc.

But then you have a specific use case that does not fit the model of the framework, and you will end up going around the limitions of the framework. You often end up spending more time that you saved with the initial implementation.

My experience with core Django is that it has so many hooks and ways to inherit functionality, that it is extremely rare to end up with such problems. But it still means you need to learn when to use what feature, and how to structure your solution. I had to learn this through painful experience. I am still making mistakes, such as the "single-file status page app", a mistake which cost me several hours of productive coding. That might not sound like much, but when working full-time and running several side projects, few productive hours is extremely costly.

Most third party Django apps, even the popular ones, suffer from the framework problem, however. For example, I have have had to rewrite django-ads and django-distill, because their structure was limiting me from implementing features I needed. I just ditched django-formset as well.

I believe the reason for the limitation is that these third party apps have not been exposed to enough use cases and users to overcome the framework problem.

> Also, by their very nature they almost require you to write a program of a minimum size even when a simpler program would do to solve your problem.

I'm not sure I agree, sure they require you to pull in a large dependency, and that might not be good for some use cases (particularly microservices) where you're sensitive to the final size of the image but for e.g. with Django you get spawned a project with settings.py which configures your database, middleware, etc. and urls.py which configures your routes, and then you're really free to structure your models and endpoints however you like.

Whats so hard about writing an SQL query and some json struct tags? I really dont like LLMs but all the pain points of Go you described above are relatively mute now if you use them for these little things

There's nothing hard about it, it's just that the sum total of code you need to write is much larger, and that takes more time.

edit, my karma for this comment has been dealing with type assertion problems in my repo functions all day.

Haha :D

> Always choose extremely boring technology. Just use python/Django/Postgres for everything.

Hell, think twice before you consider postgres. Sqlite scales further than most people would expect it to, especially for local development / spinning up isolated CI instances. And for small apps it tends to be good enough for production too.

Sqlite is mostly boring but I've found that there's just slightly more risk of something going wrong because of the way it handles locking between threads. It has tended to misbehave under unexpected load and been difficult to fix in a way that Postgres hasn't.

I'm particularly thinking of workers running tasks, here. It's possible to lock up everything with write transactions or cause a spate of unhandled SQLITE_BUSY errors in cases where Postgres would just keep chugging along.

> especially for local development / spinning up isolated CI instances

I really don't consider it a good idea to use different databases on your production vs development vs CI instances. Just use PostgreSQL everywhere, it doesn't cost anything, scales down to almost nothing, and scales up to whatever you're likely to need in the future.

I have had some bad experiences with Sqlite for local desktop apps with regards to memory usage, especially on MacOs. Insert and delete a few thousands rows per hour, and over a few days your memory usage has ballooned. It seems to cause a lot of fragmentation.

Curious to hear more about your experience, since my impression from hacking around in native Apple software is that pretty much a bunch of it is based on top of sqlite3. The Photos app is a case in point I remember off the cuff.

I think we might be using it in a slightly unusual way: collect data for a while, do one big query once in a while to aggregate/transform the data and clean everything. Rinse and repeat as it's a background app.

So lots of allocations/deallocations. If you're only storing a few key/value pairs long term, you won't have any issues.

What kind of memory usage are you talking about? That sounds wild.

Are you keeping the entire database in memory, or are you flushing to disk?

It seems wild that such a critical and pervasive piece of software would behave like that.

Keeping it in memory. After a few days, the memory usage reported by the Activity Monitor (so not the actual resident memory, but the one customers see and complain about) grows from maybe a few 10s MB to a few hundreds MBs.

But as far as I can tell, it's more an OS issue than really a Sqlite issue, simply doing malloc/free in a loop results in a similar behaviour. And Sqlite does a lot of allocations. We see a similar problem on Windows, but not as pronounced, and there we can force the OS to reclaim memory.

It's probably solvable by using a custom allocator, but at this point it's no longer plug and play the way the GP meant.

Sqlite is not a very good choice for a typical CRUD app running on a web server. MySQL/Postgres/MariaDB will be much better. You can connect to it from remote and use GUI tools etc. Also much more flexible from an architectural point of view.

Sqlite seems to be the hip new thing to use where MySQL should have been used in the first place. Sqlite is great for many things, but not for the classic CRUD web app.

I have worked on a database engine core team, and I really like the simplicity of SQLite and would love to use SQLite more. I am fully aware of benefits of the simplicity it brings to operations and I also know that the scalability of SQLite is more than sufficient for most of my use cases.

But I am running two major side projects I while still working full time. One of my side projects is a site with 2M page views per month. Additionally, depending on how you count (whether apps are included in the major projects or not), I have 3-4 smaller side projects, two of which use SQLite database.

To be able to run all of this, I have an extreme cut-throat approach to technology. SQLite is almost there, but ultimately Postgres is still more mature all-around solution. As an example, I believe (have not tested this though) it is quicker to set up full-text search for Postgres using online resources. The developer experience is better, because Postgres full-text search has been around for longer and has wider adoption.

Local Postgres is easy. If you're deploying to some cloud service, Postgres is still probably easier than SQLite, since persistent local storage is a bit more special there. I'd also be worried that SQLite doesn't provide something I need out of the box.

This isn't to complain about SQLite overall, it's perfect for local DBs like you'd see in app frontends or OS services.

It's easier to spin up a RDS instance or equivalent than to set up EBS / EFS just to host a tiny sqlite file in a pvc for me personally

hello,

as always: imho. (!)

while i personally really love SQLite for a lot of use-cases, i wouldn't recommend / use it "in serious production" for a django-application which does more than a simple "hello world".

why!? concurrency ... especially if your application attracts user or you just want to scale your deployment horizontally etc. ;))

so in my opinion:

* why not use sqlite for development and functionality testing

* postgresql or mariadb/mysql for (serious) production-instances :)

just my 0.02€

Yeah, I've also found that foregoing Postgres is one step too far. It's just too useful, especially with Listen/Notify making it a good task queue broker. SQLite is great, but Postgres is definitely worth the extra dependency.

You distilled the point well. SQLite is great, but Postgres still hits the sweet spot.

while i'm currently using sqlite, since it has only one write transaction at a time, if I understand correctly, opening a tx and doing outside requests while in the transaction could potentially block your whole application if the outside requests keep timing out... so you kinda need to watch out for that

Treat transactions like mutexes has always been the prevailing wisdom has not not? Keep them as short as possible and do not make blocking calls within one.

This would be true for any database, something read / written during a transaction would block at least that table universally until the transaction is finalised.

What do you save by going with sqlite vs how much pain are you going to endure in the future?

I save having to deal with a full-blown DBMS daemon accessed through a network boundary unless I need this, and I have approximately zero pain, because I read the documentation and set up my database schemas to work on both solutions (and use WALs so sqlite is rock stable). I also know how to open a file on the command line if I need raw database access for some reason (very rare with django anyway, django's console is just too good not to use it). I also design my apps to not deadlock on unnecessarily long transaction and don't turn every request into a database write, so I can scale out pretty far before I have to worry about write performance. And if I do, I can still use postgres. Until then, I can do unified, consistent backups of all state by snapshotting the filesystem containing uploads and sqlite files.

So I dunno why people insist on spreading so much FUD.

It's not FUD. For all the trouble you claim to have with Postgres I experienced 0 of it in the last 4 years. The only thing extra for a simple setup is a couple of lines in your docker compose files which is completely amortised because you already have a multi process architecture with Python anyway (proxy + webserver + web server works). The upfront cost is so small that for me the expected total cost will rarely make sense even if you assume that your application has 1% of chance of scaling beyond what you can do with sqlite.

> Most applications don't need to be single-page apps nor require heavy frontend frameworks. Even for those that can benefit from it, traditional Django views is just fine for 80% of the pages. For the rest, consider AlpineHJS/HTMX

Doesn't that contradict "learn one tool well"?

I write every webpage in React, not because I think everything needs to be an SPA, but because enough things end up needing client-side state that I need something that can manage it well, and at that point it's easier to just do everything in React even if it initially seems like it would be too heavy, just like your example.

Yes, I contradicted myself in that sense. I am not going to argument against your point.

But let me elaborate my approach which explains this apparent contradiction. Backend is needed in any case, and for me that is almost always Django. It is extremely fast to build traditional (non-SPA) CRUD apps with Django if you have only one model / one page. I can make a new one less than an hour.

If I need more interactivity or more data on the page, I use the same approach (models, views, forms with validation) but I create partial forms and return HTML fragments, instead of full pages. AlpineJS/HTMX allows me to implement 80% with Django/python, and I can avoid implementing a REST API and JSON serialization/deserialization, all the frontend data management, and there is usually almost no need for Javascript.

> Backend is needed in any case

Well, sort of. Much as I hate the idea, maybe Next.js for everything is the way to go, bashing out a CRUD app there is very quick.

> create partial forms and return HTML fragments, instead of full pages

I loved that approach with Wicket and made systems that way for many years, but it just isn't good enough these days, at least for a website that's going to be used in more than one location. When it takes a network roundtrip to update your UI it's always going to be noticeable.

I assume you are talking about geographical distribution of the user base. I don't have experience implementing apps that require heavy geographical distribution, but why does REST/JSON work better for this?

I don't have experience with Wicket, but with HTMX, I have tried to make components very small. For example one switch button is one component. Loading that is very fast in my experience, as compared to reloading and rerendering all data with React etc.

> I assume you are talking about geographical distribution of the user base. I don't have experience implementing apps that require heavy geographical distribution, but why does REST/JSON work better for this?

Once you see the smoothness of fully local UI transitions compared to doing a WAN roundtrip, you can't go back, even if it's objectively not a huge difference. Things like switching between tabs, or hierarchical dropdowns where the choices in the second dropdown depend on what was chosen in the first dropdown, doing a network roundtrip to load an HTML fragment is maybe OK over a LAN but it's a nonstarter over the internet.

So your state management needs to be at least partially local to the frontend. And in my experience the only approaches available for that are a) React or b) ad-hoc hacks; some frameworks build the ad-hoc hacks into their builtin components so you get smooth local UI transitions as long as you're sticking to the happy path, but that ends up not being enough.

To me it sounds like it's time for you to get to know Elixir's (Phoenix's) LiveView. It does this even better and you barely write any JS at all.

That is on my list of things to learn. It is not high priority, though, because I don't think Phoenix LiveView is extremely boring technology at the moment. The user base is not large enough.

A lot of stuff is quite commoditized already, and I never found the size of a user base a convincing metric to judge by (not to mention the chicken-and-egg problem that you just demonstrated).

People are doing genuinely hugely impressive production-grade work there, and I am currently looking at a SigNoz dashboard while >200 users are frantically clicking on dozens of places in our UI (contains many reactive bits); our code is nowhere near optimal and we very rarely get >150ms response times, most are 20-50ms, and we're talking some quite tame machines (2 of them) with 8GB RAM and 2 CPU cores... and even then 90% - 99% of the time is spent waiting on the database.

The context of the discussion is how to create apps extremely fast. If I would optimize for something else, it would be a different story.

Oh well, if you find that way then many people -- myself included -- would be very interested in your blog post. Bootstrapping remains a tricky territory, not because it can't be done very quickly but because it leaves a lot of tech debt behind that has to be paid off, inevitably, at one point.

Fully agree. I would also say it's easy enough to use Django for (almost) everything for a self contained SaaS startup. Marketing can be done via Wagtail. Support is managed by a reusable app that is a simple static element on every page (similar to Intercom) that redirects to a standard Django page, collects some info about the issue including the user who made it (if authenticated) etc.

I try to simplify the stack further and use SQLite with Borg for backups. Caching leverages Diskcache.

Deployment is slightly more complicated. I use containers and podman with systemd but could easily be a git pull & gunicorn restart.

My frontend practices have gone through some cycles. I found Alpine & HTMX too restrictive to my liking and instead prefer to use Typescript with django-vite integration. Yes it means using some of the frontend tooling but it means I can use TailwindCSS, React, Typescript etc if I want.

Agree with almost everything, but Celery is pretty common in my Django projects. I don't like the complexity cost, but especially when using some PaaS for hosting, it's usually the least painful option. I kinda always start out thinking this time I'll manage without, and then I have a bunch of jobs triggered via HTTP calls running into timeouts. At that point it's either threads, cron jobs (tricky with PaaS) or Celery. What's your approach?

This.

I use celery with the same code base/docker image. Just a different entry point to start a celery worker instead of a wsgi (web) worker.

Too many http requests? Add web worker instances.

Background jobs piling up? Add celery workers.

Clearly separate read endpoints from write/transactional endpoints and you can hit a slave postgres db or the master db depending on the http call.

This creates a very robust system that can scale easily from a single code base.

I do the same, it's easy enough and doesn't require a ton of hosting logic.

Out of interest, how do you run your migrations in production, deploy the service then run an ad-hoc job with the same container again? That was one thing I was never super happy with.

In an ideal world your code-base is always compatible with the previous migration state.

So new version can work with the previous version's DB schema.

Then, yes, simply run the migration in a transaction once the new code is deployed. Postgres has fully transactional DDL changes which helps.

Of course, it heavily depends on the actual change being made. Some changes will require downtime or must be avoided if it is too heavy on the db.

Another approach if the migration can be applied quickly is to just run the migrations as part of the deployment script. This will cause a downtime but can be short.

Easiest is just to do runmigrations in your docker image start commands, so DB is always migrated when the container starts.

tl;dr: It depends on the complexity of the migrations and your uptime requirements.

I was using Redis and Celery but was not happy with the complexity.

But then I found out that background workers are being implemented to Django.

https://github.com/django/deps/blob/main/accepted/0014-backg...

This functionality has been backported as django-tasks library. You can use the database (i.e. Postgres) as the permanent storage.

For periodic jobs and running the django-tasks, I use Superchronic, which is a Docker-compatible cron. It is compatible with cron, with the added benefit that you can run jobs with sub-minute granularity. I have had no problems with running Superchronic inside my fly.io app.

I run Superchronic and Django with honcho and Procfiles.

What Paas do you use? Pythonanywhere has an always on task feature or scheduled tasks where you can run command scripts.

Heroku, AWS Elastic Beanstalk (not a fan), Fly.io. With these, it always boils down to using their Celery/RabbitMQ offering. I also used Ofelia in the past, a simple Docker image for cron jobs, but quickly outgrew it.

Maybe I'm missing it, but I don't think any of these has a nice simple built-in feature.

You can have a queue in your relational DB (example: https://github.com/pgmq/pgmq). It'll scale pretty far.

I feel like Django has the largest RoI of any framework out there

Completely OT and apologies if rude, but you're Gary Numan the musician?

I love finding out when celebrities have talents elsewhere. And Wikipedia says you've had quite a bit of aviation experience as well.

Kinda makes my morning... lol

p.s. The inner city cynical kid in me is now required to throw in that I found Django disappointing even though every else seems to love it. Ok... identity restored... can now go back to shaking my fist at the sky as per usual...

I think Rails is stiff competition, it's just I prefer Python.

Laravel/PHP world is also not bad. It gets bad rep but for CRUD apps the PHP model of every request being isolated is pretty great.

I last touched PHP when you sprinkled it in between HTML (and deployment was FTP-ing it to a server) so I'm well out of date with that though I've heard people say that it's quite nice nowadays.

Your first point resonates. I had an idea I wanted to explore and decided to use Make.com and google sheets. After two hours I said, screw this, and spun up my entire idea in a Rails app in 30 minutes.

Knowing a good Swiss army tool very well is a super power.

> Forget Celery

How do you do background job processing without Celery? Every app/website I ever developed required some sort of background job processing, for Python I use Celery, Rails I use Sidekiq, Node.js I use Faktory. One of the biggest drawbacks (at least until a while ago) was that setting this up had to be a hacky webhook call to your own app that allowed up to N seconds request/response.

If you can tolerate 1 sec latency, Django-background-tasks is pretty good.

DB-based queues are pretty common (see outbox pattern). You actually don't have much choice but an outbox in your relational DB anyway, if you want the job to be enqueued transactionally.

You write the demand to the database, and have a worker process execute it. Exactly like you'd do with celery.

Celery is a solution to scalability, not to any business problem in particular.

Years ago I ditched Celery in favour of using BLPOP with redis. It was multiple times faster than Celery. Is this still the case today? So with vanilla Redis you need less workers for same throughput; if you need more throughput, spawn multiple workers.

I've had success with a boring event loop which looks at the state of the system and does all the background work. It's simple to implement and much easier to manage failing tasks that need manual intervention to stop failing. It also makes it easy to implement batching of work as you always know everything that needs to be done in each loop.

I also have multiple celery applications, but I wouldn't recommend it for smaller or simplier apps.

> boring event loop ... state of the system ... background work

Can you explain what you mean?

Django background workers, backported as django-tasks library, the tasks being stored in my database (Postgres). Everything is started by honcho, which starts Superchronic and Django, and Superchronic runs django-tasks. All this (except the database) is running inside single fly.io app.

I think HTMX as "simple" solutions comes into the same class as you mention "django being too heavy".

Sure, Django,Rails,Asp MVC,etc with static pages could be fully functional for something that is 99% CRUD.

But I myself foolishly tried using static view+HTMX for something that should have been built with something more dynamic like Vue/React(perhaps Alpine?) from the start due to actual client-side complexity.

In general imho one should classify what your main interaction points are and use the correct tools for each.

- Is it static pages for masses of consumers with little to no interaction? (Ie a news-site,etc) then fast SEO-friendly pages is your nr 1 priority.

- Is it mainly massive CRUD updates that needs persistence, go with server built pages with Django, Rails, etc.

- Is it an actual client/data heavy UI (games but also scenarios where you want fast client data slicing or processes/sources that complete at different paces), use an SPA.

Ie, right tool for the right job.

Who could argue with the "right tool for the job"? I am not saying that there are no use cases for other approaches.

But when you say you had a "static view" (in singular, not plural) with HTMX, what do you mean? I am very interested in hearing what kind of complex interaction you had where that approach failed?

With HTMX, I have found the most useful approach is to build your pages from lots of very small components. Initially, I don't worry about reuse of those components, it is more about decomposition than reuse.

Of course, if you want to do "fast client data slicing", then HTMX makes no sense, because it pushes logic to the backend and by definition your solution wants to process data in the fronted. So if you start from that premise, HTMX is obviously the wrong tool.

If you want to use HTMX, you need to rethink your solution, i.e. question whether you really need "fast client data slicing". So why bring all the data to the frontend, and not just the slice you want to visualize?

One reasonable use case is reducing server load. I have one app where data is loaded from static generated pages (periodically updated, for performance, stability and reduced server load), and sorting is done on the frontend using Javascript, based on data attributes on the DOM.

Yes, my default strategies are quite similar. I would also add: pre-populate data where you can rather than relying on robust data entry by users.

> Learn one tool well.

That works well until that tool stops being maintained, or is evolving in such an odd direction that it breaks your code.

Django is really an exception to something one can rely on, but reality is more like Angular -> breaking, Vue -> breaking, React -> breaking.

In Django, how do you create components well and handle user interactions?

It's been a long time (we've gone all React), but I remember doing reusable components with templatetags.

I think that's where htmx comes in

I use django-components.

re:componenets Just use jinja templates.

This is great, but it's for a very narrow set of programming problems.

You clearly work on web applications of moderate scale. For example, Kubernetes and Redis can suddenly become almost necessities for a back end service once it reaches a certain scale.

In my experience it works for a large variety of use cases.

As one of my side projects, I am running a website with 2M page views per month that has been implemented on Django, deployed to fly.io. Another side-project built-in tenant support, and several independent deployments as well.

Yes, those are moderate scale (for modern hardware), but in my experience very few sites really require "webscale". It is far more common to build prematurely for scalability than the opposite. If you have "webscale" problems, then you also have have a custom platform and team working on those.

The context of this discussion is how to build robust systems as fast as possible. My approach fits surprisingly large number of use cases.

Reply on news.ycombinator.com