Hacker News Reader

Comment by AussieWog93 | Hacker News Reader

I've done a bunch of Flask work over the past couple of years - not full time but as part of the tech stack for my small eCommerce business. Have run into all kinds of footguns and weirdness with MongoEngine, SQLAlchemy, Celery (seriously, if you value your sanity, don't use Celery!), the Python stacks for Google, eBay and Shopify but never Redis.

Perhaps that's because I'm not giving admin access to random people who think that Redis is a persistent storage, but honestly it's one of those technologies I'd describe as absolutely rock solid and well designed. The API is dead basic and every time I need to do something slightly weird, there's a sensible and well thought out way to achieve it.

View on HN

I am currently in the process of starting a project with Flask, SQLAlchemy, Celery. Say more about why I should avoid Celery and what to use instead.

Things like chaining, groups, named queues just don't work the way you'd think they would. There's a lot of footguns and things require weird workarounds. Error reporting is misleading.

It's not bad enough where I had to pull it from the old project that used it, but going forward the new ones used a vibecoded queueing system that was genuinely more reliable than Celery but consumed a lot of memory (RSS inflation). Have then shifted to rq and at least for now it seems to "just work". You're better off doing anything custom/complex (like dependencies, or progress updates across multiple tasks) directly yourself in Redis anyway; since half the time Celery's less-well-trodden inbuilt features don't work the way they should anyway.

Huh that's interesting! I found celery to mostly match my expectations. I used it in a couple of django apps. My only real foot gun was around having to set an EAGER setting for local development or tasks never got executed.

How did you find your expectations and celery's actual semantics to be different? I'm trying to document well and it seems like I might have some implicit assumptions that I could make explicit, but I don't know what they are since they're already in my head and matching celery it seems.

What are you using Celery for? Do you need to be able to recover from a reboot or crash with your queues intact? Is it a distributed system as opposed to a single machine? Do you have complex multi-step workflows?

If you answered no to each question, just `import Queue from queue`.

> What are you using Celery for?

Things like provisioning, deploying, and eventually destroying cloud instances (VMs) on-demand when a user buys a specific service.

> Do you need to be able to recover from a reboot or crash with your queues intact?

Yes, I expect the queue to be durable.

> Is it a distributed system as opposed to a single machine?

Currently, everything runs on a single machine. But I expect it will eventually have to be split up. Although I do not expect it to be massively distributed or very complex.

> Do you have complex multi-step workflows?

Depends on what you mean by complex but Multi-step, yes.

> every time I need to do something slightly weird, there's a sensible and well thought out way to achieve it.

In my world cache systems like memcached and redis are just that, a cache to put and get from. Possibly use some invalidation system like tagging.

What can you do with a cache system that is 'wierd'? What are people doing with caches other than just caching data?

Genuinely interested.

I’ve done moving window rate limiting using redis to do atomic rate calculations etc.

That requires some weirdness

> moving window rate limiting

So does that mean you are tracking how many times data is being entered into redis, and rejecting it if the entry rate is too high?

Why would you not track this before, at the point of calculating the data to enter into redis, rather than querying redis to see how much data is entered in a given timeframe?

Again, genuinely curious as to the reason for architectural decisions.

Not GP, but I think they mean usecases like limiting how many times any given IP address can access an API to a certain amount of calls per minute. For example, you might want to restrict login attempts to at most 10 per minute per IP to prevent people trying out lists of common passwords.

This is fairly easy to do if your apps runs on a single server, but many companies run multiple servers and load balance requests among them. Those servers need some sort of coordination mechanism to keep track of the rate limits and their current state. Redis has dedicated instructions these days to do this, and in the old days there were plethora of libraries that use embedded Lua scripts to do the same thing.

Spot on! It was for a decently sized SaaS app; 10k+ request per minute & and a LOT of spam traffic from china a.o. we needed to limit. The app ran across 10+ servers, this is also why we put it in the app using redis and not with something like nginx rate limiting.

I don't exactly remember how i implemented it, but it basically did a single call to redis to count the request for the IP and check the limits.

Another usecase where the more advanced data types & operations of redis are usefull, is for job queues, since you can atomically move a job from the 'queue' to the 'processing' list, thus preventing loosing jobs if the processor crashes after pulling it orso. But we do run all those on persisted redis stores, for safety :)

And if i would do it all again, i'd probably just use postgres for anything i want to keep when things crash. Redis just kinda lives between a 'real' database and a pure volatile kv-cache

We had Rails writing to memcached, and nginx pulling from memcached for full page caching.

At some point someone decided to gzip all writes into memcached, and our site looked really fun for a while.

No, you're right. Nothing crazy. But things like counting API usage across threads with INCRBY, or debounced HTML cache clears, or even an actual light db with persistence (AOF), and everything just working.

Non-caching things I regularly see people do with Redis:

- Rate limits for API endpoints via the leaky bucket algorithm

- Feature flags and stats tracking

- Websocket pub/sub

- Background job queue

In general, lots of things that need to survive deploys (so they can't be in-memory in the app) and/or they need to be coordinated across multiple horizontally scaled servers and/or things that prefer to be in a data structure which is slightly awkward to stick in a database table.

Can do fantastically weird stuff with Lua scripts in Redis/Valkey

Reply on news.ycombinator.com