Show HN: Lightbus – Python/Redis Message Bus for Eventing and RPCs
lightbus.org
lightbus.org
I’ve been using this in production now for about a year and it has faired very well. I found a few weird bugs along the way, but it has felt stable for quite a while now. I’m therefore finally making an official release. I wanted to call this version 1.0, but I’m going to play it safe and call it 0.9 for a few months and see if anything crops up in other peoples’ environments.
Before embarking on this I spent a long time talking to other developers to see if anything similar existed. The most similar I could see was Nameko, but I didn’t feel like that ticked my boxes. Lahja is another one which I saw pop up more recently. I’ve also written a short comparison with Celery, as it something I’m often asked about [2].
I’ve also spent a lot of time writing up the docs. A crude word count indicates I’ve written 23,400 words of narrative documentation :o
I was also keen to not strictly limit this too Python. I’ve therefore documented the Redis interactions required for other languages to interface with Lightbus[3], albeit without the various niceties that Lightbus provides).
Constructive feedback is very welcome!
[1]: https://github.com/adamcharnock/lightbus/issues/1
Firstly, the maintainers are extremely responsive, and have gotten most issues fixed in less than a day. IMHO, the devs are being particularly conservative in calling this 0.9 instead of 1.0 -- which is arguably a good thing! Here's a quick/informal experience report. Happy to go into details.
Pros:
- It's very "Django"-y. The declarative interface reads and writes like Django ORM code, so there's very little context switch. It should feel familiar to Django programmers.
- It's built on "boring" tech. The default transport layer is Redis. Most of us don't need a distributed queuing system like NSQ or Kafka. Config is YAML.
- The API is designed to favor ease-of-use over virtually everything else, and yet there's actually very little magic under the hood. There's a bit, but we've found it pretty easy to reason about. Default options are sensible, and easily tunable.
- The documentation is good. It's a bit spartan for some of the more complex things (e.g. writing your own client in $YOUR_FAVORITE_LANGUAGE).
Cons:
- Our biggest concern has been with logging. Message-bus architectures have all the difficulties of cooperative multitasking, with the added problem that events can also go missing. Good logs are invaluable in debugging this kind of problem. My understanding is that this is on the roadmap.
All in all, I think it's a safe bet for pure Python projects. If you're a polyglot shop, you may want to consider something else in the short term, but Lightbus is very capable of solving the problems that message-bus architectures were designed to solve [0].
[0] e.g.: "how can these isolated web apps share data in a sensible way".
> The documentation is good. It's a bit spartan > for some of the more complex things
Yes, I've documented the protocols[1] but I haven't gone into much more detail. The docs may also need some iterating on as/when people try to develop new clients.
> Our biggest concern has been with logging. > Message-bus architectures have all the > difficulties of cooperative multitasking, > with the added problem that events can also > go missing. Good logs are invaluable in > debugging this kind of problem. My understanding > is that this is on the roadmap
Yep. Two things are on the roadmap that are relevant to this:
1. Revamping the logging system, likely to have an option to support structlog 2. The ability to see a causal trace for each message (Message C was caused as a result of Message B, which was caused by Message A)
I think no. 2 should be straightforward, and I think no. 1 is just going to take a grim sense of determination and deleting a bunch of old logging code.