Show HN: µTT – A new faster MQTT broker | Hacker News Reader

Show HN: µTT – A new faster MQTT broker

github.com

115 pointsalexhultman37 comments

Maybe this post should be "Show HN"?

I am happy for you, hope this grows!

However, may I suggest you add to description on GitHub page some more info? Performance is only a part of equation and description only talks about that. Did you give any thought to reliability and robustness? How about security? It is written in C++ - was it fuzzed? Does it support TLS? Authentication?

These are much more important questions (to me) at this time (disclosure: using Mosquitto and happy with it, didn't hit bottlenecks yet). I have no doubt you will address them sooner or later, but it might be good to start working on some of them as soon as possible - or, if you already have this covered, to write about it.

Of course, I might not be your target customer, at least at this stage, in which case please disregard this advice. :)

I was very excited to see this post since I happen to be looking for a scalable high performance MQTT broker.

The first thing I noticed when I opened the GitHub page was the lack of documentation. Then I opened the source files. No headers describing what each module does. No method headers. No code comments. Nothing that would let me get a feel of how well this could fit into my project without doing a lot of code reading...

Maybe I'm just an old fart and need to accept that this is the way code is written these days, but with my own projects I always tried to make sure that at least minimal documentation was available and that the code was easy to read and understand before releasing it to the world.

I'm sorry but HackerNews thinks they know better so they mod the title into something misleading.

Original title was "I'm developing a new and faster MQTT broker" with extra focus on "developing". It is a 7 day prototype, not a finished product. The naming is not even known to be final at this point.

Thanks

Ah, that explains a lot - I was not aware that HN was editing titles. Will be watching the project with interest.

If you need a demo use case, we provide an open MQTT feed of some 1500 public transport vehicles, soon once per second (plus per-stop departure prediction feeds): host mqtt.hsl.fi, port 443 and 1883, topic /hfp/#

Ah, kool. I first need to finish this up to a stable state though. Right now it just barely passes the benchmarks test.

Out of curiosity, which vehicles/city?

HSL provides the bus, tram, suburban rail, subway and ferry system in the Helsinki metropolitan area.

I've written a few toys that use their GraphQL API to show real-time displays of trams. Pretty neat system.

Thanks! Waiting for your MQTT integration ;-)

If you want to have a quick look at how the feed looks like, there is a web client for mqtt at http://www.hivemq.com/demos/websocket-client/. Don't leave it open for too long though, it might kill your browser (Chrome struggled a bit on my old laptop).

    {"VP":{"desi":"N","dir":"2","oper":"XXX","veh":"H9449","tst":"2017-05-25T11:31:50.000Z","tsi":1495711910,"spd":2.22,"hdg":10,"lat":60.249649,"long":25.008106,"dl":0,"oday":"2017-05-25","jrn":"XXX","line":"XXX","start":"1439","stop_index":7,"source":"sm5logger"}}

That's a pretty nice use case, especially with MQTT supporting persistent messages (which are messages that "stay around" and which the client will receive immediately upon subscription, no matter how long ago they were published). That way, you can publish the messages as rarely as they come, and a client that subscribes will still get the latest time available without having to wait for the next publish.

In practise we can't use persistent messages for this, but you can fetch the latest messages that correspond to your subscription from a REST API that caches the messages: https://api.digitransit.fi/realtime/vehicle-positions/v1/hfp...

(The reason is that we use MQTT topics creatively: instead of one topic per vehicle, the topic changes dynamically to provide more filtering criteria such as the route and the next stop of the vehicle. Thus, most of the per-topic persistent messages would be obsolete.)

That's awesome, wish my city would do something like this - you're lucky to get updates once per 2 or 3 minutes on their service.

lol. i just wish mine would put departure times on the website...

Are implementation details for this available? What's the stack?

We try to be as open as possible nowadays. However, these backend systems are from an old proprietary contract. There's a computer with a GPS receiver and driver UI in each bus and tram sending the MQTT messages to mqtt.hsl.fi (IBM MessageSight hardware for now) over 3G cellural data connection.

The subway can't use GPS but the trains read RFID tags installed along the tracks and there's wifi coverage. The suburban trains have Raspberry Pi's listening in promiscuous mode in the maintenance Ethernet ports and relaying the vehicle data to us.

The ferries send AIS radio data that we fetch from here: https://meri.digitraffic.fi/api/v1/metadata/documentation/sw...

The REST API is part of Digitransit, which is a fully open passenger information platform and journey planner. Docs and code here: https://digitransit.fi/en

Specifically, the switchbox for realtime data is a simple NodeJS application: https://github.com/HSLdevcom/navigator-server

Thanks for the information, hopefully you can move the MQTT to something open in the future :)

Digitransit is very interesting, and all of this is great work!

It's great to see a message broker that's optimised for performance - having looked at the source of a few of the more popular brokers it's clear that there's a lot of overhead in many of them (object allocation/boxing, multiple copies being made of the data, inefficient serialisation, verbose wire protocols and so on).

I'd be really interested to see how uTT performs compares to Aeron[0], which, while not MQTT, as far as I'm aware it is the fastest message transport around[1,2].

[0] https://github.com/real-logic/Aeron

[1] https://github.com/benalexau/rpc-bench

[2] http://akka.io/blog/2016/12/05/aeron-in-artery

Does it scale horizontally or it is bound to a single machine?

Please note my original title was "I'm developing a new and faster MQTT broker".

HackerNews changed it and I cannot edit. Point being (a) I do not even know the naming is final, (b) this is in no way a finished product, it is a 7 day prototype.

Have you also compared with NATS?

Agree that this is a more interesting comparison than AMQP.

Interesting thread, and it is always good to see performance pushed by OSS messaging. NATS is lightweight and performant as well. Both are TCP based - any thoughts around other transports underneath (LPWAN, BLE, etc)?

I'm really excited about this. It's by the creator of uWebSockets[0], alexhultman[1]. It looks like it builds heavily on his work there. Based on the performance compared to redis, this is already an impressive start. Excited to see where this goes! I also second an attempt at comparing it to ZeroMQ, even though they're obviously somewhat different.

[0] https://github.com/uNetworking/uWebSockets

[1] https://github.com/alexhultman

Are you considering support for MQTT-SN, or willing to take pull requests to support it? The ability to handle lightweight clients could be a very important differentiator.

This is very useful if it's faster than the competition. MQTT is one of those things that's so useful that you tend to overuse it, and extra speed/reliability on the broker is always welcome.

Nice work! I'd be curious to see a comparison w/ RabbitMQ's MQTT extension. By any chance have you done such a comparison? Also I assume the comparisons are done at QOS level 0?

I have in fact tested RabbitMQ but it was not at all impressive. The two fastest brokers I have found are emqtt and Mosquitto, both good at either large or small broadcasts but not both.

Edit: yes they are qos 0 (just like Redis)

  > The two fastest brokers I have found are emqtt and Mosquitto

What about vernemq? I [ran across][1] a comment a while back about some tradeoffs with emqtt's usage of mnesia. Haven't gotten around to trying verne myself yet though.

[1]: https://github.com/erlio/vernemq/issues/83#issuecomment-1781...

We can have a big bench-off with more brokers later on. I have tested maybe 6 and from this I have decided to write my own. I wouldn't spend time reinventing the wheel for free if I didn't have a reason for it :) Then you also have memory usage which is impossible to optimize in VM/GC languages.

isn't the big + of rabbitmq that it can be configured HA, while the others can't?

Do you know why/how is it faster than say RabbitMQ? Is it some tradeoff in features for instance?

Is there some information on what makes this faster than the alternatives ?

I would like to see metrics against Apache Kafka.

They are not even remotely worth comparing. They are so different in goals.

Probably the best comparison would be to ZeroMQ and or other MQTT brokers.

Why do only mqtt ? You usually need several protocols anyway. E.g crossbar does wamp, mqtt and http, with pub/sub and rpc.

Reply on news.ycombinator.com