I discovered a critical exploit in ZeroMQ with mostly pure luck | Hacker News Reader

I discovered a critical exploit in ZeroMQ with mostly pure luck

fangpenlin.com

204 pointsfangpenlin61 comments

The Elliptic Curve Cryptography article (mentioned in the ZeroMQ article) the author wrote is really good:

https://fangpenlin.com/posts/2019/10/07/elliptic-curve-crypt...

I love how they wanted to learn about ECC and decided to write the book they wished existed. What a perfect mindset.

Meta comment on this (excellent) article: I liked the memes and the accessibility-friendly captions thereof.

Now I'm really curious how blind people experience memes. Like if you read "cat at table meme" is it funny in it's own way?

I imagine there's some amount of conditioning (for lack of a better word), and some amount of how well the meme author annotated it. Mostly just postulating, with a small amount of experience having been an assistant to a blind person for a short stint. But I imagine you're onto something, that as people who experience memes visually, we'd find the meme's description less funny than someone who relied on that description to experience the comedic effect of the meme; they might just use their imagination to fill in the gaps more naturally than we do. But there are of course limits to what details we can divine from our imagination, so I imagine there's a lot of memes where blind people are left contextless unfortunately.

I don't know that everyone has one of these. But the professor I assisted for that aforementioned short stint also had a special braille printer, of course. I believe these printers have since advanced to the point of being able to render photos in a sort of limited fashion where the paper is indented to contour to lines. I believe there are also tactile tablets now for the visually disabled.

I find any sort of animated pictures next to the text supremely annoying. I had to scroll the text to keep them out of the screen just so I could read in peace.

And yet you liked it... I wonder if this is a generational thing (I was born in 1980's)? Or "I don't run adblocker" thing?

Thanks for the feedbacks. The author here. Personally I found it more "fun" (well, I guess for everybody it's different) to have some meme in between of the articles, some people hate to read too much text all in once, but yeah, I get your point. So I was thinking maybe I should add a switch like "meme off" to hide all of those at the top of article so that people don't like it can feel better while reading it.

We should start a club.

In the meantime, I just use an image blocker extension when I encounter those articles.

My mobile browser (Cromite on Android, Chromium fork) also has a setting to toggle images, which is also good.

You can probably set your browser to play animated gifs only once; that's what I did.

I was also born in the 1980s but am fairly immature, and liked the article layout a lot.

It's extremely distracting. It's a competently written article mixed with a bunch of childish animated nonsense

> As a software engineer, I am lazy, so I always love to reuse existing tools as much as possible… That’s why I decided to take the chance to learn how Elliptic Curve Cryptography works.

Imagine if they weren’t lazy!

Laziness is a superpower. At least for some people and for some values of laziness.

This has got to be 30 years old or so:

https://thethreevirtues.com

According to Larry Wall, the original author of the Perl programming language, there are three great virtues of a programmer; Laziness, Impatience and Hubris

Laziness: The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful and document what you wrote so you don't have to answer so many questions about it.

Impatience: The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least pretend to.

Hubris: The quality that makes you write (and maintain) programs that other people won't want to say bad things about.

I have worked with non-lazy engineers too. The amount of convoluted barely working franken stuff they come up with is staggering. No obstacle will stop them from digging the hole deeper.

Nah, they are alright. The real issue is they are so busy working, they never stop to really think about what they are building.

“I divide my officers into four classes as follows: The clever, the industrious, the lazy, and the stupid. Each officer always possesses two of these qualities.

Those who are clever and industrious I appoint to the General Staff. Use can under certain circumstances be made of those who are stupid and lazy. The man who is clever and lazy qualifies for the highest leadership posts. He has the requisite nerves and the mental clarity for difficult decisions. But whoever is stupid and industrious must be got rid of, for he is too dangerous.”

-- General Freiherr von Hammerstein-Equord

This is why teams need both. Relentless, obsessive people are great for the things that need it. Most stuff is usually fine to just do the lazy way though and move on.

Different definitions of lazy.

* lazy - wants do as little work in the future as possible and so spends extra time now solving the problem the right way.

* lazy - has no consideration for the future and takes a straight line path to solving the problem now. Spends all future time fixing problems created from this approach.

There’s a subset of the second definition where you delegate all the future fixing time to some unfortunate young developers, both wasting their time and tainting their souls in the process.

Well said! Have not seen the tension expressed in this way before.

I'm of the first type - which I call "busyness in the pursuit of eventual laziness" :)

something something lies are eventual consistency?

Fantastic post, I loved everything about it. Fang-Pen, I'm interested in your book but it's apparently only 16% complete, is that right? In any case nice find, thanks for sharing, and please keep writing! :)

The author here. Thanks for the interest. Crazy things happened in life last year, so I barely find a time to write anything. Recently finally find a time to pick up writing again. Hopefully I can add more content to the book this year.

> By reading the source code, I realized that the incoming data was put into a fixed-size static buffer in the stack, and the payload was decrypted into another fixed-size buffer. There’s no boundary or size check.

This is not normal. It's amateurish in the extreme that leads to the only conclusion that whoever wrote this ZeroMQ thing is not a real software engineer. I.e. stay away at all costs.

> This is not normal. It's amateurish in the extreme that leads to the only conclusion that whoever wrote this ZeroMQ thing is not a real software engineer. I.e. stay away at all costs.

I don't think that's a remotely fair assessment. ZeroMQ is a very large and quite popular project but it's also getting close to two decades old if I remember correctly. Any large C or C++ project that is that old is going to have quite a bit of historical cruft. And looking at some of the code that said vulnerability touched, most of that code was over a decade old.

Not to claim that it's any less severe but this is the nature of long lived projects. Unless they are massively privileged, they tend to have more code than eyes to look at said code and said code often was written in the bad old days.

> it's also getting close to two decades old if I remember correctly. Any large C or C++ project that is that old is going to have quite a bit of historical cruft.

I don't think writing arbitrary data into fixed-size buffer without boundary checks is just an artifact of being historical cruft, it's a ridiculous mistake no matter which time period it was written in. Whoever wrote that code decades ago was incredibly amateurish.

That assuming the code was written that way initially. More often than not in long running projects pieces get moved around, refactored, functionality added and removed, and silent assumptions that were true before aren't true anymore. Somebody coded functionality for fixed data buffer, somebody else extended it with variable sized data but was not aware fixed buffers are being used, that stuff happens. We live in a myriad of glass castles, don't be so quick to throw stones around.

The tone of some comments like this one makes me wonder if the authors ever wrote any software of reasonable complexity, on their free time while getting paid zero dollars.

With such high standards I wonder why this people use such amateur software and not make or buy their own professional grade software.

Also sheds light why truly free open source software is such a thankless and hazardous activity

Pieter Hintjens who started ZeroMQ, advocated for 'optimistic merging' as a strategy to encourage community & project building [1] (prev discussed in [2]). For all of the benefits listed it does open it up to lower quality or malicious merges.

[1] http://hintjens.com/blog:106 [2] https://news.ycombinator.com/item?id=39880972

You're that confident in every single line of code you wrote?

No, but I would never, ever, ever, EVER write data to a buffer without specifying the buffer size or reallocing where necessary.

This just smells so much like a Javascript script kiddy who wanted to join the cool brigade and write something h4kor1sh in C. Ugh.

This comment is like opening the box to a time capsule from 2010. I haven't seen anyone communicate like this since I last touched an IRC channel.

I'm dying at the idea that someone would think C is the "cool brigade".

Yeah, C hasn't been "cool" in a few decades.

Literally every serious C/C++ project has shipped memory unsafety vulnerabilities. We have discovered, as the global community of programmers, that humans are not smart enough to write C code without doing that. It is time to blame the language (or the species) and move on.

Let's not pretend that the people writing the unsafe code are unimaginably stupid. They are extremely imaginably stupid, as we all are.

Most memory vulnerabilities are use-after-free which due to the nature of C is a very easy mistake to make.

Buffer overflows are simple inexcusable, especially if its "we didn't bother checking" rather than "we got the size wrong due to human error".

The first case is not normal, people like that should not be programming HTML let alone C code.

I have some really bad news for you about OpenSSL.

the most interesting part is the last one -> hire me..

One the bug is in curbezmq not zmq.

Two do not expose zmq to untrusted networks.

edit: lol their website doesn't even have a valid cert http://curvezmq.org/

> Two do not expose zmq to untrusted networks

one: Is this documented somewhere? I use zeromq for the (internal, but by design usually accessible on the public internet) API of my project

two: what happened to zero trust? Every network is untrusted.

Zeromq will have changed a lot since then, but some time in the 2010s, I prototyped a system using it (which was going to be a major production system in a large tech company) and had weird unexpected blocking issues with it. To debug, I sat down to read a bunch of the zeromq code, just to realize that it was using assert() to handle wire protocol errors (unrelated to the blocking bug).

I've never dropped a piece of software as quickly as that.

More or less my experience as well. Asserting on bad user configuration, asserting on OS errors that weren't in a particular list. I followed their recommendation of having a "small, simple, reliable" broker and it kept crashing on asserts in the library at the worst times.

I normally assume that any software which is not specifically designed to be exposed to internet, should not be exposed.

Security is hard, and proxies/VPNs are cheap.

I don't know if the ZMQ people have ever admitted to it, but you'd be crazy to expose ZMQ to the public internet.

This is the equivalent to "trust me bro".

Do you have any specific reasons why this is a bad idea? Especially if it's been secured, as the article implies it was.

See my response to a sibling of the comment you're responding to. The library had shocking code quality issues. It's unlikely that they're all peachy now.

The other side of this is that while Pieter's writing was marketing genius, it was also woefully understating the complexity of any practical use case. The way I tried to summarize that to folks who were keen to try zeromq then was that they should start at the back of the book with the most complex example, and that's by far the simplest setup that they could hope to end up with once they start thinking about putting something into production. And everything leading up to that - a book no less - was exclusively educational/toy use cases.

> internal, but by design usually accessible on the public internet

Your API can be accessible obviously, but put ZeroMQ behind a firewall so only the API server can reach it.

If it’s running on the same server, at least block the port ZeroMQ is listening on from the outside world.

People make fun of Kubernetes or "resume driven development" for making things more complex than they need to be, but this is why you want mTLS via a sidecar with short auto renewed certificates on a mesh inside your distributed system of a operating stack, when the system is big enough to justify that complexity. Something the size of, like, Airbnb should have that.

Or a wireguard VPN. Or even just socat with mTLS inside systemd.

There are easier ways to achieve that than kubernetes with sidecar mesh.

Evidently it has never had such a certificate: https://crt.sh/?q=curvezmq.org

> Reading code is underrated, and many software engineers don’t understand how and why they should read it. Reading code is much harder than writing code because writing code translates your thoughts into code, and reading code is the opposite.

That's like saying reading Hamlet is harder than writing it. What kind of garbage do you have to be filling your head with all day to hold such a dismal opinion of software?

It's true. The easiest code to read is the code you wrote yourself. To read other people's code, you have to decode it. And it's very easy to write convoluted "clever" code that you yourself will not be able to read six months down the line, like what Brian Kernighan meant when he said:

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

I strongly believe it's easier to write working code than it is to untangle it later on. Writing readable code is a skill.

The modal programmer barely reads stack traces that tell them where exactly something is happening, even when it tells them exactly the mistake they're making.

Digging into other people's code, reading it, and having enough education and context to understand why they did things a certain way is an even rarer skill.

Yep. Problem solving, debugging, and diagnostic skills are often non existent. "It doesn't work. Can you take a look?"

I'd be pretty rich if I got $0.01 every time I was asked that.

Also, it gives you the heuristics to decide if a code base is crazy and not redeemable.

If you didn't get at least 1c per question like that, raise your rates!

I don’t mind taking a look, but at least tell me you googled it first!

Reading Hamlet is harder than writing your own, not very good, short story about something that you’re motivated to share.

Reading Hamlet such that you fully understand its internal structures and design and could propose changes to it that maintain cohesion and do not make it drastically worse is harder still.

Proprietary enterprise production code

Reading most code is more akin to reading Hamlet fan fiction.

Reading Hamlet is as hard as writing it, no reader has ever enjoyed the same pleasure the writer indulged, being finished.

"Reading" Hamlet is something my pet hamster does for breakfast.

This was not "pure luck". Reminded me of that quote about chance favoring the prepared:

"Dans les champs de l'observation le hasard ne favorise que les esprits préparés." -Louis Pasteur

  In the fields of observation chance 
  favours only the prepared mind.

Variant translations of this or similar statements include:

  Chance favors the prepared mind.

  Fortune favors the prepared mind.  

  In the field of observation, chance favors the prepared mind.

  Where observation is concerned, chance favors only the prepared mind.

https://en.wikiquote.org/wiki/Louis_Pasteur#Quotes

edit: "Louis Pasteur's quote "Chance favors the prepared mind" means that the better prepared and more knowledgeable you are, the more you'll be able to take advantage of any chance opportunities or observations.

"If you are unaware of things that influence a situation or an event, you are very unlikely to be able to identify any opportunity or learn anything significantly new. By having insight, interest, and aptitude related to the situation, you put yourself in the position to capitalize upon any hidden "nuggets" buried at the moment."

- https://asymmetric.pro/chance-favors-the-prepared-mind/

Reply on news.ycombinator.com