_urga

I was dealing with "Nagle's delay" only yesterday, adding "setsockopt(TCP_NODELAY)" for the alpha version of a new high-performance payments database called TigerBeetle: https://github.com/coilhq/tiger-beetle

Sure, but the buffer cache is also part of the memory hierarchy. At every level, it's always good not to go wild with cache misses. Not that that's what you're proposing of course.

"It is well suited for OS and driver dev though"

Rust is brilliant, but for drivers, I would go for a language with OOM safety and strict static allocation, especially for embedded environments.

I'm with you on the mental model, but from a hardware point of view what you also don't want is hundreds of random disk seeks. I know we're almost in an SSD-only world but HDDs are still a thing and large sequential reads are always important at any level of the memory hierarchy.

I wrote something similar but for the ZIP format, called Pure, to detect hidden buffer bleeds outside and between and within structures, amongst other anomalies. It's also amazing what you can do with a ZIP file.

Pure: https://github.com/ronomon/pure

The goal of Pure is to reduce the attack surface available for zero-days by doing the simple yet critical buffer bounds checks that most end-user application software will not, and to do this for more and more popular file formats.

The vision is that one day you can deploy Pure at your email server and get much stronger guarantees on the safety of email attachments that you open... as well as on the attachments you send out... for example that your billing software is not leaking the contents of sensitive server memory into XLSX reports that you are mass mailing to clients (Pure actually found two cases of prominent investment firms that were doing just this).

It's like type checking or linting but for everyday file formats.

Thanks for writing this.

"Put another way: Most of us enjoy the mental stimulation of programming, and we enjoy the mental challenges (in general). C makes us feel clever. Witness the "obfuscated C programming contest" etc."

Yep, only C makes me feel stupid (but I enjoy that experience too!).

The first rule of C is that no one masters C, but you could try anyway and still have time to master Zig in a matter of weeks, which is a rounding error. Given that both offer a C compatible ABI, what would serve your projects better?

"Anybody can build a good chair for $10,000 or a good PC for $100,000."

And some people can build a great PC for $1,000 that runs circles around the good PC for $100,000.

There's so much more to engineering than thinking in terms of time and cost constraints. Those are real constraints, but they're not the most important.

Engineering is design. If you have good design, good insight, you can do things that people with infinite time and budget could never dream to achieve. You can start making a product that's a hundred times more powerful for a tenth of the price in a fraction of the time. If you don't have good design, good insight, then no amount of time or budget can help you.

"optimize like a Vulcan"... classic!

Rule 1 and 2 depend on context, whether you're working on an existing program or a new program. They can be true or false. They can really help or they can really hurt. Are you going into an existing system to do performance optimization? Sure, don't guess, measure. Are you designing a new system? Throw out those parroted premature optimization mantras... you are responsible for designing for performance upfront. You will always measure but depending on context you will design for speed first and then test your prototype with measurements. There's no way around an initial hypothesis when you're designing new systems. You have to start somewhere. That's where Jeff Dean's rule always to do back of the envelope guesses will pay off in orders of magnitude, many times over.

Rule 3 and 4 are gold and always true.

Rule 5 is the key to good design.

Yep... there are ways to make it happen.

Sure, and I don't disagree that uid might need to change at runtime, but here we're talking about a struct field being const.

Something as important as "uid" should be "const".

Play chess, and do physical exercise.

Both will develop your mental stamina, and chess especially will hone your adversarial thinking.

You may even find that your chess fitness is directly comparable to your programming fitness and vice-versa. When you're programming fit through working on some incredibly hard problems in the week (think distributed systems or file systems or algorithms), then your chess will show an improvement. And when you're playing several hours of chess a day against a good opponent, then your programming will likewise benefit and your bug count drop as you naturally start thinking further ahead.

Thanks for the excellent example of the TLS Selfie attack.

Having worked on protocols, I have found again and again that making the context explicit by cryptographically binding intent to the message is paramount.

Where things get interesting is the mismatch between Big-O data structure complexity analysis (as typically discussed in interviews) and actually knowing about the hardware, the operating system, the memory hierarchy and things like cost of context switches and cache misses.

Some real-life practical examples: knowing when a simple O(n) linear search beats O(log n) binary search, or knowing when a multiple substring search algorithm based on a simple Rabin-Karp and L1 cached hash table lookup will outperform the theoretically more optimal Aho-Corasick.

The backstory:

Somewhere between an HN thread on July 4 2019 (https://news.ycombinator.com/item?id=20352439) and David Fifield's generous references to @ronomon/zip on his website and in his "A better zip bomb" paper, I received an email from Maxim Vainstein, an R&D Lead at Microsoft and the head of their Product Release and Security team, asking me to port @ronomon/zip from JavaScript to C, so they could use it as a static analysis tool to scan all software released by Microsoft.

Microsoft were happy for me to licence the work under the MIT licence and so @ronomon/pure was released today. I have only a few years experience in C, and am confident there will be some fairly embarrassing flaws in the code, please let me know what you think. I am also considering a port to Zig as a safer, simpler implementation that remains C-ABI compatible for portability and easy embedding.

At present, Pure does exhaustive file format checks on zip files, but I want to expand Pure as an open-source static analysis tool for more file formats, starting with MS-CFBF Office files. This recent paper in VirusBulletin shows some staggering results for static analysis to detect 90% of zero-day exploits in Office formats (see the table at the end): https://www.virusbulletin.com/uploads/pdf/magazine/2019/VB20...

I think email might prove to be the perfect place where something like Pure could be put to good use, where a combination of policy (no executables, no macros) and static analysis on the remaining file formats can narrow the gap and obviate the need for machine learning or CVE-laden antivirus, protecting whole groups of users through an opt-in "please defend me from malware email attachments" mode, without requiring buggy software vendors to improve the quality of their software. At the same time, it's moving away from Postel's Law and helping to enforce and uphold open standards and debug software with fail-fast feedback.

Email is the number one delivery vehicle for malware but most email providers don't have the open-source tools available to protect their users. My hope is that independent email providers such as Hey and Fastmail will consider sponsoring work on new file formats in Pure and come on board to encourage adoption.

That's a nice riff on the original:

"Again I saw that under the sun the race is not to the swift, nor the battle to the strong, nor bread to the wise, nor riches to the intelligent, nor favor to those with knowledge, but time and chance happen to them all." —Ecclesiastes 9:11

To understand where Ecclesiastes is coming from:

In the wisdom literature, books like Proverbs will typically state the common case and it's Ecclesiastes that will state the exception. This keeps the wisdom literature balanced as a set of principles, not rules.

The Hugh Keough quote is a great summary then of the main point, which I won't restate.

Thank you antirez!

I started using Redis around 2010 and I learned to appreciate so many things from you along the way:

* Data structures are fun.

* In-memory is fast and can be safe!

* Append-only logs are awesome.

* Databases can be more than MySQL.

* Complexity analysis is worth making clear in the documentation.

* Hybrid L1 cache-friendly data structures beat complexity analysis for small data.

* There are only so many hours you can work in a day.

* It's cool to sit by the pool.

* It's cool to have a screen name.

* Above all, code is poetry, not dependencies.

Do Hetzner or xneelo reflash the firmware on server hardware when they recycle them across users?

If you were to test ClamAV on a few thousand malware samples you would probably find that the detection rate would be in the low single digits.

At the same time, ClamAV has a terrifying CVE track record.

There's no upside, and all downside.

Absolutely, except the ddg of email would be written by one person.

I am not sure but I wouldn't think so, unless you have to serialize/deserialize or otherwise transform or inspect function call arguments in some or other way, as you need to do when binding JavaScript with C.

> This is just plain jane software engineering.

Thanks, love this quote!

We actually wrote the original version in C with SIMD extensions as a Node.js binding, but the JavaScript version was still twice as fast. I kid you not. You will find the reason for this in the README, it's the last bullet point under "Fast": https://github.com/ronomon/hash-table#fast

We actually use this to insert 400M elements.

The only reason the example inserts 4M elements was because Set and Object start to become prohibitively slow at some point and crash the process with too many allocations, not to mention the stress on the GC which now has to follow so many pointers.

HashTable performance is a fundamental component of any language.

One of the things I love about JavaScript is that you can write an almost 10x faster hash table than the standard library if you take care of cache misses and GC: https://github.com/ronomon/hash-table#motivation

In fact, all the usual low-level optimization techniques like reducing branch mispredictions and expensive memory accesses apply, and make a huge difference, even though you're writing in a high-level language.

"No benchmarks or any other numbers."

And yet plenty of logical argument. The author assumes a level of optimization experience on the part of the reader, for example that the reader appreciates the physical cost of branch mispredictions.

And it was fun to read!