boris

"The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt."

Bertrand Russell

It is lz4. While all that you say about disk being slow makes sense, measurements show that in this specific situation disabling compression improves performance by a little. My hand-wavy explanation is that freeing CPU to doing compilation rather than [de]compression is a better use of it, with all the uncompressed disk writes probably ending up in the system's file cache.

> Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?

Yes, that would be nice, but the tricky question is can any of this information change without the compiler binary mtime changing? First off, GCC's gcc/g++ binaries are drivers and are not what does the actual compilation, it's private cc1/cc1plus binaries that do the job. Can one of these change but not the driver? I think it's plausible (some package manager optimization where the file is not touched if it hasn't changed). So at a minimum we would need to discover where those are located (probably by invoking gcc/g++) and checking them as well. Could there be something else? Who knows. We value speed very much but we value correctness even more.

I think a more fruitful direction to explore is to improve GCC itself to dump all this information in a single invocation and in a machine-readable format (JSON). I think if we go from 70ms to 14ms (and perhaps even lower because this special GCC mode could conceivably do things faster than how we do it now), it would be good enough.

[build2 author here] Thanks for the feedback! Some additional details:

> The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why.

Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table.

> The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely.

We are talking about ~2% performance difference here. Parallelizing even a small amount of work across 24 threads rather that doing it serially saving a percent or two feels plausible to me.

> There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches.

It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.

> They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.

I measured this, it costs 70ms or ~2% of the overall time.

At least you know it was written by a fallible human. But if you point out my mistake, I will be happy to fix it.

I don't know, someone who solves their borrow checker errors by adding more dynamic allocations (which seems to be common/recommended), will be taught how to program in Java, not C++.

> Commands in a CLI … I think what makes for good product design is having a very small number of primitives.

Small but not too small. Case in point: shell scripts (POSIX shell, bash) where the scripting part was decided to be modelled as commands thus not introducing another bunch of concepts. We all know what the result is (hot, slow mess).

GCC (libstdc++) as all other major C++ runtimes (libc++, MSVC) implements the small object optimization for std::function where a small enough callable is stored directly in std::function's state instead of on the heap. Across these implementations, you can reply on being able to capture two pointers without a dynamic allocation.

The strangest thing about Flow is that its compiler is implemented in C#. So if you decide to use it in your C++ codebase, you now have a C#/.Net dependency, at least at build time.

I believe the point is if something is UB, like NULL pointer dereference, then the compiler can assume it can't happen and eliminate some other code paths based on that. And that, in turn, could be exploitable.

What is the filesystem story in OpenBSD? Anything CoW/snapshot'able on the horizon?

What would be the SQLite's equivalent to indexing starting from 1, not 0? Off the top of my head I can't think of anything that would go so much against the grain.

> We have been optimized very hard by evolution to be good at running, so there shouldn’t be any “easy” technologies that would make us dramatically faster or more efficient.

I wonder if these new shoes have the same affect on natural (i.e., non-paved) surfaces? Plus, they all look quite high off the ground (probably all those plates and foam need space) and that doesn't help with stability when running over rocks, etc.

There is a parallel with database transactions: it's great if you can do everything in a single database/transaction (atomic monorepo commit). But that only scales so far (on both dimensions: single database and single transaction). You can try distributed transactions (multiple coordinated commits) but that also has limits. The next step is eventual consistency, which would be equivalent to releasing a new version of the component while preserving the old one and with dependents eventually migrating to it at their own pace.

Yes, I've seen this one in our logs. Quite obnoxious, but at least it identifies itself as a bot and, at least in our case (cgit host), does not generate much traffic. The bulk of our traffic comes from bots that pretend to be real browsers and that use a large number of IP addresses (mostly from Brazil and Asia in our case).

I've been playing cat and mouse trying to block them for the past week and here are a couple of observations/ideas, in case this is helpful to someone:

* As mentioned above, the bulk of the traffic comes from a large number of IPs, each issuing only a few requests a day, and they pretend to be real UAs.

* Most of them don't bother sending the referrer URL, but not all (some bots from Huawei Cloud do, but they currently don't generate much traffic).

* The first thing I tried was to throttle bandwidth for URLs that contain id= (which on a cgit instance generate the bulk of the bot traffic). So I set the bandwidth to 1Kb/s and thought surely most of the bots will not be willing to wait for 10-20s to download the page. Surprise: they didn't care. They just waited and kept coming back.

* BTW, they also used keep alive connections if ones were offered. So another thing I did was disable keep alive for the /cgit/ locations. Failed that enough bots would routinely hog up all the available connections.

* My current solution is to deny requests for all URLs containing id= unless they also contain the `notbot` parameter in the query string (and which I suggest legitimate users add in the custom error message for 403). I also currently only do this if the referrer is not present but I may have to change that if the bots adapt. Overall, this helped with the load and freed up connections to legitimate users, but the bots didn't go away. They still request, get 403, but keep coming back.

My conclusion from this experience is that you really only have two options: either do something ad hoc, very specific to your site (like the notbot in query string) that whoever runs the bots won't bother adapting to or you have to employ someone with enough resources (like Cloudflare) to fight them for you. Using some "standard" solution (like rate limit, Anubis, etc) is not going to work -- they have enough resources to eat up the cost and/or adapt.

The main reason you would attach a database and then jump through hoops like qualifying tables is to have transactions cover all the attached databases. If you don't need that, then you can just open separate connections to each database without needing to jump through any hoops. So the fact that WAL does not provide that is a big drawback.

I would say there is no certainty with eventual consistency, only hope.

> And because Electric syncs every change granularly, you are certain that the state of your local database is exactly the same as the server's.

I don't see how this certainty follows from "granularity" (whatever that means in this context). I believe to have such a certainty one would need the synchronization to happen within a single transaction that spans both client and server databases.

> It also has 6 seconds of inactivity before starting any useful work. For comparison, ninja takes 0.4 seconds to start compiling the 2,468,083 line llvm project. Ninja is not a 100% fair comparison to other tools, because it benefits from some “baked in” build logic by the tool that created the ninja file, but I think it’s a reasonable “speed of light” performance benchmark for build systems.

This is an important observation that is often overlooked. What’s more, the changes to the information on which this “baked in” build logic is based is not tracked very precisely.

How close can we get to this “speed of light” without such “baking in”? I ran a little benchmark (not 100% accurate for various reasons but good enough as a general indication) which builds the same project (Xerces-C++) both with ninja as configured by CMake and with build2, which doesn’t require a separate step and does configuration management as part of the build (and with precise change tracking). Ninja builds this project from scratch in 3.23s while build2 builds it in 3.54s. If we omit some of the steps done by CMake (like generating config.h) by not cleaning the corresponding files, then the time goes down to 3.28s. For reference, the CMake step takes 4.83s. So a fully from-scratch CMake+ninja build actually takes 8s, which is what you would normally pay if you were using this project as a dependency.

> Header units are supposed to partially address this but right now they are not supported in any build systems properly (except perhaps msbuild?).

They are supported in build2 when used with GCC (via the module mapper mechanism it offers). In fact, I would be surprised if they were supported by msbuild, provided by "properly" we mean without having to manually specify dependencies involving header units and without imposing non-standard limitations (like inability to use macros exported by header units to conditionally import other header units).

> In other words, hypothetical improvements by AWS (and other Clouds) are extremely overrated.

Interesting, in a recent thread (I think it was about Redis going back open source) an AWS employer was bragging about substantial concurrency optimizations they implemented in Valkey. At the time I thought it could have been a great differentiator to keep proprietary but perhaps they decide to sacrifice it to help make sure Valkey takes over the Redis midshare.

I watched that but I don't see it as convincing. Let's take the AWS example brought up in the talk. The "compete on the interface, not (open source) implementation" idea I think misses (at least) the following points:

1. AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say some performance optimizations) they can keep to themselves. It's good business sense if it allows them to further differentiate their "improved" offering from your "vanilla" service.

2. Competing on the the interface in this case really means competing on related services like management, etc. So your thesis is that you will provide a better/cheaper managed service than AWS. Even if that's true (a big if), most of the time the decision which service to use will have little to do with technical merit. I.e. we already use AWS, have SLA painfully negotiated, get volume discounts, etc. Do we really want to go through all of this with another vendor just for one extra service.

Just a couple of thoughts that will hopefully help you sharpen your thesis.

In our package repository web interface we aimed for high information density over design fluff: https://cppget.org Especially our builds page.

In GCC 14, C++ modules were unusable (incomplete, full of bugs, no std modules, etc). I haven't tried 15 yet but if that changed, then it definitely qualifies for a "great improvement".

Thanks for sharing. If I understood correctly, you have rate-limiter specific urls (those with commit ids) that are infrequently requested by users but frequently by bots. Which means, provided the bots continue trying to request them, any user request will most likely end up being denies. In this case a simpler solution might be to just block such urls outright. The only advantage of your more complex solution that I can see is that if the bots stop requesting these urls, they will again become accessible to the normal users. Or am I missing something?

> How do they know that these are LLM crawlers and not anything else?

I can tell you what it looks like in case of a git web interface like cgit: you get a burst of one or two isolated requests from a large number of IPs each for very obscure (but different) URLs, like a file contents at a specific commit id. And the user agent suggesting it's coming from IPhone or Android.

> and more broadly, there is now a portable, embeddable, and sandboxed version of Perl that others can use in their projects. I can think of a lot of use cases, and helping other developers makes any hardship I endured worth it.

Yes, exactly, and thank you for that!

I don't know of any other general-purpose scripting language that can be run in WebAssembly. This could make Perl an interesting choice for writing sandboxed build system rules.

> [M]any developers use C++ as if it was still the previous millennium. [...] C++ now offers modules that deliver proper modularity.

C++ may offer modules (in fact, it's been offering them since 2020), however, when it comes to their implementation in mainstream C++ compilers, only now things are becoming sort of usable with modules still being a challenge in more complex projects due to compiler bugs in the corner cases.

I think we need to be honest and upfront about this. I've talked to quite a few people who have tried to use modules but were unpleasantly surprised by how rough the experience was.

> If make supplied it's own shell language, a simplified one, then everything would be fantastic.

We did exactly that in build2, a modern make re-thought. And we provide a bunch of standard utilities like sed, find, etc., that work the same everywhere, including Windows. Here is an example of a non-trivial recipe: https://github.com/build2/libbuild2-autoconf/blob/17f637c1ca...

> "Processes are the software analog of processors." Yes of course.

Maybe this analogy worked in 1986 when hardware was a lot less reliable, but I don't think it goes very far these days: processes die all the time (normal termination, crash, get killed). When was the last time a processor or core died on you? In fact, according to this analogy most of us are running MS-DOS equivalent of systems since if your processor or core dies, your machine dies. And I don't see this changing any time soon.