Comment by PaulDavisThe1st | Hacker News Reader

Comment by PaulDavisThe1st | Hacker News Reader

Parent Full thread

PaulDavisThe1st

It's a fine analogy. The only problem is that I don't recall ever coming across the equivalent of a CNC machine for the class of problems I face in my work.

Thread-safe lock-free sparse integer-to-integer map? No CNC for that.

Translating time between two domains, one of which is linear and monotonic, and the other is non-linear and non-monotonic. No CNC for that.

Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.

I could go on, but you get the point.

What tends to be more like a CNC machine are libraries. For example, realizing that you need some sort of reference-counting system for lifetime management, preferably combined with pointer-like behavior ... and then discovering boost::shared_ptr (later to be std::shared_ptr) ... now that's like getting a new CNC machine. But it doesn't require a new language (and realistically, it didn't even require the library - the library just made it possible to not implement it locally).

I think what I'm really trying to say is that I rarely come across problems where I think that the kind of help offered by the putative "new CNC machine aka new language" is anywhere nearly as substantive as the help offered by an actual CNC machine to a cabinet making company. Put differently, the new tool (language) still leaves the problem essentially as hard as it was before.

p.s. a good friend runs a high end wood shop, and I'm fairly aware of the impact their first CNC machine made to what they could do.

When the web became popularized there problem of writing really fast concurrent servers that can handle 10k connections without the overhead of 10k threads.

This problem is arguably harder than the example problems you gave and this problem was solved by language primitives that now exist in basically every popular language. These primitives, when used basically change the nature of the language they are used in.

These primitives (async await) are more than libraries. They intrinsically change the nature of your code. (Though technically they could be made into libraries for languages without async await it's just the syntax would be extremely busy)

This only occured because the web was popular and the specific problem of servers and IO changed from a specific problem to a general one. So when someone wants to create a new language it's to attack a general problem.

Your issues in your example look to be somewhat domain specific, so new languages won't really help you in these specific areas you need to handle.

>Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.

I would say that for this example there are enough general issues here that modern languages CAN help you with. For example do you want to program in a language that can guarantee with helpful static error messages that your code will never have a data race or a seg fault or a buffer overflow or a dangling pointer?

Well there's a language that can help you here. In the same vein I've seen languages go even further then this and guarantee that the compiler will never ever let you write code that will make your program crash.

I think we can both agree that these general features that improve safety WILL make the issues you face easier.

Based on my experience (and I was doing web stuff starting in 92), this is a mistelling of the tale you're trying tell.

The problem was called "the thundering herd": if you had N threads all sleeping/waiting on a condition, and then that condition was raised/signalled, there were no OS primitives available that would wake only a single thread. Instead they all woke up, tried to get whatever work was available, only one succeeded, the rest go back to sleep. Incredible waste of cycles. These days, you can signal a condition in a way that will only wake a single thread that is waiting on it. Problem solved, for every language, without language modifications.

This was NOT fixed at the language level. It was fixed by adding new OS-level primitives that did the right thing.

Async-wait is another wrinkle in this, but for those of us old enough to remember life before pthreads, that was already effectively taken care of using threads (whatever the API) and existing OS-level sleep/wait primitives.

Doing this without threads is popular among the cool kids these days, but that's even harder than doing it with threads. Consequently, various languages have wrapped this sort of code into builtins in the language. Yes, that makes thread-less async wait easier to code, but it doesn't actually address the design problems where you might be using thread-less async wait to accomplish something.

The problem with waveform caching is not data races etc (though of course, those issues are hard enough). It's figuring out what you should cache and when. The best answers vary depending on user behavior, so you need an adaptive approach that isn't particularly linear, and you also a need way to recognize when user behavior means you should clear out everything in the cache and start over.

Your recollection is spot on.

Re. CnC, how about something like:

    @thread-safe @lock-free @sparse
    map map[K,V]

(Let's not get hunged-up syntax.)

> What tends to be more like a CNC machine are libraries.

If a language or library gives you composable semantics, you may have a programming CnC on your hand. CnC requires minimally one degree of disconnect between the artefact and the artisan. A language compiler/runtime (or library) that applies composable semantics to logical & computational abstractions.

> (Let's not get hunged-up syntax.)

But the syntax is precisely the only thing a language can give you that a library cannot.

I said don't get hung up on it precisely for that reason. We're not ignoring it, just simply noting that the magic is not in syntax.

Right, but that means it has to be implemented by someone. It could be a language builtin, but almost nobody is going to switch languages for such a feature (if it could even exist as a language feature anyway). Or it could be a library, in which case the question of better languages is again moot.

The idea that the compiler could somehow pick "the right" implementation of a sparse unordered map based on a list of constraints that combine to create potentially dozens of versions strikes me as far-fetched. Even specifying "lock-free" for example is very, very far from providing enough detail about what is actually required. Wait-free? Readers-not-blocked-by-writers? Writers-not-blocked-by-readers? etc. etc.

I don't dream of a future when compilers (or something) can somehow do all this, and I'm not convinced it will arrive. But I've been (very) wrong before.

Fair points. It would need to be a meta langauge of sorts, with compilation stages, stuff like that. Maybe a compromise can be reached with customizable compilers. PaulDavisThe1st's meta-compiler may know which sparse lock-free doodad you want. Even a CnC machine can't just spit out anything under the sun. Key thought here is industrialization of software production. It will (has to) happen (not that I'm pining for it :o) And our little chat here is whether programming languages will have a role to play here.

But the syntax is precisely the only thing a language can give you that a library cannot.

This might be true on a similar level to all Turing-complete languages being equivalent, but I’m not sure how useful it is beyond that level. For example, the features that a language provides as directly supported building blocks and the guarantees a language makes about how certain entities will or won’t behave and relate to each other profoundly affects the developer’s experience. That remains true even if some of those features could eventually have been recreated with a library modulo syntax and even if a perfect programmer would always use them properly and never rely on the language to prevent a mistake.

I'm referring to a similar overlapping problem called C10k. http://www.kegel.com/c10k.html

This was fixed at the OS level. But the usage of new system calls involved fundamental shifts in basically every language to account for the new paradigm.

Reply on news.ycombinator.com