Hacker News Reader

I've been programming for 35 years or more, and I've yet to come across situations where the hard problem was the language.

The hardest problem is usually something else, but IME some languages make it much easier to solve a given hard problem than others. Some languages are qualitatively safer. Some languages are much more expressive, in the sense that I can implement my solution with far less code/time/cost without compromising anywhere else.

There is a difference between whether I can solve a problem and how efficiently and pleasantly I can solve a problem. I find choice of language rarely affects the former but often profoundly affects the latter.

100%. But that's why I mentioned performance. I work with realtime audio, and this puts most of the "pleasantly solve the problem" options off the table.

It’s true that for genuinely performance-sensitive applications the options are more limited, but there are worlds of difference between assembly language, C and Rust, for example. There’s still a place for each of them, but other things being equal, Rust wins by a wide margin on both safety and expressiveness.

Might be true, but since the hard problems rarely involve safety or expressiveness, it's not clear what the benefit is.

I’m very curious about what kind of hard problems you have in mind where an implementation wouldn’t benefit from better safety and expressiveness. I’ve done my share of both mathematical and systems programming. In both cases I’d say eliminating types of bug that typically come with low-level/high-performance code was highly relevant. Also, quite often my biggest practical frustration was trying to iterate on a solution while language barriers made implementing each new attempt tedious in some way.

databases, haha. That's where modern programming throws most of the hard stuff nowadays.

in fact all of the hard parts of modern programming can be characterized by a single problem that all of these languages like go and rust are attempting to solve:

   Shared mutating state.

Really the parent and everyone here is just complaining about this one singular problem and how no language has completely solved it yet with a method that's Easy and safe.

In fact all these new languages have a sort of similar primary purpose of attempting to abstract this problem away.

That is not the "single problem" I am complaining about.

We already good methods for dealing with this. In fact, we have several, and what is often the hardest coding question is when to use which one (e.g. RCU versus mutexes).

There are plenty of other "hard" problems in modern programming that have nothing to do with shared mutating state (unless you broaden the definition of that so much that it becomes essentially meaningless).

Here's one example: resolving the tension between cache locality and time-varying data access patterns. Here's another: zero-copy data pathways between user space and hardware. Here's another: resolving the relative costs of fundamental GUI implementation choices (e.g. retained or immediate mode drawing models; pre-rendering components or using vector graphics; absolute positioning vs constraint packing)

Shared mutating state.

Really the parent and everyone here is just complaining about this one singular problem and how no language has completely solved it yet with a method that's Easy and safe.

If my comment was the parent you’re referring to, that isn’t really my position at all.

Certainly there are challenges that come with shared mutable state, but we also have some good tools for dealing with those already. There is a more general problem of managing side effects and external resources, where I think there’s room for better language support, so that might be a better example.

The biggest single frustration I find with many popular programming languages is that once I’ve figured out how to solve the real world problem we’re dealing with, I want to implement that solution as fast as my mind can think and my fingers can type. I don’t want to get bogged down with wrangling general purpose data structures and algorithms, or wondering how to implement an idea with whatever networking or database or mathematical library we use, or writing a screenful of infrastructure plumbing to talk to a cloud service, or trying to figure out all the possible ways that a function I’m calling might tell me it failed and hand-crafting a few lines of boilerplate for every single one.

I haven’t found a language and ecosystem yet that I feel give me that, even though I’ve found many features in reasonably well-known languages and libraries and tools that individually make good progress in that direction.

I gave 3 examples in one of my replies in this subthread: https://news.ycombinator.com/item?id=35036176

I love to swing a hammer as much as the next programmer. But if you offer me a nail gun, even with slightly lower precision, I will happily use it the majority of the time and revert to my hand tools when it's most appropriate. This is about developing force multipliers and producing leverage, not avoiding the craft.

I would say that a better analogy is that you've already got an experienced crew, some swinging hammers, some lugging nail guns and everyone's really quite good at what they do. But the plans from the architect are just really hard to build. You and the crew can do it, for sure, but the issues are not going to better or worse if the guys use the table saw to crosscut some of the framing elements or someone gets one of those new german tool widgets. I mean, sure, the tools might make a small difference to the process, but the overall experience will be dominated by the fact that the thing is just hard to build.

Let's keep going with the building analogy. You're running a cabinet making company. You've got all kinds of hand tools and power tools. You build jigs to make certain repetitive tasks faster. Then a ridiculously difficult design comes in for you to build. You and your crew are flummoxed by its complexity. Suddenly, someone offers you a CNC machine that you've never used before. What seemed hard is now easy. The nature of solving problems with a CNC is different. Using a CNC presents other challenges. But you have entered a new realm of what's possible. Analogies are a lot of BS but hopefully that gets across the flavor of what I'm talking about.

It's a fine analogy. The only problem is that I don't recall ever coming across the equivalent of a CNC machine for the class of problems I face in my work.

Thread-safe lock-free sparse integer-to-integer map? No CNC for that.

Translating time between two domains, one of which is linear and monotonic, and the other is non-linear and non-monotonic. No CNC for that.

Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.

I could go on, but you get the point.

What tends to be more like a CNC machine are libraries. For example, realizing that you need some sort of reference-counting system for lifetime management, preferably combined with pointer-like behavior ... and then discovering boost::shared_ptr (later to be std::shared_ptr) ... now that's like getting a new CNC machine. But it doesn't require a new language (and realistically, it didn't even require the library - the library just made it possible to not implement it locally).

I think what I'm really trying to say is that I rarely come across problems where I think that the kind of help offered by the putative "new CNC machine aka new language" is anywhere nearly as substantive as the help offered by an actual CNC machine to a cabinet making company. Put differently, the new tool (language) still leaves the problem essentially as hard as it was before.

p.s. a good friend runs a high end wood shop, and I'm fairly aware of the impact their first CNC machine made to what they could do.

When the web became popularized there problem of writing really fast concurrent servers that can handle 10k connections without the overhead of 10k threads.

This problem is arguably harder than the example problems you gave and this problem was solved by language primitives that now exist in basically every popular language. These primitives, when used basically change the nature of the language they are used in.

These primitives (async await) are more than libraries. They intrinsically change the nature of your code. (Though technically they could be made into libraries for languages without async await it's just the syntax would be extremely busy)

This only occured because the web was popular and the specific problem of servers and IO changed from a specific problem to a general one. So when someone wants to create a new language it's to attack a general problem.

Your issues in your example look to be somewhat domain specific, so new languages won't really help you in these specific areas you need to handle.

>Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.

I would say that for this example there are enough general issues here that modern languages CAN help you with. For example do you want to program in a language that can guarantee with helpful static error messages that your code will never have a data race or a seg fault or a buffer overflow or a dangling pointer?

Well there's a language that can help you here. In the same vein I've seen languages go even further then this and guarantee that the compiler will never ever let you write code that will make your program crash.

I think we can both agree that these general features that improve safety WILL make the issues you face easier.

Based on my experience (and I was doing web stuff starting in 92), this is a mistelling of the tale you're trying tell.

The problem was called "the thundering herd": if you had N threads all sleeping/waiting on a condition, and then that condition was raised/signalled, there were no OS primitives available that would wake only a single thread. Instead they all woke up, tried to get whatever work was available, only one succeeded, the rest go back to sleep. Incredible waste of cycles. These days, you can signal a condition in a way that will only wake a single thread that is waiting on it. Problem solved, for every language, without language modifications.

This was NOT fixed at the language level. It was fixed by adding new OS-level primitives that did the right thing.

Async-wait is another wrinkle in this, but for those of us old enough to remember life before pthreads, that was already effectively taken care of using threads (whatever the API) and existing OS-level sleep/wait primitives.

Doing this without threads is popular among the cool kids these days, but that's even harder than doing it with threads. Consequently, various languages have wrapped this sort of code into builtins in the language. Yes, that makes thread-less async wait easier to code, but it doesn't actually address the design problems where you might be using thread-less async wait to accomplish something.

The problem with waveform caching is not data races etc (though of course, those issues are hard enough). It's figuring out what you should cache and when. The best answers vary depending on user behavior, so you need an adaptive approach that isn't particularly linear, and you also a need way to recognize when user behavior means you should clear out everything in the cache and start over.

Your recollection is spot on.

Re. CnC, how about something like:

    @thread-safe @lock-free @sparse
    map map[K,V]

(Let's not get hunged-up syntax.)

> What tends to be more like a CNC machine are libraries.

If a language or library gives you composable semantics, you may have a programming CnC on your hand. CnC requires minimally one degree of disconnect between the artefact and the artisan. A language compiler/runtime (or library) that applies composable semantics to logical & computational abstractions.

> (Let's not get hunged-up syntax.)

But the syntax is precisely the only thing a language can give you that a library cannot.

I said don't get hung up on it precisely for that reason. We're not ignoring it, just simply noting that the magic is not in syntax.

Right, but that means it has to be implemented by someone. It could be a language builtin, but almost nobody is going to switch languages for such a feature (if it could even exist as a language feature anyway). Or it could be a library, in which case the question of better languages is again moot.

The idea that the compiler could somehow pick "the right" implementation of a sparse unordered map based on a list of constraints that combine to create potentially dozens of versions strikes me as far-fetched. Even specifying "lock-free" for example is very, very far from providing enough detail about what is actually required. Wait-free? Readers-not-blocked-by-writers? Writers-not-blocked-by-readers? etc. etc.

I don't dream of a future when compilers (or something) can somehow do all this, and I'm not convinced it will arrive. But I've been (very) wrong before.

Fair points. It would need to be a meta langauge of sorts, with compilation stages, stuff like that. Maybe a compromise can be reached with customizable compilers. PaulDavisThe1st's meta-compiler may know which sparse lock-free doodad you want. Even a CnC machine can't just spit out anything under the sun. Key thought here is industrialization of software production. It will (has to) happen (not that I'm pining for it :o) And our little chat here is whether programming languages will have a role to play here.

But the syntax is precisely the only thing a language can give you that a library cannot.

This might be true on a similar level to all Turing-complete languages being equivalent, but I’m not sure how useful it is beyond that level. For example, the features that a language provides as directly supported building blocks and the guarantees a language makes about how certain entities will or won’t behave and relate to each other profoundly affects the developer’s experience. That remains true even if some of those features could eventually have been recreated with a library modulo syntax and even if a perfect programmer would always use them properly and never rely on the language to prevent a mistake.

I'm referring to a similar overlapping problem called C10k. http://www.kegel.com/c10k.html

This was fixed at the OS level. But the usage of new system calls involved fundamental shifts in basically every language to account for the new paradigm.

> I've been programming for 35 years or more, and I've yet to come across situations where the hard problem was the language

The language can cause hard problems by itself. Arguably dynamic typing has been partly responsible for driving microservices beyond what's reasonable and thus introduced accidental complexity, largely because they don't have a typing discipline that enforces proper module boundaries.

That's the most obvious example to me, but far from the only one.

As a developer I would agree. As a maintainer I would disagree.

Some problems require paradigms to solve, and developing bespoke implementations makes maintenance extremely painful.

I've also been doing this for over 30 years and agree with you 100%. Never has the language been the impediment, or the thing keeping me awake at night.

We don't need more programming languages. All languages are either hated or not used. The language isn't the problem and it a poor craftsman that always blames the tool.

I've never been on a retrospective of a failed project and the reason being the language.

> The problems are hard because the problems are hard

I would respectfully submit that this is exactly where functional programming shines.

I would respectfully submit that you completely missed Paul's point.

He's doing real-time audio processing. So, first, he's got performance issues. In that environment, it can be very helpful to re-use a buffer without having to allocate one (and without a garbage collector). For most functional languages, that's not a place where they shine.

But, second, he's (probably) got problems like building echo cancellers. The hard problem is, how do you determine whether something is an echo? Writing it in an FP language doesn't help much with that, because the hard part is the definition (and tuning) of the algorithm, not the implementation.

>He's doing real-time audio processing. So, first, he's got performance issues. In that environment, it can be very helpful to re-use a buffer without having to allocate one (and without a garbage collector). For most functional languages, that's not a place where they shine.

It can be via streaming. The functional portion of the program does all the map and reduce operations on streams. Buffer allocation (and reuse) is automatically handled by the compiler. It's very similar to the way NodeJS hides the event loop. I'm not familiar with audio stuff but I'm sure there's libraries out there that allow you to do extremely efficient audio processing with high level language primitives.

>But, second, he's (probably) got problems like building echo cancellers. The hard problem is, how do you determine whether something is an echo? Writing it in an FP language doesn't help much with that, because the hard part is the definition (and tuning) of the algorithm, not the implementation.

Neural networks can be employed here to "recognize" echos. NNs are basically functional primitives with deterministic inputs and outputs. FP is remarkably appropriate here given that audio is basically a function.

> I'm not familiar with audio stuff but I'm sure there's libraries out there that allow you to do extremely efficient audio processing with high level language primitives.

I am extremely familiar with audio stuff, and I can assure you that there are no libraries out there that solve the problems you face when writing a digital audio workstation.

Doing non-realtime processing of file-based data? Of course! Oodles of stuff for that, and you can write it in whatever language you want.

Realtime processing of 240 tracks of audio feeding 64 channels of physical output, with hundreds of plugins? Sorry, but no, you can't do this "with a library" or in "a high level language". This, however, is what DAWs do.

Well, there are certainly use cases where purely functional languages (at least the ones that currently exist) aren’t the tool of choice… But in the grand scheme of things, like you yourself pointed out, these types of problems are not what the vast majority of engineers are working on.

I had assumed that initially you were referring to the difficulty of the “domain problems” when I wrote my comment.

I'm a Haskell fanatic, even wrote it professionally recently. I have also done a fair amount of small-scale audio programming, including trying to write a basic "soft" real-time-keeping app in Haskell, and it's not a great fit. I would probably choose Rust for anything that had to interface with hardware or tell time at this point, and that's largely because it would probably be easier for me than C/C++, which seems like the obvious choice based on ecosystem and prior art.

I think these kinds of comments made by FP advocates hurt FP advocacy more than they help, by implying that FP advocates can't see the forest for the trees. We are not in a world where the answer to ever programming problem is "throw FP at it!" We will never be in that world, no matter how awesome FP is, and I'm a huge fanboy.

Go take a look at the Ardour (http://ardour.org/credits) codebase and ask yourself in all seriousness how you could approach a project like that, what the tradeoffs would be, etc. and then ask yourself how serious any of your suggestions sound to someone like who you're responding to.

I've written haskell that interfaces with hardware. I've also done real time video processing with C++ which is magnitudes of more data then audio.

This by the way is 15 color cameras of extremely high resolution video so it's not trivial. At this level of data you can't expect real time performance without a GPU, so I go straight into CUDA for massive parallel processing performance.

You can think of each pixel in a video as 3 channels of sound, multiply that with HD resolutions and you have thousands of channels of sound flowing through the system at the same time in REAL TIME. 15 * 1080p * 3. Our pipeline also involves real time ML model processing and object tracking all written in C++/cuda.

This of course is across hundreds of similar edge devices all which feed relevant data into the cloud for further processing.

I am not an FP extremist. I work in systems programming which is pretty much the opposite of FP. I am simply stating a specific point here in that I think audio programming for FP can work. I respect the authors comment and I left it at that, even though I'm positive it's possible to write an FP language/framework that handles this specific use case of writing daws. It's basically a massively parallel streaming framework not so dissimilar to what I work on.

The actual processing is easily a functional issue, and the low level memory details can definitely be abstracted away. In fact, it's LARGELY what a daw is when you think people who write plugins.

What really pissed me off about your comment is the superior attitude. Like the "learn how to respect the experts NEWBIE". Seriously I hate shit like that. Who the hell are you?

Yeah, I'm not into expertism either. Even if I'm the expert (or maybe especially if I'm the expert).

Anyway ..

> In fact, it's LARGELY what a daw is when you think people who write plugins.

This is the problem though. Yes, that's how people who write plugins think about the host. But it's nothing really like the actual internals.

Comparing video and audio requirements is rarely fair and balanced. I tend to only be interested in realtime processing, not offline. Video involves far more data processing, but also missing deadlines is rarely noticeable by a human (unless you really miss a bunch of them). A single missed audio deadline (even 1 sample) is audible to any human. That's just a reflection of the different ways our eyes and ears work. So it's a bit of complex thing to try to compare the loads.

It's relatively easy to design and implement a parallel streaming framework, especially one without GUI interaction and realtime requirements. Most of the hard stuff comes from the user interaction side and the requirements it imposes on the realtime side. An engine that just takes a set of processing instructions and applies them as fast as possible to as much data as possible is significantly easier in a number of important ways.

> He's doing real-time audio processing. So, first, he's got performance issues.

FFTW, long the fastest FFT software available, is a program written in OCaml that generates C. You need to think more outside the box!

Another example: Copilot is a Haskell EDSL that generates a hard real-time C program for avionics.

You can write programs functionally and get your imperative performance too.

I definitely agree. I think somewhere along the way we went from solving problems to bickering about _how_ to solve problems. I feel there are now plenty of languages now that exist purely for aesthetic reasons (C++, Swift, etc) rather than having some deep technical reason for existing (C, Rust).

I guess I get it though, designing your own language is great fun and I’ve definitely made a couple myself in my spare time.

>I understand that I'm not the typical programmer these days, and I don't work on entirely typical problems, but I can't help but feel that sometimes the focus on "languages to help programmers" comes from programmers who just don't enjoy their work enough and/or don't have enough to do.

Indeed. There's a subset of programmers who like to solve "problems" and there's another subset of programmers who like to actualize their intent with minimal exposure to problems.

The later subset is the overwhelming majority both for what programmers themselves want and for what users want out of their programs. Solving "hard" problems is simply a niche desire.

I came across a quote the other day that is a far more accurate summary of the experience I've had:

"We don't do hard things because we like doing hard things; we do hard things because we thought they would be easy"

Certainly I naively believed that writing a digital audio workstation would be relatively easy. That was 23 years ago and I'm still writing it.