Hacker News Reader

It has become increasingly clear to me that C's influence on the systems landscape is even stronger than it would initially appear. Our ability to move beyond what we have now is not merely hampered by the fact that we have to create an entire systems infrastructure, it is hampered by the fact that we always get sucked into having to support C shared libraries in addition to that, and in general that requires certain things to be true of your language, which pulls you into the C local optima. And now you've failed to write a systems language that has moved fundamentally beyond C.

In short, to be able to run a C library means that you language must hand over total control to that C library, who may then dance all over memory, if it chooses. You can't build a system that enforces any further constraints. And you don't have to be an "academic weirdo" language anymore to want to be able to enforce constraints; even something like Go has a significant runtime that you need to work with properly to get the benefits of go, and a C library simply expects to own its thread for as long as it wants, do whatever it wants to allocate memory, use whatever resources it wants, not be scheduled in any way by anything other than the OS, and just this immense laundry list of requirements that all looked OK 30 years ago, but it's increasingly clear that to get to the next step of systems design, some of those are going to have to be modified. And C is not the language that will allow this.

The only language I know that manages to have radically different semantics that C at the deepest levels, yet can still talk to C libraries without (much) compromise of those semantics, is Haskell.

Somehow we've got to escape from C-type linking dictating requirements to the deepest levels of the semantics of the new language, or we're not going to escape from the local optima we are in right now.

Why are so many people intent on taking C away from us?

The fact that you can run everything from Erlang to Python to Lisp to a JVM on the same machine and under the same OS is thanks to C, not in spite of it!

C exposes the machine at a low level, which means that you can innovate in the language/VM space without having to beg the runtime gatekeepers to please implement your new experimental semantic. Anyone can generate machine code that will run directly on the hardware -- I recently wrote a JIT that parses Protocol Buffers 2x the speed of any existing parser. Don't take away from the the ability to do this!

Higher level VMs like the JVM can impose further constraints -- this is already possible today. But those VMs can compete for popularity in user-space without imposing limitations on the kinds of VMs you can write directly on the hardware.

"The fact that you can run everything from Erlang to Python to Lisp to a JVM on the same machine and under the same OS is thanks to"

Turing completeless, not C.

"you can innovate..."

We have. We've innovated for 40 years. It turns out we've learned some stuff since then, and it's time to move some of those innovations down to the lower parts of the system. Sorry, that means bumping C. But I assure you, I will celebrate C and put it on a pedestal and remember it fondly, even as I'm shoving it out the door and glad that it is no longer the constraint for all future systems.

It's not that it sucks, it's that it's basically used up. An immense amount of problems with modern computing basically boil down to having C at the lowest level of the system, and we aren't going to fix them until we get something else at the lowest level. The hardware model that it embraces is simply not appropriate for building the networked future on.

> Turing completeless, not C.

Be serious. Brainfuck is Turing-complete: are you suggesting that other programming languages could reasonably be implemented on top of it?

Almost every single VM or programming language implementation is written in C. What are people going to write programming languages in when you take C away from them?

> We have. We've innovated for 40 years. It turns out we've learned some stuff since then, and it's time to move some of those innovations down to the lower parts of the system.

So your opinion is basically that it's time to stop innovating, because now we know the best answer. What do you have to say to the fact that I wrote a JIT two weeks ago that effectively improved the state of the art in network protocol parsing? How is that not part of "the networked future?"

Your arguments that C is holding us back are vague and unconvincing. People write VMs all the time. Code that runs on these VMs can be as insulated as you want from the underlying system. The entire web is built on JavaScript which doesn't know a lick about C -- how would that whole ecosystem be any better if C wasn't underneath it? I argue it would be much worse, because the radical performance improvements of the last 4 years (which came from writing JITs that target the bare metal) would not have been possible.

You can pry address spaces and machine code from my cold, dead hands.

"Be serious. Brainfuck is Turing-complete: are you suggesting that other programming languages could reasonably be implemented on top of it?"

Be serious yourself. With that sort of hostile characterization, I have little interest in following up in a dead conversation. You don't appear to have spent even a second trying to understand what I'm saying before leaping to a nonsensical strawman. (And I am well aware that understand != agree. But you didn't take the time to even know what you disagree with.)

"So your opinion is basically that it's time to stop innovating,"

No, it's time to resume. C is not where the innovation is.

I spent a lot of time trying to understand what you were saying, and because it was unclear to me I asked very specific questions, none of which you answered:

    - what will people write language implementations in, if not C?
    - why don't you consider my JIT an innovation worth supporting?
    - if C is a barrier to innovation, why is the JavaScript landscape so thriving?

My point about Brainfuck (that's a language by the way, not a slur, in case that wasn't clear) is that Turing-completeness says almost nothing about the systems-level capabilities of the language. Turning completeness means that a language can calculate Pi: it doesn't mean that it can open a file or send data over a network, and it says nothing about efficiency. So when you argue that Turing-completeness is what makes it possible to run lots of different kinds of VMs on the same machine, it's hard to take such an argument seriously.

"So when you argue that Turing-completeness is what makes it possible to run lots of different kinds of VMs on the same machine, it's hard to take such an argument seriously."

Turing completeness is why you can run a JVM on top of a C environment. It means that any computation that C could do, the JVM could do. It means that I can run a JVM on top of any Turing Complete environment. The JVM could be ported to a modern day Lisp machine. The JVM could run on a Russian ternary machine. C does not get "credit" for enabling the existence of the JVM, because the JVM could run on any Turing complete substrate. You have the causality backwards.

"- what will people write language implementations in, if not C?"

The new systems languages will be written in themselves! What do you think C is written in? "Bootstrapping."

"- if C is a barrier to innovation, why is the JavaScript landscape so thriving?"

Wrong question for what I'm talking about. Javascript has no innovation of interest to systems level programming, it's just a thin layer over C. I'm talking more like something like Go, which is hampered by the fact that it can't really use C libraries properly because Go can't grab and schedule C routines properly. Or how E [1] is really hampered by the fact that it basically can't work properly in a C environment, its whole schtick really only works if it's down to the OS level. The entire point is that I want to replace the foundation, so it can do things that C basically can't. Like, write a secure operating system, which C is incapable of, despite immense amounts of effort.

Basically, why is UNIX still the preeminent OS? Yes, it was good, I know that, there's a reason why it's the only survivor of its time period, but where's the capabilities-based system? Where's the thing we can't even think of because we're too stuck on C? Why is one of the preeminent kernels of the day, the Linux kernel, still adding an average of two security vulnerabilities a week, the same ones, over and over? Why are buffer overflows still coming out in 2011? Why do I still get segmentation faults in 2011? Why are we building the very foundations of our system on a language that all but mandates such failures, when we know how to not do that?

C.

As for any response you might have as to why C is still worth it, my reply in advance is that every one of those things is room for "innovation", much of which has already been done.

[1]: http://www.erights.org/

> Turing completeness is why you can run a JVM on top of a C environment. It means that any computation that C could do, the JVM could do.

I'm genuinely confused: I thought this was the argument you called a strawman when I argued against it before.

I've already rebutted this argument twice: Brainfuck (http://en.wikipedia.org/wiki/Brainfuck) is a Turing-complete programming language, and yet you categorically could not implement the Java class libraries on top of it. For example, the java.io.FileReader class could not be implemented on top of it, because Brainfuck does not have any API for opening a file.

Even in cases where you can implement one language on top of another, it may not be efficient to do so. For example, Adobe Alchemy runs C/C++ code on top of Flash/ActionScript at a 2-10x slowdown compared with running the C/C++ directly on the hardware. Implementing a JVM on top of Java appears to be 4-9x slower than implementing it in C/C++ [0] (and that's only comparing two interpreters: the difference is far greater when the C/C++ implementation generates machine code directly).

C gets credit for efficiently supporting a whole host of different VMs with divergent GC schemes and synchronization primitives. That is what makes C special, and why it continues to dominate in system space.

[0] http://portal.acm.org/citation.cfm?id=1294345

> Turing completeless, not C.

C's mix of portability, simplicity, flexibility, and expressiveness made it the ideal choice for developing operating systems that had to run on a wide variety of architectures.

Perhaps a simple example of where C gets in the way would help clarify your case.

This is, indeed, why Rust originally compiled only to shared libraries (that also worked as executables). We ended up distinguishing shared libraries from executables for performance and OS integration reasons, but shared libraries are still key to the design.

Is there something fundamental in Go's symmantics that prevents dynamic linking? Or is this just a tooling issue?

Personally, I think I might avoid a language that defined how the end product could be packaged, that just seems outside the range of what a good language should focus on.

Go definitely bucks the trend in modern web age language development,for starters it generates raw binaries. That alone puts it in a different category than just about everything else of significance that has been developed over the last decade. Rather than coming out with a full auto-completing IDE from the start or a plugin for netbeans or eclipse, they've got a lightning fast object code compiler. The natural evolution of these things is to get raw binaries working, get the langauge pretty much stablizied, then add some default libraries and tools to make shared libraries and get dynamic linking and PIC code type stuff working and the language shouldn't have to actually change for that to happen, you just change the tools and create different ways to compile stuff. I'd expect shared libs to come along.

> I believe that so-files must have been

should, no? Considering you can not — as far as I know — compile to shared libraries in Go.

You can compile your own packages to statically-linkable libraries. Code sharing is what libraries are all about. Now, if you want dynamically-loadable libraries (written in Go), Go's tools do not support that. (You can, however, link to dynamic libs written in C and that linking happens at runtime.) Static linking is precisely what prevents dependency hell.

I'm somewhat hijacking the discussion with this, but with storage getting cheaper, why do we need dynamic linking anymore?

Plugins that don't require developer tools to relink your executable.

And when you have a security bug in your static library (let's call it "library Z", just for kicks), what do you do? Is it enough to just let the software know? What about big vendors, whose products might have 42 different copies of Z kicking around in the source base.

What if they miss some? Wouldn't an architecture that allows the distribution to define and replace Z on its own be valuable?

> "Wouldn't an architecture that allows the distribution to define and replace Z on its own be valuable?"

In theory, sure. But how many more years of practice do we need before we accept that it doesn't work that way in reality? That any change in Z is as likely to break apps that use it, as save them some hassle?

Clearly I was too subtle. It, uh, wasn't actually a hypothetical: zlib had known exploits that bit Microsoft and others for years because they had cut and pasted the library into a zillion places. Linux distros just updated.

There's no theory at work here. Static linkage of common components is a security vulnerability.

And I wasn't saying it never works. I was simply saying that it doesn't work that way regularly, or even often.

so that you can update a dependency without relinking every binary that uses it.

Dynamic linking was never about cheap storage, it was about RAM. Except that dynamically linking libraries into your program almost always uses more memory than statically linking your program. So there is no point. Most systems (including Unix, which did not have dynamic linking until late 80s/early 90s) worked just fine without it.

Here is a good list of reasons why you should never use dynamic linking:

http://aiju.de/rant/dynamic-linking

FUD. Dynamic linking is about saving on RAM. Nobody ever said it's faster or "secure" therefore the entire post is mostly about made up agruments.

And dynamic linking does save a lot of memory. That's one of big wins for Google's Dalvik: standard JVM is unable to share code - that's why all JVM processes are memory hogs.

Looking at my process map right now. Gnome-panel has 20MB in resident memory (not all of it is code, mind you) - and 13 of them are shared with other processes. Unity-window-decorator shares over half of its code with others. Just for the kicks open your process monitor, have "shared memory" column open, and sum up all those numbers - those are megabytes you would have lost without the shared code.

It is also about CPU as well. Moving sort() implementation in and out of L2 cache is expensive, better to have a single instance of those opcodes do sorting for multiple processes.

Sorry, but our operating systems and programs we run on them are mostly composed of shared libraries. The debate of either they're good or not is largely pointless unless we migrate to different OS designs.

"Sorry, but our operating systems and programs we run on them are mostly composed of shared libraries. The debate of either they're good or not is largely pointless unless we migrate to different OS designs."

What do you mean when you say that the operating system is dynamically linked? I don't even think Linux kernel modules count. I run OpenBSD, and all the base packages are statically linked. Many applications (esp. graphical ones) are dynamically linked, but there's not many reasons they have to be. The biggest one is bloatware libraries full of buggy crap you don't want or need. But dynamic linking isn't a solution for that problem.

"What do you mean when you say that the operating system is dynamically linked?"

By that I mean this: grab a debugger, attach to a random process you have running. Pause and look at the current stack: you'll notice that the code you're looking at resides in a so. And everything on the current call stack is all shared code, loads of it, sandwiched between the kernel at the bottom and a thin layer of hosting executable on top.

Here's another way to look at it: http://paste.ofcode.org/diZdtuH8uPs2UBWHEvTYWh (try to ram all that code into gnome-panel itself and ship with next Ubuntu - see what users will tell you)

One more time: shared code is absolutely essential. Only "one-process-per-machine" datacenter approach can do without. That's why there is no Java on the desktop.

Except that dynamically linking libraries into your program almost always uses more memory than statically linking your program.

If your metric is the memory usage of one arbitrary program. If your metric is the total memory usage of the system, it's likely that dynamic linking is a win because many applications can reuse the same pages in memory.

Most systems (including Unix, which did not have dynamic linking until late 80s/early 90s) worked just fine without it.

That's neither-here-nor-there; most systems worked just fine without operating systems until those were invented, too.

I know that shared libs are prominent and a better support for them would probably help Go's popularity. (Though the idea behind shared libs is somewhat flawed.)

But it rocks to deploy a Go app by just sending the binary to a dozen of servers and never have to worry about any dependencies on shared libraries.

Savvy Win32 devs used static linking whenever they could to minimize deployment hassles way back when. :)