The state of low-level system programming
erik-poupaert.blogspot.com
erik-poupaert.blogspot.com
1. M:N scheduling. We wanted high scalability with hundreds of thousands or millions of tasks, which means we need the capability to have our own scheduler. OS threads aren't lightweight enough for this, especially on Windows, where over 80% of our Firefox users are. Having our own scheduler means we need our own ABI. We would like to make the scheduler optional, but getting it working was a higher priority than making it optional.
2. Safety. On many platforms (basically, anything but MSVC on Windows), the C stack is not bounds-checked; either there is no guard page or the guard page is unreliable because you can overshoot it. This is not acceptable for our Rust code, since stack overflows are a frequent source of exploitable security vulnerabilities for us. So we do our own stack management with a custom ABI that performs stack checks at every function entry, freeing us from this source of pain on all operating systems. Again, this is planned to be optional, but it was more important at this early stage for us to demonstrate advantages over C than for us to duplicate its ABI exactly.
Many libraries, however, would prefer direct interoperability with C to M:N scheduling or stack overflow safety. So we plan to provide a mechanism for Rust code to be called from C without having to set up a scheduler or perform stack checks. It's not implemented yet, but it's very much part of the current design, and I would be thrilled to see it happen.
Now technically you can compile a special python which can handle Go extensions, which is the wrapper mentioned in the second answer, but you can't distribute python modules that ask people to install "this special python" to run (which will reliably work with all their other installed code..)
I would like to see a "go init" function in a C API, that would let you interop the two whichever had the `main()`, since that would let you link with all the other c-implemented scripting languages (python, ruby, etc) which would mean that Go extension support could be added to the main cpython trunk as a compile-time option.
Rust issue: https://github.com/mozilla/rust/issues/3608
Go issue: http://code.google.com/p/go/issues/detail?id=2790
I wonder who will get there first.
Of all the new languages billed as "systems languages", Clay is the only one that excites me.
It can also "quack like a duck" (to put it in TFA terms).
https://github.com/jckarter/clay/wiki/Clay-for-C---programme...
Later the author says, "If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already."
I think the reason that scripting languages exist is because of limitations in the established languages. What we think of as "a scripting language" is defined by the gap. If someone else could be a system programming language without the gap - maybe it doesn't need to be accessed.
The traditional unix platform is (1) C and a compiler for it, with access to the unix standard library and (2) some sh or csh derivative, and access to the likes of sed and awk.
Shell gave people a glimpse of what was possible. But shell has poor access to the unix API, and hits the wall quickly once you do anything other than very basic tool development.
So these things called scripting languages started to pop up. People wanted easy access to the unix API like C. They want easy string manipulation like shell. And they want access to functions, maps and lists without having to think about memory allocation.
If you start from the shell and awk and work into that gap you get perl - the canonical scripting languge.
TCL inhabits a similar space to perl. It's also "a scripting language". But it started at the opposite end. TCL starts as an extension language for C.
So I can understand the author's emphsais on close interaction with C. But I don't think they're important if you can take away the shortfalls they're compensating for.
I hope Go will mean I don't have to think about these things. That I'll be able to get a standard build and a go compiler/runtime and have immediate access to lots of power. No more headaches from C build tools or reaching around in the murky water underneath my scripting language.
Now if someone produced a better solution to that it would be useful...
And also say you want performance, then using getters and setters sucks compared to direct struct access, as every access needs a non inlineable function call.
I've had much better luck with exhaustive testing with Klee and related libraries (e.g. for testing multi-threaded programs).
For whatever reason, they map better to how I think about problems, and I've found it easier to develop new code that is testable within those constraints.
You could write tests as needed and when a test fails write types that make that failure as close to compile time as possible.
Another approach is to just write C code, either embedded in ATS files or in C files that are linked into ATS. Then slowly convert C to ATS as you feel comfortable.
For transparent access to compiled C libraries, external C functions need to be declared in a 'vapi' file and although these were already provided for a large part of the GNOME ecosystem, documentation on vapi syntax was nearly non-existent. Object-orientation and standard libraries are integrated with Gnome's GObject system and Glib libraries which I found unfamiliar and usually better documented in C at the time.
[0] It is possible to write a systems language with these features, but low-level systems stuff is not what Vala was designed for.
As far as I know there is no Vala runtime or garbage collector. Memory management is by automatic reference counting but there is support for manual C++-style pointer syntax with new and delete operators.
I agree that Vala was not specifically designed for 'low-level systems stuff' but I see no reason why this is not technically possible. The Vala compiler is a source-to-source translator to C and in my previous experience worked fairly well interfacing with fairly low-level C libraries like MPI and OpenCL. I assume that a kernel writer does not require much more than a limited subset of the language that supports C pointers, structs and arrays.
And yes, a language with which you cannot dload and call any shared library you wish is, how to say, handicapped language.
NodeJS, Java to name a few. Look what they do due to inability to call, say, libpq.so.)
But of course, re-monkey-patching everything from scratch by amateurs coders is much better idea. And, you know, you could create an entire "standard" with hundreds of pages of specification and then push your bloatware - JDBC.
How handicapped is C++ considering it subsumes C, can use C right at the source level, has an inline FFI, and yet C++ libraries can't be called by C because C++ provides type-safe linkage.
Advanced languages are advanced because they improve on C and this inevitably means they must go beyond C. This is why such languages require setting up an environment, usually by providing main().
Even if you could set up an environment with a C function call, which is trivially achieved by replacing main with mymain(), you would simply be starting a foreign environment, not providing a library of functions which could be individually called by C.
My system Felix (http://felix-lang.org) is specifically designed to live with the compromises of C/C++ compatibility, but you still cannot put arbitrary Felix code in C callable functions because Felix provides an advanced environment supporting, for example, fibres with channels. Fibres are scheduled by returning control to a scheduler so they cannot be nested inside a function which puts its return address on the machine stack.
If you Felix write code without using features requiring advanced support that code can be made into a C callable library (both static and dynamic linking is supported).
Felix is probably better at C/C++ integration than any other language except C++.
There are general ways to call procedures - there machine instructions for this. ABI defines how exactly parameters and return values should be passed - using, registers or stack or both.
As long as you follow the rules there is no difficulty in calling everything you wish.
The whole idea of JVM as something disconnected from reality^W OS and hardware is simply wrong, and all the confusion is the consequence of that premature decision.
Of course, slogan "you don't have to know" always wins.
You can run Python 2.5 on lastest Jython, then you have access to JVM libs and the JVM libs to Python libs 2.5 compatible.
Calling JVM from CPython is actually possible: - with pyjnius from kivy fame built with cython it is still in development, AFAIK it is missing the possibility to call Python code from Java code aka. submitting a Python "callback" to a Java method, I think it is called IoC pattern and Hollywood principle «Don't call me, I call you». jnius is used in Kivy to make it possible to call Android libs. As an example, I created Python bindings of Blueprints with it. Blueprints is an abstraction library for graphdb like Neo4j, OrientDB, Titan, available here https://github.com/Printemps/python-blueprints/. Tutorial: «JavaClass = autoclass('path.to.JavaClass'); JavaClass.JavaMethod(python_object)». Those bindings are merely Pythonification but they are gotchas in types conversions.
- There is also jpype which looks like an established solution used by Neo4J fame to make it possible to embed Neo4J embedded into CPython.
I choosed jnius because it's faster; Those are funny benchmarks in log scale: https://docs.google.com/spreadsheet/ccc?key=0Ajc7MnpSjCokdGt...
And yes, it would crash when you use threads. Since all those C libraries pretend that threads don't matter. Guess what, our Java program has 500 threads and feels fine.
btw, you still must call libc.so and libpthread.so
As for pthreads, guess what, they are also bad decision, as long as they are not isolated as processes supposed to be, and crash of one thread messes up rest 499.
People who did some research, like developers of Erlang, found that neither one-process model or threaded-threaded process model is even applicable when you need a reliable system.
So, again, jvm.exe is mere yet another .exe.)
From the Rust doc linked: "The primary motivation of extern functions is to create callbacks for foreign functions that expect to receive function pointers."
That said, the problem with D: not enough marketing plus the two compilers / 2 libraries issue.
And it's close to C++ complexity enough to not let the scripting guys feel welcome (like they do in Go).
This is a myth and part of the vicious circle here. I'm not a C hater by any means but the reason hardware vendors have historically included it is because there are free compilers that are nearly C enough that they can include and they have to include something. C is just a portable assembly, you can almost write a simplistic compiler in pure yacc and more than a couple of these C compilers hardware vendors produced weren't much more than that. Software guys then wrote in C and the circle continues, intel has a great C compiler now and in fact all the best optimizers are from hardware vendors but that wasn't always the case. There was also a time (quite possibly it's still that time) when you'd buy some customed up MIPS or PowerPC chip and get an ancient version of gcc that they'd hacked to support their shit and forward porting to a modern GCC wasn't always a straight forward thing to do. They have to provide something and tools usually isn't their business.
Now so much is built on the C non-runtime that Go, Ada, Rust, anything with any runtime looks completely backwards to a lot of system folk. What's funny, you look at Linux and there is a fairly small hardware dependent directory of code where the register poking happens, it's really not that much code but its called from all the drivers and such, and that's the holdup. It's not like that stuff makes up 50% of the code or anything.
X86 provided efficient call/return protocol instructions including display management, specifically targeting Pascal/Modula style languages. Microsoft Windows 3.1 even mandated this calling protocol. However old style C didn't use this API because it doesn't work with varargs and C doesn't have nested functions.
Instead Intel observed real code and optimised their instruction set to speed it up, locking everyone into lame languages like C. Attempts by Intel to break from this, for example with x64, have failed (x64 has two stacks).
Similarly segmented architectures, which implement extension of the Harvard machine model, have been largely unsupported in favour of linear addressing von Neumann machines, primarily because that's what C and Unix use. Interesting to see Microsoft adapt their OS designs based on this trend.
To support more advanced features in a language often requires only a few very low level primitives. This is why Linux kernel doesn't need much assembler, to do things like stack swapping. Control exchange is one of those things completely missing from C.
Also, when talking about systems programming, we can't just limit our scope to linux - there's hundreds of other kernel projects, and embedded systems which don't have the luxury of a massive portable codebase already - they only have a basic C compiler, library, debugger.
Using a different language doesn't eliminate C, because chances are, the compiler you're using for that language has been written in C, because it's all the manufacturer supplied. Compiler vendors contribute to this cycle by expecting their compiler to run on any platform if they write it in C.
I was suggesting that if we want to liberate ourselves from C, we need the hardware vendors on board too - since they're where the cycle begins.
Of course, you don't need the vendor specific compiler if you have a cross compiler, and can build a different language compiler from another platform - however, if you look at what's available for doing that, there's nothing other than GCC - which is why new languages are unlikely to replace C as the defacto systems language, unless they can replicate that functionality.
Also, most computers have a GPU which can't really be programmed in straight C.
That way, you could have your low level language use pointers, nullables, inline assembly, goto and other "dangerous" things that are the most efficient ways to do stuff. You wouldn't want to have that language crippled like modern C++ though, you would still want modules, a well planned object oriented syntax, a better solution to name mangling, and better readability.
You could have an interpreted Java/C# esque applications language that would mostly be similar to modern Java/C#, maybe with auto, a better namespaces implementation (in C#, a class defined in two places with the same namespace and signature have situational behavior that gets really messy, for example). You would use this language for the application layer.
Scripting could just be an autoboxed dynamic language like Python. Maybe have a whitespace significant and insignificant dialect. I'd like to see dialects of such a language, where the context it runs from dictates available modules. Since it would be permissions based, it could function as a shell, web, and glue language depending on where you use it.
I do agree on this sentiment though. The wide base of languages in modern computing makes the whole thing a lot messier than it needs to be, and the lack of any full stack that really ties together nicely and is well thought out without the smudges would be really nice.
I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.
Perhaps this article misses the point a bit. For several decades we have been using scripting languages to glue together sharp pointy C[++] tools (sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from within (VB/elisp approach).
More recent practice, on both mobile and server platforms, has been to abstract away from C[++] with a middle layer of memory-managed, compiled C-ish goop - JVM, .Net, Objective-C, whatever. This is the level where people are doing everything from amazing high-availability big data things to amazing little bleep-bloop handheld things. I think the answer to "what's the state of low-level systems programming?" is "not much is going on, but damn, there's a lot of fresh stuff happening in mid-level systems programming."
Go and Rust are late entrants to this party, and probably exist more as a hedge against Oracle litigating Java/JVM into the ground than anything else.
Google at large barely uses the language. All the examples they've given are of peripheral stuff, like a load balancer for MySQl for YouTube and an even simpler replacement for an aged and cranky C++ system for Google Downloads. Nothing to write home about, and no replacing C++/Java for core systems at all.
Hadoop held the world record for sorting a Terrabyte for a while. If sorting large amounts of data as fast as possible isn't a system's programming task, then I'm a three-headed monkey.
Before writing all these BS, maybe he should do a little research first?
Here's Rust (which is not even released yet) being capable of being called from C. It's a feature they are actively work on:
There's a subtle difference between being able to call back into Go/Rust code from C (where the main function is in Go/Rust) which both languages allow now, and allowing a call to originate from a C program, which neither currently do. The author is talking about the later.
I would agree the article could go into more detail about what's currently possible, and why it's not enough. But calling BS, I think, is going a bit too far.
That one is also under consideration by the Rust developers.
Even a screenshot would be better than this.
Please move on to some actual html website that just works or rehost the content elsewhere before posting it here. Thanks.
Disabling JS usually have the effect of doing away with such nonsense but here that is no option.
This time it was worth the hassle since it was an interesting article, but I definitely prefer it when JS is optional.
Google's blogspot allows you to do normal html blogs (maybe now they've made this their "default", but still, it's OP's choice to have it this way)