Hacker News Reader

>The question is whether we will end up using their efforts or whether we keep on going back to the same monolith kernels we have been using for the last thirty years or so. Whether it is NT, Linux, or any of the BSD kernels, they each have decades of history behind them.

Please just... stop. You are using terms that you clearly don't understand the meaning of. What is a "monolith" from your point of view? Because I can tell you that Windows NT is most certainly not a monolithic kernel. Its architecture is very much that of an impure hybrid kernel. You're using the term "monolith" as some sort of buzzword in a realm where that exact term already has a highly specific meaning.

>At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools,

Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite-the-same thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation.

>I could see linux evolve to a point where integrating drivers like that is both possible and common. Once that happens, it will be a hybrid kernel effectively.

Hybrid kernels have absolutely nothing to do with the language they are written in beyond the fact that they're written in it. Monolithic, hybrid, microkernels and exokernels are all terms to describe the architecture of a kernel.

>So one outcome would be a lot of Rust code running in a wasm sandbox on top of legacy kernels.

So then Rust and wasm isn't a part of the kernel is what you're saying.

> Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation.

Is there a reason that command line tools cannot be re-implemented in Rust and maintain GNU and POSIX compliance? If not then the fact that ripgrep is not compliant is irrelevant to the larger point that Rust command line tools could eventually replace existing C command line tools.

I think it's more that part od the attraction of rewriting the software is being able to upfate and improve the interface. If you want the old interface then the existing tools are usually perfecrly good.

>Is there a reason that command line tools cannot be re-implemented in Rust and maintain GNU and POSIX compliance?

No, there isn't, but what "re-implementations" that I know of don't even try to do that.

>If not then the fact that ripgrep is not compliant is irrelevant to the larger point that Rust command line tools could eventually replace existing C command line tools.

Of course it's possible. You can go ahead and re-implement them in BASIC even. The point is that it hasn't been done, and until it has been done then such a replacement will not happen.

At that point there are further considerations. The C linker is ubiquitous, and remains a strong, underlying presence even on Windows, the one platform where C stands the weakest. The reason that the C linker is so widespread is that while C has no standard ABI, the platforms have very strongly defined ABIs (e.g. systemv abi), and those ABIs are extremely stable and haven't had a single change in over a decade. Those ABIs are also very, very simple and the result is that writing any type of software towards a C library in any language is dead simple.

C++ at least has extern "C" going for it, which disables name mangling. Rust doesn't have that, nor does it have a stable ABI. C/C++ are the two most interoperable language out there, with an extremely mature and reliable toolchain. It's simply not justifiable to replace C/C++ with Rust as a systems programming language until this serious problem is fixed.

Then furthermore, a vast amount of effort has been put into these tools over the last years to ensure that they run on any *nix and any architecture, and even so they can be ported with relative ease. For example, how do you suggest implementing glibc in Rust? You might argue "why would we need the C standard library when we're porting things to Rust", and the answer is that if you want to make a move towards rust, having a C library (particularly the GNU one, a lot of software depends on GNU extensions) is of utmost importance until that goal has been achieved.

> C++ at least has extern "C" going for it, which disables name mangling. Rust doesn't have that

It does, it's called "#[no_mangle]", and together with an extern "C" fn declaration you get a non-mangled function directly callable from C.

I see, I admittedly didn't know that. Thanks.

I don't want to be defending Rust but your post has so many misconceptions. May I ask where you got all that information and what your background is?

> C++ at least has extern "C" going for it, which disables name mangling. Rust doesn't have that, nor does it have a stable ABI.

Rust has #[no_mangle] and extern "C" and those two guarantee that ABI stability you're looking for.

> Then furthermore, a vast amount of effort has been put into these tools over the last years to ensure that they run on any *nix and any architecture, and even so they can be ported with relative ease.

That's actually a breeze in Rust, including cross-compilation (which is relatively painful in C/C++).

> For example, how do you suggest implementing glibc in Rust? You might argue "why would we need the C standard library when we're porting things to Rust", and the answer is that if you want to make a move towards rust, having a C library (particularly the GNU one, a lot of software depends on GNU extensions) is of utmost importance until that goal has been achieved.

There's work towards that: https://gitlab.redox-os.org/redox-os/relibc

>I don't want to be defending Rust but your post has so many misconceptions. May I ask where you got all that information and what your background is?

I am primarily a systems programmer and write software for the petroleum industry in my country. I work with C++ and in some (rare) cases C, and I've been doing that for the past 4 years. I have my own WIP hobby unix-like microkernel project written in C as well.

>Rust has #[no_mangle] and extern "C" and those two guarantee that ABI stability you're looking for.

I admittedly did not know this before it was pointed out to me.

>That's actually a breeze in Rust, including cross-compilation (which is relatively painful in C/C++).

It really isn't. Have you looked at rustc's compiler targets? It's not a very long list. Support is improving, but there's still a lot of key areas that are completely missing. To my knowledge it supports x86, ARM, MIPS and POWER. That's not a long list.

>There's work towards that: https://gitlab.redox-os.org/redox-os/relibc

That's good, and is genuinely what's necessary to put Rust in key components.

Based on [1], also full support for s390x (IBM mainframes), as well as support for SPARC, WebAssembly, and asm.js as targets only (you can't run rustc itself on them, but can cross-compile to them). RISC-V support was recently merged into master (not sure why it's not on that page). AVR and m68k (seriously) both have actively developed ports which are living in forks until they're ready to be merged.

Anyway, most of the work of porting Rust to new platforms is not in rustc itself, but rather porting LLVM to target that platform. This is bad in a way and good in a way. It's true that LLVM doesn't have as extensive a list of supported targets as GCC. However, for people developing new architectures and OSes, LLVM is often the go-to choice as the first compiler to port. For example, WebAssembly, eBPF, and AMDGPU are all supported in upstream LLVM at present, but not in GCC. On the other hand, RISC-V got GCC support first – but lowRISC also actively developed an LLVM backend, which is now upstream. [2]

Of course, C benefits from having multiple implementations; even on platforms that only LLVM has a backend for, you can just use LLVM's C compiler (Clang), whereas there's (currently) no Rust compiler with a GCC backend. I'd love to see that change in the future. Still, I'd say Rust target support is already pretty strong when it comes to the (embedded) platforms most people are doing new development for, and rapidly getting stronger.

[1] https://forge.rust-lang.org/platform-support.html

[2] https://www.lowrisc.org/llvm/status/

It’s not on that page for the simplest reason in open source: nobody sent in a PR.

I’ll look into it tomorrow, thanks.

>> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools,

> Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite-the-same thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation.

That ripgrep isn't GNU or POSIX compliant is indeed a distinction worth making, but certainly not in this context. You're taking on a very pedantic interpretation of "re-implementation" here that doesn't really matter to the general point. Specifically:

> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, etc. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

None of this requires any of the "re-implementations" to be 100% strict bug-for-bug compatible tools.

This general confusion comes up so often that I have a FAQ item for it: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...

> a distinction worth making, but certainly not in this context.

It is a worthwhile distinction, because the "re-implementation" talked about is often materially different, to the point that it no longer adheres to what little standardization there is.

So realistically we are talking about a new product, or a radical evolution of an existing one. It and everything above it also have to change.

I commented because ripgrep was specifically brought up as a counter-example to the notion "At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C." What specifically, about ripgrep, removes it from consideration as evidence that Rust programs can operate in the same domain as C?

Consider this. If instead of

> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, etc. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

they wrote

> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing popular command line tools, albeit with different interfaces that lack POSIX compatibility. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

Other that the words being more precise for pedants, does the central point of the statement change? No, it doesn't.

I believe seeing people re-implement libraries or popular command line tools is a behavior so common in rising in popularity programming languages that we might almost describe it as the default.

That's what nebulet is doing!

> wasm is also going that direction

Genuine question: why would WASM in the kernel make any sense at all?

Security. WASM is sandboxed by design.

Many things are "sandboxed by design." Few things are true sandboxes.

I'm very skeptical of the idea of throwing out virtual memory and process isolation to depend instead only on software sandboxes.

That and performance. System calls have a lot of overhead when running in user space. You can dodge that when running as part of the kernel.

It is incapsulation of code into a sandbox. You can do it by moving code into user-space and use hardware implemented sandbox, or you can do it by making your own sandbox in software.

Hardware implementation runs code faster (kind of "executes more instructions per second"), but switching from user-space into kernel-space and back are costly. The costs are very high, high enough to projects like "user-space TCP/IP stack" start popping up: it is all about avoiding switching to and from kernel-mode.

Software implementation of sandbox in the kernel allow costless (or near costless) switching of context, but runs slower. It is tradeoffs, and not every one piece of software would benefit from moving into kernel-WASM, but some would do.

It's sometimes useful to run "arbitrary" code in the context of the kernel. For a concrete example look at the Berkley Packet Filter[1]. In it's original usecase it allows packet filter expressions to get compiled into a bytecode, which in turn get verified & executed in a kernel-space virtual machine. This is useful because it allows the user to implement arbitrary expressions, in this case firewall rules, but have them executed in the context of the kernel. When you're trying to filter millions of packets per second, or instrument operations that are measured in sub-microseconds, avoiding that context switch matters.

[1]: http://www.brendangregg.com/ebpf.html

> it will be a hybrid kernel effectively.

What do you mean by this exactly?

> A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

Depends on who you ask. Hipp on Rust in 2016:

Rewriting SQLite in Rust, or some other trendy “safe” language, would not help. In fact it might hurt.

https://blog.regehr.org/archives/1292#comment-18452

> drivers written in Rust

If you have to write inline assembly, and generally bypass Rust's security features and use "unsafe" code, why wouldn't you just use C?

Because it is a big difference to have 100% of the code unsafe or just a set of visible code regions marked as such.

Hyperbole, or do really believe every line of C is unsafe, in the literal sense, by default? Pray tell me what is the difference between

  // safe, tested, C
  ...
  // potentially dangerous C deemed thus for whatever arbitrary reason

And

  // "safe" Rust
  ...
  // "unsafe" Rust

Static analysis.

Life experience since I touched C for the first time in 1992.

So yes I do belive, given the quality of the code bases I have seen in production.

And as discussed at Linux Kernel Security Summit 2018, which videos are freely available for you to watch as well, even with the rigorous process to accept patchs into the kernel, CVEs keep increasing.

68% of them were caused by memory corruption, something mostly unique to C and languages that are copy-paste compatible with it.

Then another good chunk was related to UB, something that even Linus already ranted a few times.

So yeah, Safe C is an oxymoron, unless we are talking about a new variant with safety turned on by default, where warnings are dealt as errors and everyone uses static analysers that break their CI builds on error.

Because C has big flaws (signed/unsigned automatic conversion, arrays without length) that Rust doesn't have. That said, you're right that I find Rust "story" that you should wrap unsafe operation behind safe APIs as something easier said than done.

Regarding SQLite, here's the official tune (https://www.sqlite.org/whyc.html#why_isn_t_sqlite_coded_in_a...):

----

All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. Some preconditions that must occur before SQLite is recoded in Rust include:

A Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.

B Rust needs to demonstrate that it can be used to create general-purpose libraries that are callable from all other programming languages.

C Rust needs to demonstrate that it can produce object code that works on obscure embedded devices, including devices that lack an operating system.

D Rust needs to pick up the necessary tooling that enables one to do 100% branch coverage testing of the compiled binaries.

E Rust needs a mechanism to recover gracefully from OOM errors.

F Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.

If you are a "rustacean" and feel that Rust already meets the preconditions listed above, and that SQLite should be recoded in Rust, then you are welcomed and encouraged to contact the SQLite developers privately and argue your case.

----

I follow the Rust story from afar, so I don't know the status of every one of these points.

B and F (FFI and speed) are covered.

C and E (embedded story and OOM recovery) is being actively worked on.

A will improve with time (for example, I suppose that the Rust 2015 dialect will settle at some point, feature-wise, once Rust 2018 is out).

D (branch coverage of compiled binaries) is on the radar, but not addressed yet.