Memory Safety in a Modern System Programming Language Pt. 1 | Hacker News Reader

Memory Safety in a Modern System Programming Language Pt. 1

dlang.org

80 pointsingve40 comments

It's amazing what strides languages are making in the realms of memory safety. I think D's got the right idea, to have pockets of zero-cost memory safety, on top of an overall architecture that embraces shared mutability's benefits.

A lot of languages are going this direction too:

* Cone is putting a full borrow checker on top of GC, RC, and single ownership: https://cone.jondgoodwin.com/

* Vale is putting opt-in "region borrow checking" on top of single ownership with generational references: https://verdagon.dev/blog/zero-cost-refs-regions

* Lobster is employing an automatic-borrow-checker-esque algorithm on top of RC for some brilliant speedups: https://www.strlen.com/lobster/

* Ante has some designs for lifetime inference, which I think will work really well: https://antelang.org/

* HVM is a runtime for Haskell that uses borrow checker semantics and falls back to clone. It's not shared mutability, but Haskell without GC is pretty cool! https://github.com/Kindelia/HVM

I particularly like D's approach because it's opt-in; we can use it in the areas of the program where we need performance the most, and prioritize flexibility and development velocity everywhere else, which I think is the right balance for most programs developed today.

It also means one can gradually ease into learning how to use static analysis and their more complex constructs (like return scope, mentioned in the article). One can be proficient in the basic language quickly, and then improve their craft gradually. In modern software engineering, this is a big win, in my opinion. Props to D for going this direction!

As much as I don’t want to complain about all the new low level languages (it’s really cool to see so much innovation!)

But I can’t help but feel a bit overwhelmed with options, and I know very few will ever catch on.

At least D seems to have some momentum, but I have fallen into the trap before of learning new languages that die.

I want to learn one of these, but I want it to last!

D is pretty safe choice in that regard. It has been around 20 years already and still marches on. Also it has many companies using it that depend on it. So while there is no guarantee it will ever be among top-10 languages, it's unlikely to die off anytime soon either.

For sure. Most of those are pretty young and experimental, so I'm glad D is adding something solid!

D is not popular but has a strong long-time following.

what about Rust ? Is rust not trying to be memory safe ?

Rust has a great approach to memory safety for plenty of use cases, but I personally believe that the future lies with languages that have a more seamless blend between shared mutability and static analysis. Going all-in on borrow-checked single ownership causes a bit too much complexity, in my opinion.

I would be very interested in any advances in the Rust realms that could address the added complexity! I haven't heard much coming from that direction. Polonius sounded like it would remove some edge cases from borrow checking, but it didn't seem like it would really help with the fundamental complexities from borrow checking.

Glad to see experimentation in this space, because I'm not quite sure Rust hits the optimal point in the design space. I think some of Rust's complexity is due to lacking what Cyclone calls regions and having to always deal with lifetimes, which can get complex: https://www.cs.umd.edu/~mwh/papers/cyclone-cuj.pdf.

Author here. Fire questions away!

What is the quality of support for these rules at the moment in external tools such as linters?

I'm personally kind of anarchic, because I use hardly any tools. Text editor and terminal, that's how I program. The compiler is my linter. So I don't know.

Used the wrong word. Meant "archaic", not "anarchic".

Side note, D has source code tools, this being probably the most famous one: https://github.com/dlang-community/D-Scanner . It's just that I don't use them myself.

The rules are enforced by the compiler not a linter.

Linting in a way that requires semantic analysis is not well trodden ground in D tooling.

deadalnix and myself are aiming to rectify this somewhat by ramping up work on sdc, which is basically a D frontend done right.

The idea with D is to not require a linter!

What do you think prevents D to become more popular? It seems Rust and Go captured some of the mind share of D.

Why isn't D more popular? (2014)

https://news.ycombinator.com/item?id=7692230

(25 points, 16 comments)

The short answer is, I really don't know. There are lots of speculation about that around. The only confident opinion I have is that good does not mean popular and vice-versa.

But since you asked, I think I should make my guess. I think people tend to use what life brings them to. Most will not voluntarily go hunting for the best language the Internet has to offer for them. So the languages become popular by forcing people to learn them, by having them to use them at their jobs or implementing important APIs in them. This means that what the project managers of big organisations have enthusiasm or business reasons to push tends to end up popular, not what is preferred by the common programmer. D is largely a free-time volunteer effort AFAIK so it does not get as much corporate influence than it would get if it were an academic or commercial effort. But again, that is just a gut guess, no research to back this up.

I think your second paragraph rings true.

I also think there is a cost to using a non-mainstream technology that is often overlooked or under-estimated. I also think the benefits of non-mainstream languages are often over-stated. In other words, when you really look at the details the cool/niche languages do often have notable drawbacks. I suspect that for most organizations the pro/con or cost/benefit comparisons of C++ vs. D don't often come out in D's favor and never did.

In the early days D chose to use GC which today, and for the past 30 years I have done C++ work professionally, has always been a dis-qualifier (mainly because of the extra memory overhead involved as well as the GC pauses). I have always looked at D with some interest, but never come close to formulating an argument that I could take to my company and suggest we start using D where we currently use C++. C++ almost uncompromisingly holds true to its backwards compatibility goals with C and the early versions of itself. C++ also almost always holds true to its "zero overhead abstraction" goals. D chose GC and replaced its standard library at one point. It is an interesting language for sure, but I think there are some things C++ has done better, from a practical use point of view, at least for the use cases where its use is still justified today, by being a more conservatively designed language.

The ironic thing is that the GC is one of the biggest productivity boosts available to a programmer.

C++ likes to prattle on about zero cost abstractions but the cost is nearly always to the programmer - and even then D does a lot of the zero cost concepts like templates objectively better than C++: Write a struct with N members in C++ (e.g. for autodiff), you're looking at a page full of recursive templates and SFINAE. With D it's orders of magnitude less code because it was specifically designed to avoid this kind of nonsense that C++ has conditioned people to think is worthwhile.

My guess (as someone who hasn't used D) is that it looks/feels too much like C for people who want a better Java/C#/etc, and too much like Java for people looking for a better C, resting in the middle, not really being appealing for either use-case.

GC makes it look like you won't do low-level things (e.g. pointer tagging or very-hand-optimized hot loops which might need to temporarily discard any and all safety), and being called a "systems language" makes it look like not very high-level either. Emphasis on "looks like" - not many will get to the point of actually looking for the options (especially when in a more opinionated language you don't even need to, as there either is an obvious way, or there's none).

Rust managed to stay distant enough from Java-likes (mostly due to manual memory management), and Go is pretty distant from C all things considered (but I'm not too sure why it's so popular. Probably more just being backed by Google?)

> GC makes it look like you won't do low-level things

Maybe this is true, I don't know, but to be clear other than the C runtime the language can be entirely implemented in D. You can do all these tricks and patterns easily and D and even have the compiler generate them for you (e.g. build your own loop unroller using metaprogramming if needs be), but I think people don't believe it to be true because of aesthetics.

Most languages never become popular. D is doing pretty well by becoming as popular as it has.

Novice question about D: From my brief reading, it looks like appending to a slice in D avoids the "invalidate all references into the old allocation" problem in a similar way to how Go does it, by having the GC keep the old allocations alive as long as there are references into them. Is that right? How does that interact with optional garbage collection? Is there anything like C++'s std::vector or Rust's Vec that D programmers reach for when the GC is off?

Not really. It's basically the holy grail for any future lifetime proposals for D.

The GCC static analyzer is actually clever enough to spot these types of bugs (memory ones anyway, not database handles) but this doesn't scale.

Yes, that is correct. There are all kinds of gc-less containers implemented for D, including the standard library std.container.array that is much like C++ std::vector. The downside of those is that they are not entirely memory safe.

EDIT: Even slices do not mandate using the GC. The GC will only collect arrays registered to it in the first place. If the slice was malloced, it is not registered and thus won't be collected. The built-in ~ operator allocates the new array with the GC though, so you have do something custom to concatenate slices in GC-free code.

"int* can point to the null address, but this is generally not a memory safety problem because the null address is protected by the operating system. Any attempt to dereference it would crash the program before any memory corruption can happen."

This is the most cursed definition of "memory safe" I have ever seen. I think the article would benefit from a link to https://dlang.org/spec/memory-safe-d.html explaining explicitly that this is actually what you intended to write, because it made me do a double-take and a Google when I read it.

Mind explaining why it's a bad thing to rely on the OS's built-in protections around nulls?

I'm familiar with a few arguments w.r.t. large structs and arrays running past the protected pages, but they seem pretty trivially solvable. Would love to hear why it's a cursed definition.

Most definitions of memory safety include not crashing.

What about bounds checking on array access, like most memory safe languages?

Having the program exit when a memory safety fault is detected is entirely safe.

Nope. Memory safety means no undefined behaviour. Crashing is defined behaviour. Indeed, at least in the D philosophy it is better to crash than to just do something and continue when a bug is detected. If that were not the case, assertions in released code would be total humbug.

Now, detecting such errors statically at compile time is still valuable where it is practical. But it's common knowledge that no language guarantees no bugs. It follows that a complete "no crashing" guarantee is not even desirable, let alone part of a memory safety guarantee.

Well for one the code might be running in kernel space, so a null pointer access could very well be fatal. There are also use cases that D supports (e.g. certain bare metal arm targets) where the all-bits-zero pointer is validly addressable and contains Important Stuff.

Those protections are there as last ditch safety measures. You shouldn't rely on them .

Reasonable answers, thank you! I wonder how D might address that.

I'm not sure about D's situation specifically, but one issue with null ptr dereferences in C, C++, and (unsafe) Rust is that they're UB by definition, so you're at the mercy of the optimizer if it can ever prove that you're dereferencing null. This has led to some interesting security issues in the past: https://lwn.net/Articles/342330/

An optimiser may not treat null-dereferencing D code as undefined behaviour. It must either error at compile time (if it can prove the dereference would be null no matter what), crash at runtime, or derefer the null address, which also results in a crash.

Exactly - the language isn't memory-safe. Instead it's effectively (implicitly!) relying on the fact that it runs on an abstract machine that has certain memory-safety properties. By contrast, a memory-safe language like C# is defined to be running on a memory-safe abstraction (the CLR, or more generally anything that faithfully implements the Microsoft intermediate language), while a memory-safe language like Rust either proves behaviour via the type system (no null pointers, no use after free) or inserts runtime checks (no out-of-bounds array accesses, panic on out-of-memory).

I would say "Bring the Burroughs system architecture back"... but we might actually be doing that with CHERI!

> This compiles and is fully memory safe, despite dereferencing those pointers in the same completely unchecked way they are dereferenced in C. This is memory safe because @safe D does not allow creating an int* that points to unallocated memory areas, or to a float*, for instance

Who on Earth wrote this? This is not at all true.

Isn't supposed to anyway...

Example?

I wrote it. Where's the problem precisely? I don't see it.

Reply on news.ycombinator.com