Programming in D – Tutorial and Reference | Hacker News Reader

Programming in D – Tutorial and Reference

ddili.org

120 pointsingve68 comments

Serious question: are D and Rust trying to be the same thing?

I'm not asking whether or not their approaches are different, but whether or not they're trying to solve the same problem(s). I get the sense that this is the case; they're both trying to be a "safe C(++)".

No, but there is some overlap. Generally speaking, Rust puts safety above anything else. D is more pragmatic, expressive, and generally has better performance (e.g. bounds checking is optional). I think D is a better choice anywhere memory safety and deterministic memory management are not the primary concern.

Bounds checking is optional in Rust too. And I've never seen bounds checks show up in instruction-level profiling of Rust code, due to the way iterators work.

Can you elaborate as to other ways Rust has worse performance?

> Bounds checking is optional in Rust too.

Could've sworn I've read somewhere the Rust maintainers stated that Rust will never have a switch to disable bounds checking. Has this changed?

> Can you elaborate as to other ways Rust has worse performance?

I don't know of any other specifics, but benchmarks usually show D outperform Rust (and often enough, everything else), e.g.:

https://togototo.wordpress.com/2013/07/23/benchmarking-level...

https://github.com/nsf/pnoise

Rust has iterators which avoid the need for direct indexing in many uses of contiguous-memory storage, and hence avoid unnecessary bounds checks (just like D's ranges). When indexing is required, bounds checking can be disabled individually by indexing with `get_unchecked` (or `get_unchecked_mut`): http://doc.rust-lang.org/std/primitive.slice.html#method.get...

In any case, when comparing language performance, it's generally good to avoid comparing backend performance, i.e. focus on the LLVM-based compilers in the pnoise benchmark. A GCC-based Rust compiler would almost certainly outperform the current LLVM-based one for that benchmark (as suggested by gcc vs. clang and gdc vs. ldc2). Of course, such a compiler doesn't currently exist, so if you care about getting your code to run as fast as possible right now, Rust isn't the best choice based on that benchmark, but Rust has several reasons that it's likely to outperform both C and D by default:

- better aliasing information (than both C and D), so compilers can optimise more aggressively

- lifetimes mean more aggressive stack allocation/ownership patterns can be used without risk (a bonus over C, and something that D solves via GC, which has its own upsides & downsides)

- the type system has been designed to be particularly good at concurrency & parallelism, so again this can be used more aggressively and in more situations (e.g. libraries can expose interfaces that allow mutating data directly on the stack of other threads without risk of data races or dangling pointers)

Rust iterators are a bit of a crutch IMO, until it's possible to return them unboxed; if you use them heavily you end up with types one paragraph long.

Yeah, it's annoying that one can't return them unboxed to be easily able to construct zero-overhead lazily-evaluated sequences, but that's entirely unrelated to their avoid-bounds-checks property.

Instead of iterating over a range of indices and indexing, use the slice iterator, it's basically a drop-in replacement. The former can either be done with a manual loop & addition (which is not at all lazy and definitely can't be returned, i.e. Rust's current iterators are strictly more flexible), or it is done with the Range iterator. The two lazy forms are basically the same:

  let v: &[T] = ...;

  let x: iter::Map<ops::Range<usize>, _> = (0..n).map(|i| {
      let elem = &v[i];
      // do stuff
  })

  let y: iter::Map<slice::Iter<T>, _> = v.iter().map(|elem| {
      // do stuff
  });

> Could've sworn I've read somewhere the Rust maintainers stated that Rust will never have a switch to disable bounds checking. Has this changed?

Not having a switch to disable bounds checking is not the same as not being able to disable bounds checking.

> I don't know of any other specifics, but benchmarks usually show D outperform Rust (and often enough, everything else), e.g.:

It's been years since I saw those benchmarks. Those are benchmarks of the built-in library random number generator. Rust uses a more secure random number generator by default than most of the other languages, so it will go slower. If you really want to benchmark random number generation, pick a faster but less secure algorithm.

> Not having a switch to disable bounds checking is not the same as not being able to disable bounds checking.

Not sure what you're getting at. Marking the entire program as unsafe is not realistic. Marking specific portions leaves a long tail of code that "rarely" runs, but still adds up.

> Those are benchmarks of the built-in library random number generator.

The levgen-benchmarks programs implement the XOR-shift RNG in both the Rust and D versions.

> but still adds up.

No, it doesn't. Bounds checks in cold code don't matter (and there aren't many to begin with). Even bounds checks in hot code frequently don't matter. Have you profiled Rust code and found bounds checks to be a problem?

> The levgen-benchmarks programs implement the XOR-shift RNG in both the Rust and D versions.

Oh, I see that they changed it. Still, that benchmark is from 2013 and doesn't compile anymore. It predates unboxed closures and many other performance improvements. And 15% is in the "different register allocation decisions will affect the outcome" range, where comparisons are fairly meaningless. The fact that D and Rust beat C doesn't mean that they're faster than C.

> No, it doesn't. Bounds checks in cold code don't matter (and there aren't many to begin with). Even bounds checks in hot code frequently don't matter. Have you profiled Rust code and found bounds checks to be a problem?

I've measured some D code yesterday and it was like you said, bounds check didn't change performance which was very suprising (this was for JPEG loading).

I can't reply to pcwalton directly for some reason.

In reply to https://news.ycombinator.com/item?id=10139287 :

> No, it doesn't. Bounds checks in cold code don't matter (and there aren't many to begin with).

OK, I trust your expertise on this. The "long tail" applies to optimizations in general. I don't know why the benchmarks are slower for Rust then, could be the difference in backends.

> The fact that D and Rust beat C doesn't mean that they're faster than C.

I don't know what's going on there, but some languages allow writing fast code easier than in C, e.g. SIMD vector array operations.

Here is the relevant thread:

https://mail.mozilla.org/pipermail/rust-dev/2014-March/00921...

In some cases, bounds checking doesn't make sense, e.g. in video games running on consoles.

> In some cases, bounds checking doesn't make sense, e.g. in video games running on consoles.

As if it couldn't be used to exploit and pirate the games, get extra lifes, buy in-game items without paying, automate bots...

"Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."

- C.A.R Hoare Turing Award speech, 1980

Amen! A couple of years back, I worked at a company where I helped maintain and improve a couple of programs written in C, and words cannot express how much I wished for a compiler switch to get bounds-checking, even if it was just for testing and debugging... Garbage collection/memory management was never much of a problem, but those nasty little off-by-one errors causing the program to corrupt the heap (which of course did not make it crash until a while later, so the location of the crash was usually totally unrelated to where the actual bug was located... sigh) drove me nuts!

This is the reason why D and Rust keep pointers and length packed together in array slices.

I think most languages other than C do this in one way or another. In idiomatic C++, arrays and raw pointers are frowned upon, I am told, in Go arrays have a fixed length that is part of their type, and slices carry their length with them, of course. And languages like C#, Java, Python, ... have done this for a pretty long time.

Of all the languages used to implement desktop and server software (these days, at least), C is pretty much unique in that it makes no effort whatsoever to help the programmer avoid or uncover these errors.

And don't get me wrong, I pretty much like C, even though I do not use it very often, but if there was one thing I could change about it, it is this. Given the amount of bugs that stem from this property of the language, I guess I am not alone. ;-)

Pretty much everyone in the video games industry is going to disagree with this. Performance is the primary concern there, period. Languages that reject such ideas in favor of others simply will not be used, given a choice. Consoles also only run signed code, enforce NX, and mprotect doesn't exist, so there is very little in what can be exploited.

> automate bots...

This has nothing to do with memory safety.

> Languages that reject such ideas in favor of others simply will not be used, given a choice.

That is the key issue, "given a choice".

I saw game developers being dragged into accepting C was replacing their beautiful Assembly, followed by game developers being dragged into accepting C++ was replacing their beautiful C.

You can be pretty sure if tomorrow Sony, Apple, Nintendo, Google or Microsoft said "you must use language X", they would do it.

So yeah, given the choice they can continue to lose money thanks to memory corruption exploits.

Signed code and NX don't protect against memory corruption. One can use ROP to work around it.

Again, bounds checking really doesn't show up in profiles. Iterators have none, and LLVM removes most of the ones that remain. It's really a non-issue.

> D is more pragmatic

Being pragmatic is a very questionable claim. Its exact meaning varies from time to time, and a property that a certain people consider to be pragmatic can be considered to be not pragmatic by another people. It's mostly meaningless.

For example, for embedded devices D is a no-go from the beginning. I'm not just saying the reliance on GC. D has a few ridiculous behavior that may cost significantly on the low resource devices, such as http://forum.dlang.org/thread/mr6bl7$26f5$1@digitalmars.com D is never pragmatic in this area. But of course, D may be pragmatic in other areas.

> expressive

D and Rust have different kinds of expressiveness. While D has much better compile-time metaprogramming, how do you express affine type in D? In Rust affine type is very fundamental and we can utilize it to define the exact meaning of our programs.

> and generally has better performance (e.g. bounds checking is optional).

What? How can a GC'ed language be faster than a non-GC'ed language? Besides simple numeric evaluation, D is generally more heap-allocation-heavy than Rust, so it can never beat Rust. After all, there's no point in Rust if it's slower than a GC'ed language. The benchmarks you linked are questionable too, as their results say D is faster than C. Are you really sure D can outperform C in general cases?

> How can a GC'ed language be faster than a non-GC'ed language?

Why should a GC'ed language be slower than a non-GC'ed language?

The evidence [0,1,2] seems more like: Given enough memory, GC is faster.

Non-GC means explicit deterministic free at specific points in the source code. A garbage collector has the freedom to delay the free, which means potential for optimization.

[0] https://people.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf [1] http://www.cs.princeton.edu/~appel/papers/cmljava.html [2] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.14.1...

> The evidence [0,1,2] seems more like: Given enough memory, GC is faster.

You really have to take Appel papers with a large grain of salt in 2015. Caches and multithreading have come to dominate nowadays, and assuming that all memory access is equal isn't reasonable.

Also, the first paper doesn't make the claim that GC is faster than manual memory management—in fact, manual memory management was faster.

> A garbage collector has the freedom to delay the free, which means potential for optimization.

People commonly say this, but I don't see how a malloc couldn't do the same thing if it actually helped. Malloc implementations typically don't do delayed reclamation because it doesn't help; if it did help, they would do it. But cache is king, and prompt reuse of space is most important nowadays.

> Being pragmatic is a very questionable claim. Its exact meaning varies from time to time, and a property that a certain people consider to be pragmatic can be considered to be not pragmatic by another people. It's mostly meaningless.

I disagree that it is meaningless. It's simply subjective. I've used D since 2006 and tried most new programming languages that appeared since, and it seems like they all tried to solve a specific problem (e.g. Go - concurrency, Rust - memory safety) while giving less attention to everything else.

> For example, for embedded devices D is a no-go from the beginning.

Existing D projects contradict that. They simply don't use the standard D runtime when it's not appropriate. Alternative runtimes exist. Rust users do the same thing: http://mainisusuallyafunction.blogspot.com/2015/01/151-byte-...

> D and Rust have different kinds of expressiveness. While D has much better compile-time metaprogramming, how do you express affine type in D? In Rust affine type is very fundamental and we can utilize it to define the exact meaning of our programs.

Can you provide a good example of affine types and how they help with expressiveness?

My view of expressiveness in D does not stop at compile-time metaprogramming. For example, here's an excerpt from one of my programs:

http://dump.thecybershadow.net/fb6c5a080c7a42ecd8d1f062b58a5...

> How can a GC'ed language be faster than a non-GC'ed language?

It is entirely plausible. For example, deallocation/destruction can run in a background thread, heap compaction removes heap fragmentation and improves cache locality, and of course throwing away all memory after the program is done executing is very practical if the program would otherwise do little deallocating overall.

> Are you really sure D can outperform C in general cases?

I don't know what's going on there, but "higher-level" languages outperforming C is not that uncommon, for example due to stricter aliasing rules or SIMD array vector operations, or simply due to language or standard library facilities being readily available (e.g. STL or associative arrays) which would be cumbersome or impractical to write/maintain in C versions.

> Can you provide a good example of affine types and how they help with expressiveness?

For example, let's say there's an HTTP server handler that must send each header line first, and finally send a body. So you should:

  conn.send_header("HTTP/1.1 200 OK");
  conn.send_header("Connection: Close");
  conn.send_header("Content-type: text/html");
  conn.send_body("Hello, world!");

In Rust, you can construct the API so that you cannot call `send_body` before any occurrences of `send_header`. So if you do:

  conn.send_body("I'm sent first!");
  conn.send_header("Content-type: text/html");

you will get a compile error. This greatly reduces the risk of having incorrect logic in your program. In this case Rust helps ensuring our intention: `send_body` can only be the last call of the request handler. I consider this a good example of expressiveness of Rust (and affine types).

That's very cool. Do you have a link with more information? I couldn't find anything when searching for "rust affine types".

DISCLAIMER: I'm not well-versed in type theory at all, so take my words with a grain of salt.

Unfortunately I haven't seen any comprehensive article that discusses affine types in terms of Rust. So I can't recommend a link to you, instead I'll try to explain them in my own words.

Affine types are those whose values can only be used at most once. They are implemented in Rust as ownership and borrowing. In my example above, `send_header` borrows the `conn` variable, whereas `send_body` takes the ownership of the `conn` variable. As you lost the ownership of `conn`, you cannot use it anymore, thus resulting in a compile error.

This is similar to C++'s move semantics, where you are not permitted to move the same value more than once. It is just that in C++ "move only once" is a convention, whereas in Rust it is enforced by the type system.

In Rust, if you pass an argument by value, you're automatically using affine types. You cannot use the passed variable anymore. To "borrow" instead, you need to opt-out using references. So, the hypothetical API I mentioned above would be written as:

  fn send_header(&self, text: &str) { ... }
  fn send_body(self, text: &str) { ... }

Note the difference of `&`.

Finally, all of these aren't just hypothetical. The exactly same design is used in Hyper[1], which is an HTTP library, and in nickel[2], which is a web framework in Rust. If you look at the API doc you will see there is no '&' in the self parameter. That means it's passed by value, using affine types, and the ownership is transferred, so you cannot use the original variable anymore.

I hope this helps.

[1] http://hyper.rs/hyper/hyper/client/struct.RequestBuilder.htm...

[2] http://docs.nickel.rs/nickel/struct.Response.html#method.sen...

> How can a GC'ed language be faster than a non-GC'ed language? Besides simple numeric evaluation, D is generally more heap-allocation-heavy than Rust, so it can never beat Rust.

I would be interested in benchmarks showing that. Do you have any?

Sorry, I don't have specific numbers. I was saying generally for all GC'ed languages.

D has lots of tools to avoid heap allocation. There is no reason why D should use more heap allocations than Rust.

A GC might only encourage you to use more heap allocations.

IIRC D has a GC that you can opt-out of but in practice it's difficult to use its standard library when you do.

I'm just repeating a critique that I read, I haven't evaluated them against one another.

Having a GC excludes you from some programming domains -- well controlled latency, bootstrapping/operating systems.

The GC situation is getting better:

- The language has the @nogc attribute to ensure at compile time that the GC is not used in a function or an entire module.

- The number of standard library functions that have the @nogc attribute is increasing by the minute. :)

- There is some effort to port Sociomantic's proprietary multi-threaded GC to D (theirs was written for D1).

> Having a GC excludes you from some programming domains -- well controlled latency, bootstrapping/operating systems.

Even without the GC, D is still a good "better-C" language. The GC didn't stop people writing bare-metal / operating system kernels in D:

  https://github.com/JinShil/D_Runtime_ARM_Cortex-M_study/wiki/1.1---Hello,-World!

  https://github.com/xomboverlord/xomb

> bootstrapping/operating systems.

Oberon, Modula-3, Singularity designers don't share the same opinion

Rust is interested in zero-overhead memory safety and data race freedom, while D doesn't seem to be. From a design perspective, that's a major difference.

Note that the safety benefits must of course be weighed against the number of moving parts in the type system you desire for your project.

> Rust is interested in zero-overhead memory safety and data race freedom, while D doesn't seem to be.

Of course D is interested in those concepts, simply not to the extent that Rust is. For example, D has:

- pure functions

- opt-in strict memory safety (@safe)

- sealed references (http://wiki.dlang.org/DIP25)

- transitive immutability

- shared types

Sealed references and transitive immutability are interesting, but they aren't nearly enough to allow most programs to be expressed without GC. You really need (a) the ability to store references in structures; (b) the ability to have multiple lifetimes per function; (c) propagation of immutability through heap objects and unique loan paths.

I am the author (of the book). AMA.

Ali

What development environment do you prefer for D? Old fashioned Vim/Emacs and the like, or any of the IDEs I see listed in the wiki (http://wiki.dlang.org/IDEs)?

I've been using Emacs for about 20 years now. Sorry, can't change that. :)

Ali

Why do you choose to write a book about dlang?

Can you give us us some godo area when dlang can be used?

Can we use dlang to build IOS/Android apps?

What is the advances of this language over go, python, java, ruby, rust?

I've discovered D six years ago after reading Andrei Alexandreuscu's article "The Case for D" in one of ACCU's publications. D was a gasp of fresh air especially for people coming from C++ like myself.

I used to frequent C and C++ forums on Turkish web sites and was aware of the need for up-to-date software documentation. I saw D as an opportunity to ensure that for once, Turkish documentation would not be behind the English ones. I started writing HTML pages without any idea that they would become a book later on.

Then I've decided to use the tool Prince XML (free for personal use) to see the pages in book form. I made that PDF version available as well.

I had never thought that my work would be valuable for an English-speaking audience. Then, Andrei and others convinced me that it would be a good idea to translate the book to English as well. I did that, added many more chapters, edited heavily (mostly by Luís Marques), and it worked!

As far as I know, the book is the first software documentation that was translated from Turkish to English, as opposed to the other direction.

D is a general purpose language. I don't know the status on the IOS/Android front. gdc, the D compiler using GCC back-end; and ldc, the D compiler using LLVM back-end can produce code for the platforms that those compilers support. However, don't take my word on that. I would expect issues.

D is a compiled language that is as low as C (and of course inline assembly) and as high as Python and others. One benefit over Go is the fact that D's template support is the best that I know of. When compared to D's C++'s templates are almost unusable; very clumsy.

Ali

Advantages compared to ...

Go: Generic type-safe highly efficient code. Can go lower-level like inline assembly. Can go higher-level because more meta programming.

Python: Faster. Static typing.

Ruby: See Python.

Java: Less boilerplate. More control over memory management. More meta programming. Natively compiled.

Rust: Easier. Garbage collection makes programming just as safe and is easier to use. Looks and feels more like C/C++/Java, if you already know them. More meta programming.

(of course, downsides of D are omitted ;)

Rust's safety is not limited to memory safety; you get data race freedom too. But the memory safety is the same (assuming you opt into SafeD).

Java is also native compiled. Language != Implementation.

OpenJDK is only the reference implementation. There are plenty of commercial implementations with AOT compilers.

Edit: did not write != properly

Yes, but it shows that the language was designed for interpretation. E.g. lazy static initializers is weird in AOT compilation.

It is not weird in functional programming languages.

I think you misunderstood me, because I used "lazy". It has nothing to do with Haskell lazyness.

In Java static fields are initialized when a class is loaded by the class loader. A class is only loaded when used. If anything (IO,scheduler,etc) changes the order in which classes are used, then initializers are called in a different order and their side effects as well. This semantic is mandatory by language definition.

In a Java AOT compiler this means every access to a static field must be prefixed with a (synchronized) check if the class is already initialized and optimizing it away is rarely possible. A JIT compiler does not even care, because it knows the class is already loaded.

For contrast, in C++ static fields are all initialized before main is called.

Actually I don't know how the current AOT compilers approach that situation.

Now with Oracle taking the steps to eventually make AOT compilation also a feature in the reference JDK, I suppose whatever issues might still exist they will be tackled.

Can you expand on the downside part of D?

Downsides compared to ...

All of them: Language is more complex and takes longer to learn. Less community. Less libraries.

Python: D cannot be as close to pseudo code, mostly due to static types.

Rust: Borrow checking looks like a nice idea. The question is, if the cost is worth it.

As a language, I think D is clearly better than Java and Go. Go has better tooling. For example, go-fmt is production quality, while dfmt is alpha quality.

> Can we use dlang to build IOS/Android apps?

It's a work in progress, but simple things are possible.

https://github.com/smolt/ldc-iphone-dev http://wiki.dlang.org/GDC/Installation/Android

FWIW, I installed Debian on my Android phone, on which I installed GDC. Building and running D programs just worked.

Deleted comment

He means the author of the book, not the programming language.

Thanks for making the book available in EPUB and AZW3!

Just ordered one!

What's the difference between D and C++? What's the difference between D and C#? What's the difference between D and rust? Who(which project or company) is using D? What's D is mainly designed for?

I like the syntax of D instead of rust, because it is much more like c++. The syntax of rust is weird to me.

This might be a silly question: what is D used for? I "scanned" the site fairly quickly and didn't grasp the general application idea behind it.

I used to use D for larger tasks and Python for quicker ones, like processing some text file and so on. One day I realized that I prefered using D even for those smaller tasks, where quick and dirty solutions would do. One thing that helped was that the standard algorithms (from the std.algorithm module) are really useful and compose very well, once you get to know them, and allow solving those kinds of tasks both quickly and efficiently.

Facebook has made a C-Preprocessor. D is great for such command line tooling. The standard library is not as extensive as Python's, but the situation is much better than in C-land. You can also use it as a scripting language.

D has a nice framework for servers (vibed.org), so you can write webservices (REST,JSON,etc). There is no great full-stack web framework like Rails or Django though.

GUI stuff works in theory, but I don't know any non-toy project. Games (OpenGL,SDL) are possible, but no serious project, yet.

Android/iOS does not really work, yet. Same with embedded, standalone, kernel stuff. ARM in general needs more work. For HPC it really depends on the use case (see dlangscience.github.io). Vector support and GPU integration is lacking. I don't know any image, video, or audio processing stuff in D.

Actually, that C Preprocessor was written by Walter Bright for DMC; just about every file has his copyright notice, and no Facebook copyright notice. I think they just translated it from C++ to D.

Its like C++, but "better"

Exactly. D is "C++ done right".

I am usually fairly skeptical of "X is Y done right", but in this case it seems to be true. C++ lovers might debate this, but D does take many lessons to heart that C++ programmers have learned the hard way.

I also suspect, that C++ is more of a result of design-by-commitee, whereas D is - to a substantial degree - the brain-child of a single person.

> I also suspect, that C++ is more of a result of design-by-commitee, whereas D is - to a substantial degree - the brain-child of a single person.

That's exactly how I see it. However, "single person" era ended long ago. Since 2007, others (especially Andrei Alexandrescu) had influence in the design of the language.

D is a general purpose compiled programming language. So, it can be used for almost anywhere e.g. C++ can be used.

what is D concurrency model ? does it have channels or classic threads ?

D is not really opinionated about concurrency and parallelism. It provides access to the OS mechanisms (pthreads). The standard library also has some task-based stuff. Then there are fibers (user-level threads), which vibed.org makes convenient to use. There is no definitive channel/queue mechanism provided by the standard library so far, but people have implemented them in libraries.

tl;dr: With D you use whatever is best for you.

Reply on news.ycombinator.com