Porting a Golang and Rust CLI tool to D | Hacker News Reader

Porting a Golang and Rust CLI tool to D

pingfrommorocco.blogspot.com

117 pointsnurettin55 comments

> For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function.

I want to note that you can call actual C functions from D without much extra effort.

Here’s a program that calls getpass: https://glot.io/snippets/fqf04b6wg2

Thanks! I added it in this PR: https://github.com/azihassan/hashtrack-client/pull/11

Oh, oops.

I forgot to edit my comment to add that, for some reason, it doesn’t seem work on macOS. But if that’s not a problem for you, then I’m happy I could help.

Edit: I just read the code, and it seems like you found out about that before I could warn you. :)

I didn't know actually, I only did it so that it would also compile on Windows (which I haven't tested yet tbh). Good to know it would still compile on macOS

I'm surprised that the rust binary is 3.5 times bigger. D is already templated to the nth degree, what more is rust doing? On top of that D links in a whole GC whereas rust does not?

Rust will generate a lot of guarding code similar to what ada does.

The only "guarding code" that I can think of Rust generating would be for bounds checking on array access, which isn't going to be any significant contributor to binary size.

When it comes to generated code, I can imagine the panic unwinding code being a noticeable contributor to the size of a Rust binary, but since D has exceptions I imagine the D binaries have comparable code for unwinding.

Yes, it will generate Exception unwinding and bounds checks and godbolt seems to agree. Anything else you can think of? I think D is less strict when optimizing out exception code.

Exception unwinding is not particularly cheap on the compiler although I think it's in the sense that it muddles the CFG and the register allocation rather than inserting huge amounts of code. The LLVM codebase has exceptions turned off for those reasons and binary size (it plays a role at their scale) although its quite hard to compare exceptions Vs error codes or monads for obvious reasons (LLVM's coding standards are rightly or wrongly a C++ sleuths dream with their custom dynamic casts etc.)

D also generates significant amounts of said code (unless you turn it off, obviously)

One feature D stole from Ada that is an absolutely inexcusable ommision in a modern programming is contract programming. Every time I start a new python script I end up writing helpers for function pre and postconditions because otherwise I can't rely on anything in the language's syntax.

Invariant clauses in D (e.g. this integer in my struct must be less than 5 when the struct is in a valid state) are an absolute godsend although they do not aid compiler optimisations despite some people saying otherwise (it could have changed but last time I checked the invariant does not make it's way to LLVM or GCC)

how is that different than in c/c++ simply adding asserts at the start and end of functions?

It's the same but it's a different switch to turn it on and off.

Having separate syntax is also very useful because it's much easier to see through my usual spaghetti to find what the code is actually supposed to do in the abstract (i.e. if the string should be a valid email, check it in the in {} clause rather than in the middle of the function)

> One feature D stole from Ada that is an absolutely inexcusable omission in a modern programming is contract programming.

Incidentally, Walter Bright considers contracts to be a low value feature of D that he considered removing. [0][1]

[0]: https://youtu.be/p22MM1wc7xQ?t=1h34m30s

[1]: https://digitalmars.com/articles/hits.pdf

Rust, like Go, statically links everything by default, and only uses dynamic linking if you ask it to. Is D using dynamic linking here?

Rust is statically linked by default, not sure how D works. Rust still includes some debug features that can give you a stack trace. Size will also depend on the compile options, if LTO was enabled, how much generics were used, how much string printing was used (print is a macro in Rust that generates code at the callsite), etc. Most, if not all, of these can be resolved.

See https://github.com/johnthagen/min-sized-rust for aiming for small binary size.

This article shows a 93% size reduction, 2.7m -> 200kb, with some relatively simple modifications https://www.collabora.com/news-and-blog/blog/2020/04/28/redu....

Here is someone using Rust for a 3kb demoscene program https://www.codeslow.com/2019/12/tiny-windows-executable-in-....

D is usually dynamic although I try to be static for the most part. IIRC The runtime is linked statically (or not at all) but Phobos is dynamic by default

Rust compiles seem to blow up pretty easily, e.g. I was cleaning out my SSD the other day an I noticed that a rust GUI framework (Iced?) was sat on about 2Gb of dependencies despite being significantly incomplete (but very promising)

> Phobos is dynamic by default

That depends on your distro. The packages from dlang.org all do the static link config out-of-the-box, but some distro packages dynamic link by default instead.

Given the size of that file, it looks static linked. It dynamic links libc, pthread, and a couple other OS-level things, but all D libs including druntime, phobos, etc, are statically linked.

Might just be difference in dependencies.

Oh boy I definitely understand you, in fact I built a Rust CLI tool to solve exactly that ;) https://github.com/tbillington/kondo

The 3kb demoscene example is just to open a window, not a full, self-contained 64k demo in 3kb. Is the program basically calling the Win API through unsafe functions? I am trying to learn Rust, and I am wondering how it might actually be used here or more beneficial than C in this case if it is all done "unsafe"? I am still comparing Zig and Rust to see which I am going to go with. I don't think I'll ever give up C though.

That post is, but https://www.codeslow.com/2020/07/writing-winning-4k-intro-in... is a 4k demo.

And yes, a lot of it is unsafe, because it's calling into all of that stuff directly, and doing pointer arithmetic, etc. Check the post, it's good, the Conclusion section talks about this specifically.

> I am still comparing Zig and Rust to see which I am going to go with

This is really puzzling to me: in what context are you working to have hesitations between Rust and Zig? One is a production language, the other is an exciting experimental language mostly written by a single genius. Go for Zig if you want to have fun, and Rust if you want to actually build something. It's like hesitating between Kotlin and Idris!

Haha at: written by a single genius. And so true.

For a low-level, systems programming language it really comes down to Rust and Zig if you are migrating from C. I like Idris, but I don't see it in this category, nor Kotlin, although Kotlin Native perhaps? Yes, Zig is exciting! I find it easier than Rust and I have given both equal time so far.

Most likely largely because of this:

>Rust downloaded a lot of crates, but that's probably because the Rust version of the code has more features than mine has. For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function. That's one of the many things I don't have in my code.

How can D use so little memory despite sporting a GC?

    Go Memory usage: 13mb
    Rust Memory usage: 9mb
    D Memory usage: 7mb

Being around for longer means more optimizations

D has been around for longer but Rust and Go both have (I'd guess) on the order of 100 full time staff each. D has something like 3 at most.

What? Rust has less than 5 full time paid devs. And with the fall of Mozilla it's shrinking. But yeah external contributors summed up are a big human resource

But how many companies outside Mozilla use rust in a capacity to contribute to the compiler, standard library, and crate ecosystem? e.g. Now rust is in chromium if Google want stuff fixing they can pay for it to be done in the same way that Linux is developed mainly by 9-5ers not working directly for the Linux project specifically. D has no paid developers at all full time IIRC (Walter Bright is living off his empire empire I think).

Go has Google's direct backing, which is more what I was pointing towards.

According to the recent blog post that was on the front page, Chromium is still exploring the possibility of using Rust. They’ve yet to commit.

I guess it's a case of quality over quantity.

D developers are all experts in the art of crafting compilers and language design.

9 women can't make a baby in a month

As a occasional contributor I can attest that most people on the D forums/GitHub are very sharp, although (I do not like Go) Rust's team seem excellent too (albeit slightly more academic?)

The rustc developers are excellent as well.

Never said the contrary.

Imagine what they can do in 19 years.

Go GC has some smart people working on it too.

And Go, by the virtue of having less features, has to allocate quite more. Eg: Go doesn't have lazy ranges / streams. Dynamic dispatch using interfaces has to allocate memory on heap.

Also, I expect D standard library to be optimized for Non-GC use case at least a part of it, because that's one of its use cases.

Compiler's escape analysis matters too. I'm told Go's escape analysis is not very sophisticated.

Last, it may be case of missing features in D version as a comment mentioned.

Post-hoc: I'd say that it's easier to communicate between 3 people than between 100. And, so, it's more likely that 3 people will be able to maintain a system where they each have a pretty clear understanding of the whole thing.

I would not have predicted D would be better than Rust in any measure if those are the number of maintainers.

I'd guess that the larger team must be able to do more. "Faster alone, further together".

What do you think a GC does? It will use more memory but the design of the library using it pays a huge role. D's standard library is intended to be as lazy and efficient as possible within its API (i.e. most ranges are lazy and work on the stack to start with)

Also, quick tip for anyone writing scripts with a GC or malloc/free: turn the collections/free off and just allocate until you finish the script. The OS collects the memory and you save a few seconds.

I agree that this is helpful sometimes, but reading this causes flashbacks to deleting files and killing processes manually while watching htop, hoping that a serveral-hour-long process won't die in its last few minutes

Oh yeah. Definitely small batch jobs only, it's a good trick for parsing too if you know (MEASURED) it's a small % of the task but very memory intensive or fucking the cache up.

Another answer for this particular case is that the D program has less features. Quoting the blogpost:

> Rust downloaded a lot of crates, but that's probably because the Rust version of the code has more features than mine has. For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function. That's one of the many things I don't have in my code.

Thanks, that could be part of it. I added some of the missing features, namely colored terminal output and getpass (thanks to the reply above), but the memory usage didn't fluctuate much. The D version is still missing websocket support for the "watch" command, but none of the clients use it in the benchmark.

Lisp had a GC in the 1960s when 7mb of disk was hard to come by, let alone ram.

Lisp also had a reputation for wasting memory. The joke, from back when 4MB was still a lot of RAM, was that emacs is an acronym for "eight megabytes, all continuously swapping."

How can D use so little memory despite sporting a GC?

My guess is that you have experience with platforms such as the JVM or CLR (dot NET) and expect garbage collected languages to use up a lot of memory.

The JVM and CLR use generational/copying GC. These types of GC sacrifice memory to get efficient collection -- the more memory they use, the more efficient the GC [1].

Both Go and D use non-copying GC which has a lower memory overhead.

I am not sure about D, but Go also uses escape analysis so that data structures are allocated on the stack instead of the heap when possible which can help to reduce garbage and hence memory usage.

[1] The principle is similar to the one described in this paper where the GC time is independent of the amount of garbage: https://scholar.google.com/scholar?cluster=48058747733581794...

> I am not sure about D, but Go also uses escape analysis so that data structures are allocated on the stack instead of the heap when possible which can help to reduce garbage and hence memory usage.

D sometimes does this but for the most part it depends on how you write the code. strings, large arrays, and class objects are typically GC'd but struct instances and small arrays are often not.

The code in the blog post appears to mostly use small stack memory.

The only weird thing i see in the blog post is 30 second compile. Yikes, this should be more like 3 second from scratch, max - it must use some heavier-than-necessary dependencies.

It seems to include botan, which is a fairly large C++ library. You can be assured that by far most of those 30 seconds are linking. The compile step itself for D is going to be done in <1s for the code shown.

I've never really dived into D, but why was it never more popular? It seems like it was posed to take off, but never quite did.

D had an unfortunate early history. For a long time being a one man project, surviving through Python2 to Python3 akin transition and slowly maturing during all these years. Many top D features were added along the way but were not available from the start. Also in the beginning D focus was to be a better C++ with GC which is not something that excited C++ crowd who smirked as soon as they heard about GC and never actually approached the language. It is only later that it became clear that it is a separate unique language. I think D just grew and matured naturally into something very solid. D has a small but a very skilled team of seasoned C++ veterans and language designers. I dare to say, I am yet to see better forum answers than on forum.dlang.org both on the account of the language and programming in general.

Interesting, thanks for the synopsis. :) Sounds worth looking into.

For a really good background reading on D language landscape, check this paper by its authors:

https://dl.acm.org/doi/abs/10.1145/3386323

...go Lang AND rust? Why would anyone do this to themselves?

If only there was a link to a post with more context somewhere, like in the first sentence on the page.

That doesn't resolve the question. It's a very odd combination of languages to decide to sling together arbitrarily.

Two statically compiled languages that get a lot of attention for writing tools in doesn't seem that odd choice to compare?

I mean you can statically compile a lot of languages and literally every language is used to write tools.

Go doesn't even use a stack compatible with the system abi and has probably the worst FFI of any language I've ever used, it has a very, err, unique type system. I mean there's a shit ton of great stuff too (compile times, great test patterns and pretty good tooling, the http stack and string manipulation and green threading is fantastic), I don't have it out for go! Meanwhile rust has a completely different memory model and type system and concurrency model, has probably the best FFI of any language outside of maybe some lisps and cpython and probably D (never used it), and they've taken forever to role out production ready http client/server stacks (though I think this is finally smoothing out!) I just don't see why they're tied together except a) static compilation and b) they emerged at roughly the same time.

Rust and go are very, very different languages aimed at completely different niches.

Reply on news.ycombinator.com