How I Came to Write D
drdobbs.com
drdobbs.com
Where I fall down is that without encouragement I'm not a 'completer-finisher'. Walter has that great ability to grind out the finished quality product. I love D for its expressiveness and speed, here's hoping it wins out.
- being motivated by people telling you you can't do it.
- writing what you need instead of what other people say the world needs.
- having a weakness for getting into arguments with people who are "wrong." :)
- being inclined to work solo, and being uncomfortable giving up control but realizing at some point that you have to cede some.
I'm saddened by all the stories I see posted here and on Reddit where people are discouraged from doing things by negative remarks from others.
"It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat."
The critics often seem to be saying, implicitly, "_I_ could do it better". It's a claim of superiority. I guess it's beguiling, because criticism of this form is so much easier than genuine creation. I think many of the critics don't consciously realize this, since they've never created something of great worth, and have no idea how difficult it is.
I've known very accomplished people who could be haters. But usually accomplished people seem to be appreciative of creative projects, and understanding of flaws (while wanting them eliminated). I think it's because they understand just how difficult it is to do good creative work. They know that someone can bring immense gifts and effort to bear on a project, and still have it go awry.
That's great to hear that you didn't let the negativity stop you.
I was recently told "If you're going to get bogged down trying to get the lexer/parser to work, you're not ready to work on a full blown language/compiler." Hopefully my story will go the same way as yours.
Not that anyone has ever followed my advice, anyway :-)
Anyway, thing that helps me do stuff - focus on getting a piece of program right. Then move onto another part.
Another good saying, all good software starts simple and evolves into a complex solution. So maybe start with Lisp syntax and evolve it into something more complex, when you're ready.
[1]: http://www.drdobbs.com/architecture-and-design/so-you-want-t...
People usually have good reasons why they say negative things. It's worth understanding the negative sentiment.
For example, victims of certain types of fraud are often warned repeatedly by friends and family that they are going to be defrauded. Some still go ahead and lose staggering sums of money.
It's important to really listen to the naysayers, if only to understand the risks. Then you can make an informed choice as to if you'll try your hand at something that's potentially costly.
I wanted to learn these arcane incantations so that I could fill this black void with my own Universe. Whether it be CRT, LCD, LED, or Plasma, I have been trying to find a way to bring worlds of light to the darkness of a computer display ever since. Yet, as the scale of my endeavour dawned on me I realised that it would be better to spend a couple of years creating tools that I was comfortable using than rely on C++ as others recommended.
Unfortunately, my estimate was out by an order of magnitude.
Some twenty years later I have only managed to wrangle the process of research and (re)design to a point where I am finally ready to write a comprehensive specification of my multiparadigm "live" programming language and its alternative document-centric graphical user interface only after a self-imposed deadline made at the beginning of last month - otherwise, I'm sure I would still be amusing myself exploring endless "rabbit holes".
Despite this delay, I feel that my project is stronger as a result. I really didn't know enough about computer science when I started and I have found that I need to grok OOP and FP to know that they aren't appropriate for my needs. I was denied the opportunity to study the subject at school and had to travel to Foyle's in London to buy obscure computer books for years before unlimited broadband became affordable. Admittedly, I was paranoid that I would find myself several years into programming my videogame only to discover that "high productivity", "spare me the details", programming language was weak in some respect and could not accomodate the retrofit of some unanticipated, but very necessary, feature. Hence, a lot of the work I have done has been defensive: trying to create a future roadmap that specifies how concurrency and parallelism would work even if I plan to leave the implementation of these features until much later.
Hopefully, it won't be too long before I am using an integrated suite of development tools (created with my own language), to build my own procedurally-generated intergalactic MMORTSFPSRPG (Massively-Multiplayer Real-Time Strategy First-Person Shooter Role-Playing Game), or "adventure" if you prefer. Without my language/tools I very much doubt I would be able to complete such a grandiose endeavour unaided, and I very much prefer working alone without social expectations or professional deadlines - despite how much of my disposable income it has cost.
If I had started implementation sooner I would have made something naive and half-baked. I did not have the benefit of a formal education in Computer Science and I probably wouldn't have attempted a project of this size if I had known all of the work that was involved. Walter Bright wrote a compiler by himself and Paul Woakes wrote Mercenary unaided and Elite was made by just two people, with David Braben on graphics and Ian Bell doing the trading (okay, three if you count the novel included in the box by Robert Holdstock), but that is still just 1% of the staff of a Ubisoft game like Assassin's Creed employing a whole bunch of artists, animators, scriptwriters, voice actors and composers for an estimated thousand man-years to create its content-rich high production values - all of which can be circumvented with procedural content generation as in No Man's Sky (initially, just four developers), and supplementary user-generated content such as or the seven million user-created levels in Little Big Planet, or about seven thousand competitive Halo 3 maps on the Forgehub community website.
Rather than waste mine and everyone else's time "doing the sensible thing" and writing another Tetris clone, I've gone and jumped straight to what I wanted to do, mindful that I will need productivity boosting tools in order to make it and that to write those all by myself I will need a highly productive exploratory programming language and that in order to make THAT it would help if I knew what the hell I was doing and did PLENTY of preparatory reading so that I didn't go in to it uninformed.
One question I have for Walter, when you were writing your compilers (both back in the day, and for D), how much did you just try to figure out yourself, and how much did you rely on literature to guide you? I tend to find it more fun to just try to do it myself, and only turn to literature when I'm really stuck. But obviously there's a bunch of valuable things to be found in books and articles, and they can save you from bad decisions or going down fruitless rabbit holes. As someone who's very experienced in language implementation, do you tend to invent your own algorithms, or rely more on academic and industry literature?
I learned how to do data flow analysis by taking a short course put on by Ullman and Hennessy.
I learned about GC from the famous GC book.
I learned a lot from people who started out saying "Walter, you damned ignorant fool! [...]"
I've learned a lot about things that sound like great ideas but just don't work, from bitter experience. I try to pass this stuff on to the next generation, but they are usually determined to make the same mistakes. Oh well.
As someone working in the area of compilers... Do you have that advice written down anywhere?
Is there other books, papers, authors you would recommend related to compilers, optimisations, runtimes (vm or otherwise) and programming lanugauges?
I'd recommend the Dragon Book, but it's a bit dated these days.
Anything by Agner Fogg is good for optimization.
There's always the D forum, where we're always going hammer and tong over every detail of how things work.
I almost never heard/read something about Rust from a D developer and the other way around, it's a bit like they live in different parallel universes :)
There was an interesting discussion on a golang thread about Go vs C++ vs D, with great insights in conversation between Andrei Alexandrescu and some of the Go guys (can't remember where to look for the link right now...), but it only made clear the fact that Go and D target very different niches and it was basically an apples to oranges comparison... but D and Rust would be an interesting comparison, they really are in the same "zone" but the communities seem very different and each seems not to know or care about what the others are doing (there was a Rust guy on that thread mentioning the way Rust uses the pointer types system for memory safety and the others were something a long the lines of "can a type system really be used for that?!" that clearly suggested the two groups didn't share their ideas a lot).
EDIT+: this might be the link to the thread I was referring: https://groups.google.com/forum/#!topic/golang-nuts/8k59Rgke...
Currently I spend more time on D forums than Rust ones, and to be honest given my type of work I can only use JVM/.NET/C++ languages, as customers have the last word.
However as language/compiler geek I do follow many discussions.
Trying to avoid a flamewar here.
D has a GC and follows the school of thought from Cedar, Oberon, Modula-3 and so forth, where it is assumed you can have a systems programming language with GC, which also allows for manual allocation when required to do so.
Still room to improvement there in terms of performance, though.
Rust leaves GC to the library, at the expense of a complexer type system as a means to allow the compiler to reason about automatic memory usage.
Both provide very powerful and modern abstraction mechanisms.
Which one is better? I think it is a question of use cases.
Rust has an overriding emphasis on guaranteeing memory safety and correctness, and when the programmer must resort to potentially unsafe code it isolates the unsafe portions of the source code so that they can be more easily audited by hand and thoroughly tested. It specifically incorporates safety features with zero runtime overhead (with the exception of array bounds checking, which can be turned off on a case-by-case basis) in order to appeal to C++ programmers who need to work close to the metal.
D also emphasizes greater safety than C++, but not to the fanatical degree that Rust does. D smooths and streamlines the experience of writing C++-level code and favors expressiveness, with especially impressive compile-time programming abilities.
The most important conceptual division between D and Rust is that D guarantees memory safety via garbage collection (which can be disabled, at the cost of losing memory safety), whereas Rust guarantees memory safety via compile-time checks (which makes Rust code less convenient to write, since it forces you to think about the lifetimes of your data, but provides the benefits of memory safety in environments where GC overhead is unacceptable or where the code must run without an accompanying runtime present).
Both languages share many similarities to C++, but this is merely convergent evolution. D deliberately began with a very strong C++ influence, whereas Rust began as more of an OCaml-like language and gradually evolved toward C++ due to the pressures of designing a production-grade systems language. The result is that Rust favors ideas more prevalent in functional languages: immutability-by-default, algebraic data types, everything-is-an-expression, and so on.
A good oversimplification might be something like: D was conceived by people who were tired of how clunky C++ is. Rust was conceived by people who were tired of how unsafe C++ is.
This stood out to me as the #1 difference between me and successful people. I always notice it in every article or bio of someone successful. Something that has never, and would never happen to me at any point in time, but in the bios, it's always there:
The first person I mentioned my idea to suggested a lunch.
1. He had a colleague to speak to about this issue. This indicates he and the colleague had a relationship conducive to speaking about such problems, and was in a position to speak to him in the first place.
2. That colleague was engaged enough to want to hear more.
3. The colleague knew a "local C guru" and in turn had enough of a relationship with that guru to arrange a lunch between the three of them.
All of these things point to an environment that helps foster this kind of work. It shows that the author was hard working enough already to be in that position.
I don't know if others interpreted his statement to mean the same things, but that's what I have.
Even after I wrote the compiler and was shipping it, a different colleague asked me one day: "Walter, I have a friend that needs a C compiler. Which one do you recommend?"
Me: "Why, mine (Datalight C), of course!"
Colleague, laughingly, "Not yours, Walter, a real compiler!"
A friend of mine, who was in earshot of this little exchange, thought it was most hilarious.
If you want to do things like this, you've got to have a pretty thick skin for this sort of "help" from your colleagues.
Successful people affirm the consequent [0]. For example, naming your language "D".
[0] http://en.wikipedia.org/wiki/Affirming_the_consequent
Edit: add scholarly footnote and tighten up language.
I once used D to implement a toy LISP language under a thousand lines including the native and loads of built-in functions. I can't imagine doing it this easily, this cleanly and this performant using either C or C++ (I even had fixnums and clojure-like vectors and maps, although not persistent.)
There's a lot of features to learn, and a lot of ways to use them. But the result is something almost as powerful as LISP, almost as safe as Haskell, as low-level as C and even more with inline assembly, and most importantly as productive as Ruby. Definitely a lot to digest as a first language!
Congrats on making the top 20 most used languages, well deserved!
With D he found his mojo, but people are still raving about Rust and are still mostly ignoring Walter. Big thanks to Facebook for deciding on D. Only ObjC came close but D is better.
I guess this guy never heard of Leor Zolman and the BD Software C Compiler [0] :-)
I got a lot of use out of that compiler back in the day, but I can tell you, Leor had no idea what he was doing when he started.
But we go way back -- we were in the same class and same dorm at MIT (before he dropped out and wrote BDS C), and were housemates for about a year c. 1981-82.
Ever hear of MINCE? It was a micro-Emacs that first ran on CP/M, later MS-DOS. The first version was developed using BDS C.
My degree was in mechanical engineering. But mechanical engineering was frustrating because of the large expense involved with building anything ... With programming, on the other hand, I could build the most intricate machines imaginable at no expense, needing only access to a computer.
Work is happening with ARM, but it needs more pre-early adopters. http://wiki.dlang.org/GDC/Cross_Compiler
Or releasing D++ !
(Amazing job with it all BTW)