So You Want to Write Your Own Language (2014)

Great comment. And if a domain-specific-language (DSL) of really custom syntax is needed, an alternative is to use Racket (a Lisp-family language), which allows for creating DSLs of all kind. Racket is already used for designing your own programming language and trying it, so I'm surprised that the author did not mention suggest Racket as a test-bed for the new language.

> This article is sort of fundamentally wrong that it's difficult to write small, purpose built languages. It's not, and even outside the functional and homoiconic meta-syntactic world we have seen code generators deployed regularly.

Correct. For example, most of the common systems used today do use a 'code generator' internally: Every software that uses an ORM library (i.e. Hibernate), is already using a sort of domain-specific-language (i.e. HQL) that, via a code generator, translates to another language (i.e. SQL). So most major platforms (Java, .NET, etc) already have one or many popular libraries that do implement some sort of (limited) DSL and code generator.

I think the aim of the article was to create a new compiler from the ground-up. I think the correct choice for syntax is something that can't be boiled down to three pieces of advice or one article. It is a topic that would require very deep discussion.

You can leave away higher-kinded types if you leave away types altogether, but I don’t think anyone has had a practical benefit from a monadic effects system or transformer stack in e.g. Javascript. I’m not even sure the plain »monad« concept even exists as an abstraction instead of in docstrings.

This "correction" is mischaracterizing Walter Bright's article. He's writing about industrial "professional languages" and that context is made clear multiple ways: 1) he's referencing his D Language and presumably languages like it, 2) desire for the compiler to display quality error messages, 3) a good runtime library. The comparables to D Language would be Java/C#/Golang/Rust. All those languages took years by a team of people to get to v1.0.

> we have seen code generators deployed regularly.

But no professional language's (like D's) canonical compiler I know of is the direct output of yacc or ANTLR.

It seems like you have made reasonable and correct statements about programming languages -- for someone else's article -- but not specifically Mr. Bright's. The context really matters here.

If you're designing a language as a learning exercise, or a domain-specific one, my advice is to go right ahead, using a Lisp variant. If you don't know one, now is the time to learn one.

If you're developing a general-purpose language, ask yourself how your language will be better than, or at least different to, every general purpose language in current use. Be ambitious. As languages belong to families (e.g. Algol variants, Lisp variants, functional languages, visual dataflow languages, etc.), you should aim for your language to be the best in its family.

Your language should also be one you'd use yourself.

I'm developing Full Metal Jacket (http://web.onetel.com/~hibou/fmj/FMJ.html). It's very different from almost everything else, so I'm not expecting popularity overnight.

The Dr Dobb's article is way too conventional for my requirements. My language doesn't require any parsing. Syntax, type, and memory errors can't happen. The ideal target machine would just run the code directly without any need for compilation, though I will at some point need to cross-compile it onto the von Neumann architecture, and for that dataflow analysis is unnecessary, making optimal object code simpler to generate.

This is the direction I'd like to see computing go.

So why not try letting the program exist as an embellished AST, that you edit in multiple IDE views? You could still have text views that present the program in one or more programming languages. But also views like circuit diagrams, graphical call graphs, other standard programmer diagram types. A run-time view with a time axis and time controls, with graphical representations of the program, ability to rewind, fiddle with state, travel visually down all the code paths.

Can we really do no better than text + our imaginations and whiteboards? I'm almost certain we can do better.

Or, if you find those ideas a bit eccentric, then start with just having views of source code. For example, my team requires verbose symbols, but I like very short symbols. I wish I could just have a simple map from the long symbol names to short ones, and toggle my view of the source code without affecting the source code. But since no IDE I know of has the concept of a view, I don't know how to do this today with existing tools.

I've seen you lump perl and bash together before, but around the idea of sigils like $ and @. This sounds more like a complaint about "default" operations. Like Perl's $_ perhaps?

Other than that sort of thing, I don't see where Perl reaches that far in being concise. Maybe regular expressions being a first class thing? Like "if ($foo =~ /bar/)" ? Though that seems straightforward to me.

Other than that, reducing the number of tokens is good.

If we can express a solution in fewer tokens of one language than of another, that's a good indicator that our language having better abstractions for that problem. The longer solution is perhaps spending some tokens to solve some distracting sub-problems which the shorter one doesn't have to.

But if the token count stays the same, and we simply rename the tokens to one or two character sequences, we haven't gained much, and it is possibly even detrimental.

Of course, missing semicolon errors are still annoyingly bad, I wish GCC and clang could produce better errors messages in this case.

A lot of good that will do if your language doesn't live long enough for a critical mass of those people to keep it alive. shrug

Food for thought and discussion, rather than a "correction": Terminators are not required in a language. See the concatenative languages like Forth or Factor, in which

    z = func1(x, y);
    func2(1 + 2 + 3, z);

    1 2 + 3 + x y func1 func2

take foo("one", "two") mis-typed as foo("one" "two"),

instead of getting a "missing comma" error, you get "wrong number of arguments", unless the function can take one argument, in which case you may get some other random error.

This might not seem all that useful in a practical sense, but I think that's basically the author's point: Such a language would be essentially impossible to write, and thus you're always going to have redundancy. As a result, you should evaluate redundancy in practical terms, rather than viewing it as always needing to be removed.

It works the same for programming languages—when designing a notation, you need to consider how you’re going to produce useful error messages when people make common errors, and the simplest way to do that is to add a bit of redundancy so you can infer the programmer’s intent.

For example, I’m working on a dataflow-type language where the syntax for introducing local variables is evocative of a labelled edge in a graph; the original syntax to introduce three variables was this:

    -> x y z;

    // Whoops, accidentally declared 6 variables
    -> x y z
    foo bar baz
    { … }

    -> x, y, z;

    -> x, y, z
    foo bar baz
    { … }

Likewise, in C-style languages you have to “redundantly” specify the number of arguments to each function at each call site using commas:

    foo(1, 2, 3)

    foo 1 2 3

This statement feels imprecise in a couple ways. It seems to imply that some IDEs actually use context-free grammars for syntax highlighting? Which ones?

As far as I can tell, Vim and Textmate bundles (i.e. what Github uses for syntax highlighting) don't use anything close to a context-free grammar for their syntax highlighting models. They are more like ad hoc lexers -- a collection of rules and regular expressions.

Certainly an editor doesn't want to parse the entire file to highlight text, because it has to potentially re-highlight at every keystroke. Also, you want to be able to highlight malformed programs (i.e. code with syntax errors). As far as I understand, that's generally why they don't use grammars.

I think you might mean that a language should be designed with a concise grammar so that somebody else can reimplement it by hand more easily? That is, you want multiple independent implementations.

If that's true, "context-free grammars" is the wrong term to express that notion. Context-free grammars don't handle many real languages, not just C and C++, but also Python/Perl/Ruby, and even JavaScript and Go (semi-colon insertion.)

Also, a lot of tools are enabled not by grammars but by producing specific data structures for your front end. (e.g. here is the way I think about it: http://www.oilshell.org/blog/2017/02/11.html )

The different between Clang and GCC is that Clang is a library that enables a whole ecosystem of fantastic tools, including IDE support. But Clang doesn't do much at all with CFGs. The real difference is that the front end is a library that produces a very rich representation of the code.

The C# front end does similar things. More links here: https://github.com/oilshell/oil/wiki/Lossless-Syntax-Tree-Pa...

In other words, I would say that integrating the compiler front end into the IDE became the more popular and successful approach. And of course the compiler very much has to be designed with this use case in mind.

Additionally a it's not usually that hard to create your own tokenizer and parser by hand.

So... C#?

I understand that your aim would be to create a 'beginners' language. But why keeping the syntax of C? C's syntax is not so much appropiate for beginners, Python is.

C's advantage, or reason-to-be, was to create a better alternative to assembler. That is, to create a language almost as fast as assembler, but with higher-level constructs. Still, C is a pretty "low-level" procedural language. That's why all the features for manipulating pointers, etc. C wasn't really thought as a beginners' language.

From your description, perhaps you would better keep Python syntax as is, and perhaps simplify some features a little bit?

Python may have its failures, but the syntax is so clear that it is very suited to beginners. And I dare to say that it is already used in many introductory courses to programming.

Many had these fantasies before realizing the intrinsic conflict. First, it is merely a library away to have high-level data types in C, but that is not enough. So the wish is not merely to have, it needs to be native. What does that mean? First, it needs syntax support. That can be managed by macros and, in c++, templates. But that is still not enough. So the wish really goes into the semantic level, that is, high-level data types that do not need worry or understanding of its low-level subtleties. But, you now should realize, that without understanding and worrying about these low-level subtleties, there is simply no reason at all to use C versus, e.g. python. We use C because we want to program in low-level that is impossible or difficult to do in the other languages, e.g. efficiencies. Wanting to do low-level programming and refusing to distinguish and understand low-level subtleties are intrinsically conflicting.

So any such effort of adding high-level data types natively to C will end up with syntax full of details of low-level subtleties: types, memory management, runtime, etc. The language will no longer be simple and they still doesn't feel like native (as python).

So honestly don't make another C++. Embrace C. Use Python.