Designing a Language Without a Parser
thunderseethe.dev
thunderseethe.dev
I can't really relate here. The parser is the easiest part of a compiler; the work only increases from there. I feel like if you ran out of steam at the parser, you never had enough steam to write a whole compiler. I don't think removing the parser will take you across the finish line if you otherwise were running out of steam.
My advice is to write your language in vertical slices. Write the parsing, semantic checking, and code generation for the simplest features first and progressively add feature slices, rather than trying to write the entire parser for a fully-baked language before proceeding. Consider including "print" as a built-in statement so you can print things out (and thus write tests) before you have working expressions and function calls.
I feel it's just that it is possible to say that work on the parser side is completed
meanwhile optimizations? you can probably endlessly improve stuff
I would absolutely recommend not supporting incremental compilation or error recovery in your first compiler. Just stop everything at the first error. Save that for your second hobby compiler, or better yet, the first commercial compiler that you get paid to work on.
Writing compilers is an especially useful project when you learn roughly how much you can chew. (Algol 68 is a nice example of a bunch of smart people, with relevant domain expertise, biting off way too much for the machines of the time.)
Everyone's different. The problem with parsing isn't the difficulty, but rather the potential for endless bikeshedding. You're having to make a ton of opinionated decisions that in turn produce more questions about your syntax. If your personality is like mine in that it's a bit obsessive "completing" a phase, then parsing feels like an endless quagmire. In comparison, AST -> type inference -> codegen feels more structured and straightforward.
I fail to see your point. OP was commenting on the technical difficulty of actually rolling out a parser, but you're not arguing the technical side. You're arguing the project management sides where "endless bike shedding" results in code churn. That's not the parser's fault, but the person's fault. You're bound to get hung up on details regardless of what part of the project you're working on, because it's not the parser compelling you to get hung up on details.
Sintax design, not implementing a parser.
You're arguing that you might not make up your mind and feel compelled to churn parts of your implementation due to your indecisiveness.
That's not the parser's fault, and I fail to see how anyone avoids this indecisiveness by replacing a proper parser with a bunch of ad-hoc hacks.
> chances are really high that you're gonna want me to do something different.
So what? Just work on it. How many parser generator tools are there? It's as if people have been doing this for decades.
Disagree. The fact that you're working on "a parser" creates the bad project structure that leads you to get hung up on details: a parser whose backend doesn't exist yet inherently has unclear requirements and a lot of bikesheddy decisions to make.
Thanks for chiming in!
Forth also has an interesting simple syntax.
Which version of Lisp are you talking about? A very brief web search showed eg https://stackoverflow.com/questions/40210896/syntax-error-in...
For something really simple, try `(define define)` in Scheme.
I'd say Forth has less syntax than Lisp. But that's perhaps a reflection of most Forth implementations using some clever tricks to offload much of what's handled by syntax in other languages.
Eg defining new functions (ie 'words' in Forth terms) is not done via a special syntactic construct in Forth, but via 'immediate words'. The lines are a but blurry, though.
The language has no syntax, the constructs within the language can and do impose additional restrictions (not extensions - restrictions) when evaluated. You may not find the distinction useful, but it is there.
Welcome to Racket v8.9 [bc].
> (define (define x) (display "Look ma, no error!\n"))
> (define define)
Look ma, no error!
Anyway, if you would be allowed to reimplement a built-in operator, then from then on you would be using the new operator and you would be limited to its syntax.
Common Lisp OTOH provides namespaces (called "package") and there is a pre-defined namespace "COMMON-LISP" for the core language.
This hasn't been a problem in this thread thus far, but to be explicit, this seems like a good definition: https://en.wikipedia.org/wiki/Lisp_%28programming_language%2...
Given that definition, we can see that there are many actual standards for programming languages with specifications under the name "Lisp". Scheme - as was mentioned in this thread, is one such standard, along with Common Lisp.
> Anyway, if you would be allowed to reimplement a built-in operator, then from then on you would be using the new operator and you would be limited to its syntax.
I think you're arguing the merits of doing something like redefining define here, which is not the subject of this comment chain. Of course such behavior is questionable at best. The comment chain is discussing if such a thing would be valid in Lisp - and, in some dialects (such as Scheme) it is.
Scheme also has a feature called "libraries" which makes such behavior more feasible, as you could still have access to the shadowed primitive. Consider:
gosh[r7rs.user]$ (import (prefix (scheme base) base:))
gosh[r7rs.user]$ (define (define x) 5)
define
gosh[r7rs.user]$ (base:define x (+ (define define) (define define)))
x
gosh[r7rs.user]$ x
10
Then see the Scheme standards. They include a syntax definition.
> The comment chain is discussing if such a thing would be valid in Lisp - and, in some dialects (such as Scheme) it is.
Lisp is not a defined language, thus you can assume anything.
My point is: Lisp dialects have syntax. This syntax is usually defined on top of s-expressions. S-expressions also have a syntax, but this syntax usually (there are exceptions) describes only data types (lists, conses, numbers, strings, symbols, arrays, records/structures, bit vectors, pathnames, ...) and not the actual programming language with its operators (functions, macros, special operators) & their way to define and invoke them.
Example: The LET operator in various Lisp dialects there have a specific syntax. This syntax is not the syntax of simple function calls like (operator arg0 ... args).
R7RS small defines this syntax (and also calls it syntax):
(let ⟨bindings⟩ ⟨body⟩)
(let ⟨variable⟩ (⟨binding spec⟩*) ⟨tail body⟩)
⟨Bindings⟩ has the form ((⟨variable1⟩ ⟨init1⟩) ...)
Every language construct has a syntax definition: function calls, macro calls, lambda expressions, special operator calls, ..., every macro.
It isn't. It's just a valid S-expr.
For example
((a b c) (d e f)) is not a valid Common Lisp program.
(let foo bar baz) is also not a valid Common Lisp program.
The syntax for LET is:
let ({var | (var [init-form])}*) declaration* form*
; in: LET A
; (LET A
; B
; C)
;
; caught ERROR:
; Malformed LET bindings: A.
;
; compilation unit finished
; caught 1 ERROR condition
Lisp has (simple) syntax for creating literals and writing code. But when you inspect your data from the inside of Lisp, they don't have any more (or less) relation to that syntax than eg Python does.
Eg no modern Lisp stores a function as an S-Expr of its syntax. You get an opaque piece of data that you can do certain operations on, just like in eg Python or Haskell.
Lisp using S-Expressions is an interesting syntax choice, but that's it. It doesn't by itself have any deeper impact on the language. (Of course, socially it does have a deeper impact, because it makes it simple for language users to write macros that look exactly like built-in parts of the language, instead of bolted on.
But that's all in human brains and social interactions, nothing fancy going on here on a technical level.
If you wouldn't mind much more tediousness, you could implement seamless macros in almost any language as a pre-processing step.)
I'm just having a problem with the concept of homoiconicity being touted as both coherent, and something that goes deeper than the very surface syntax level.
Do you mean something like implementing Prolog with macros in Lisp as a 'deep' macro? Or something else?
They aren't magic.
Besides the social stuff, S-expressions make it orders of magnitude simpler to build the infrastructure required to support seamless macros (just look at procedural macros in Rust for a recent example)
I have used Scheme, Racket and Common Lisp (SBCL) since the 1990s (before mostly switching to the ML family of languages). I read 'On Lisp' cover to cover. I know what I am talking about. You can still disagree with me, of course. But don't do it of the basis of me not having used enough Lisp.
S-Expressions make it easier to write macros that blend seamlessly into the host language. But there's no magic to them (compared to macros you could do as pre-processing in any other language) beyond that.
S-Expressions are a neat syntax. But homoiconicity is a weird concept; if it is coherent at all, it only pertains to the surface syntax level of the language.
Sorry, but you manifestly don't:
> there's no magic to them (compared to macros you could do as pre-processing in any other language)
The "magic" of s-expressions is that they make it easy to operate on the source code of a program as a hierarchical data structure (i.e. as an AST) rather than as text. That turns out to be an extremely powerful lever. It is is one of the reasons Lisp has lasted as long and been as influential as it has. I'm sorry if this sounds like an ad-hominem, but if you think there is "no magic" to S-expressions the most likely explanation is that you don't really understand them. There is nothing comparable in any other language. There's a reason that they are still in use today. Indeed, there is a reason that they keep getting re-invented again and again. HTML, XML, and JSON are all (badly) reinvented s-expressions.
> homoiconicity is a weird concept
No, it isn't. It is simple and straightforward: homoiconicity is where the primary representation of programs is a data structure in a primitive type of the language itself. It isn't any weirder than (say) recursion. If you think it's weird that is more evidence that you don't actually understand it.
where exactly is the magic? it's just tuple unpacking? like congrats you've constrained yourself to the absolute bare minimum and you've made it work. i mean i guess congrats for getting it to work at all but it's not magical, it's tedious. if you showed me a homoiconic language that did somehow pull off the magic trick of being more expressive than tuples then i would be indeed enchanted (mathematica at least manages to put lipstick on the whole exercise).
Magic lies in the eyes of the beholder. So you may not find it magical while I do. But I don't understand how you say that it is tedious.
With the homoiconicity of Lisp, you use the Lisp language itself to write macros and you manipulate the lists that make your Lisp programs directly.
Mainstream languages these days either need you to learn a special syntax for macros or they need you to learn their AST structures/classes and manipulate them or sometimes both. Isn't this more tedious? Isn't the Lisp way simpler and less tedious?
There is a reason I put the word "magic" in scare quotes.
> it's just tuple unpacking?
It's not even that. S-expressions are just a serialization of linked lists. The "magic" happens because the details of the serialization happen to make them particularly good for writing code.
See my reply to /u/eru for more details. https://news.ycombinator.com/item?id=36599470
Well, it's more like serialization of a binary tree. Cons-cells can branch in both the car and the cdr direction.
As https://news.ycombinator.com/item?id=36597550 points out, you can do this with more complicated syntax as well. It's just a bit more annoying.
I agree that ergonomics are a big deal when using programming languages, and it shapes their culture.
Just like it's a pain to re-use code in C, so everyone always re-implements linked lists from scratch every time they need one.
> It is simple and straightforward: homoiconicity is where the primary representation of programs is a data structure in a primitive type of the language itself.
Why does it have to be a 'primitive' type?
Eg Haskell supports representing ASTs just fine, but you would use user-defined types for that.
Imagine a variant of Haskell that used S-Expression syntax for the sake of argument. That version of Haskell would still represent S-Expressions internally with a user-defined type. See https://hackage.haskell.org/package/s-expression and mentally translate all the code into S-Expressions (but leave the semantics the same).
Or just imagine a variant of Lisp where cons-cells are a user-defined data-structure. That wouldn't make their S-Expressions any worse, would it?
As long as whatever data structure you use to represent your AST is easy to work with, that's surely good enough?
And what do you mean by 'primary' representation? A Lisp compiler (or interpreter) has many different layers, and the AST is but one representation used in one of the layers.
For many purposes, S-Expressions aren't particularly useful, because eg they don't keep track of which variables are bound where. And, of course, an interpreter that works by walking S-Expressions is really, really slow.
So in what sense are S-Expression a primary representation?
You could imagine a re-skin of C that used S-Expression syntax. It's actually relatively easy to write such a front-end as a pre-processor that translates S-Expression syntax into classic C-syntax before feeding the result to a C compiler. (And allows you to run C programs on your S-Expressions as macros.)
But that change by itself wouldn't change too much about the language. And wouldn't make S-Expression-C as pleasant a language as any old Lisp.
Well, yeah, but that is missing the point rather badly. There is nothing you can do in any high level language that you can't do in assembler. The only reason high level languages exist at all is to make programming less annoying.
> For many purposes, S-Expressions aren't particularly useful, because eg they don't keep track of which variables are bound where.
S-expressions are nothing more than a serialization of binary trees. There is a long, long list of features that they do not provide. Again, if you think that is relevant, you have completely missed the point.
For the record: the point, the thing that makes s-expressions cool, is that they provide a serialization of linked lists, and this serialization is super-simple, to the point where writing code in it is actually practical. They weren't designed for this. The fact that you can actually write practical code in s-expressions was a surprise, a discovery, not an invention. You can serialize binary trees as XML or JSON, and you can even use those to write code if you want to, but no one in their right might would actually do that. You would go nuts typing all those angle brackets, double-quotes, and commas. The reason s-expressions are cool is that they are parsimonious. You don't need all the angle-brackets and quotes and commas. S-expressions are actually a reasonable syntax for writing code whereas XML and JSON are not despite being fundamentally the same thing under the hood. Indeed, once you get accustomed to s-expressions, they are a superior syntax for writing code than conventional languages because they are easy to parse (no precedence rules to deal with).
Oh, there's plenty of stuff you deliberately _can't_ do in high level languages (compared to assembly or low level languages).
Eg Python doesn't let you use naked jumps (also called GOTO). Haskell doesn't let you mutate state willy-nilly.
> You don't need all the angle-brackets and quotes and commas.
Well, you have quoting and quasi-quoting. Very useful, and they use some more interesting syntax.
Yes, S-Expressions are a reasonable syntax. I used to like them a lot. But I also came to appreciate more traditional syntax when switching from Scheme to Haskell.
I was not arguing against S-Expressions at all. Only against 'homoiconicity' being a coherent or useful concept, or saying anything deeper about a language.
As I said earlier, if you start from C, change its syntax to use S-Expressions (rather simple to do), and allow S-Expression based macros (that use the same C-as-S-Expression syntax), you still don't have a language that's as pleasant to work with nor makes macros as easy as your favourite Lisp.
Not natively, but you can do it. It's just "a bit more annoying".
> you have quoting and quasi-quoting
That is obviously not what I meant. What I meant was that you can write (x y z) instead of ("x", "y", "z"). The latter is just "a bit more annoying" but that "bit" makes all the difference.
Also, ' and ` are not really a core part of s-expression syntax, they are just a convenient shorthand. Any s-expression written using ' and ` can also be written without them. The core syntax includes only parens and dots, and of course the constituents of atoms. Modern implementations also include native support for double-quoted strings, but that was not part of the original spec. Everything else is just syntactic sugar.
> And, of course, an interpreter that works by walking S-Expressions is really, really slow.
Common Lisp interpreters work by walking S-expressions. Typically you'll see the s-expressions directly in a debugger -> modifying the s-expression will modify the running program.
In Common Lisp
(defun times-two (x) (\* x 2))
(print (function-lambda-expression #'times-two))
> (LAMBDA (X) (BLOCK TIMES-TWO (* X 2)))
Yes, Common Lisp stores the s-expression. And you can get at it by applying an operation (in your case, `function-lambda-expression`) to some opaque handle.
The function itself isn't stored as its s-expression. Neither does any reasonable interpreter work by walking the s-expression. (Doing so would be very slow.)
S-Expressions (or more precisely, the AST) is just one representation amongst many that's in use during the different stages of the compilation process.
They are still a great syntax choice for the kind of languages that Lisps are. Totally agreed.
My point was that homoiconicity is a surface level property that you could bolt on to eg C or Cobol, via a simple pre-processor, and it wouldn't change those languages much.
Just like you could bolt on using Python-style significant indentation to C or Fortran or Lisp, and it wouldn't change those languages much. (Apart from changing their syntax, of course.)
Besides pointing out like others have that pretty much all Lisps have an interpreter that stores programs as s-expressions, why do you think compilation makes making a DSL in Lisp harder? Do you think you somehow need a parser all of a sudden?
> My point was that homoiconicity is a surface level property that you could bolt on to eg C or Cobol, via a simple pre-processor, and it wouldn't change those languages much.
Writing in C instead of assembly does not make your processor more powerful. Similarly, when designing languages, homoiconicity allows you to work directly with the AST in a way that is most ergonomically simple. Saying this is a surface level thing is an understatement
In version control jargon, only primary objects should (ideally) be tracked in a repository, not secondary/derived ones. The most common example of primary objects are source code files; and of secondary/derived objects, compiled files.
Programs first exist as source code; primary means first. The semantics of the language is defined with reference to the syntax, whose description refers to the source code form. Everything begins with the semantics: it is primary. Other forms, like compiled, have to produce the same results like the reference evaluation model over the source code. Expansion of macros is defined over the source code. Other tooling is based on source code, like text editing, tags generation and whatnot.
Compiled code is not primary in any way. It is non-portable, sometimes not even fully documented, and supports almost no useful operations beyond invoking it (calling a compiled function or loading a compiled file) and disassembling compiled code.
The representation of compiled code can change between releases of a Lisp; the worst that happens is that users have to recompile their code since old compiled files do not load any more. If you break the source language, there will be gnashing of teeth.
The basis of portability among implementations of the same dialect is always the source code.
Lisp programs can quote pieces of themselves as literal constants. Atoms can be quoted, as well as symbols and entire lists. These quoted lists have to survive into the compiled form. Thus, the syntax is important because any piece of it can potentially be a literal which has to retain its shape and content no matter how the program is translated.
Textual source code contains valuable information not found elsewhere: formatting and comments.
In Lisps that have programmable readers, like Common Lisp, only the source code contains instances of the original read syntax. For instance, in some implementations of the backquote syntax, the reader expands it, and this may be done in such a way that the resulting object cannot be printed as backquote syntax any more. Application-defined read syntaxes usually do not support printing back in the original form. Thus they appear only in the textual source code.
Primary information is already lost when we have merely read an expression into memory without doing anything else, like expanding and compiling it.
You are not alone:
http://calculist.org/blog/2012/04/17/homoiconicity-isnt-the-...
Btw, it seems you could implement a 'read' that also recognises Python/Haskell-style significant indentation in addition to parens only. That would be an interesting elaboration of the point in the post.
Are you sure this blog post is making the point you have been trying to make?
Here is blog post that explores writing C in s-expressions.
Many Common Lisp implementations do that, especially those with an interpreter: SBCL (yes, SBCL has an interpreter, too -> one can tell SBCL to use an interpreter), LispWorks, a Lisp Machine, Allegro CL, CLISP, ...
LispWorks:
CL-USER 18 > (defun foo (a b) (* a (let ((a (* a a))) (break) (+ b (expt a b)))))
FOO
CL-USER 19 > (foo 1 2)
Break.
1 (continue) Return from break.
2 (abort) Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.
CL-USER 20 : 1 > :n
Call to INVOKE-DEBUGGER
CL-USER 21 : 1 > :n
Call to BREAK
CL-USER 22 : 1 > :n
Interpreted call to FOO
CL-USER 23 : 1 > :lambda
(LAMBDA (A B) (DECLARE (SYSTEM::SOURCE-LEVEL #<EQ Hash Table{0} 82200D6A93>)) (DECLARE (LAMBDA-NAME FOO)) (* A (LET (#) (BREAK) (+ B #))))
CL-USER 24 : 1 > (pprint *)
(LAMBDA (A B)
(DECLARE (SYSTEM::SOURCE-LEVEL #<EQ Hash Table{0} 82200D6A93>))
(DECLARE (LAMBDA-NAME FOO))
(* A (LET (#) (BREAK) (+ B #))))
Considering the author is talking about an endless graveyard of abandoned projects it kind of sounds like ADHD. I have ADHD and have a similar problem.
I still do endless bikeshedding where I'll do things like insist on zero memory allocations or streams/iterators over slices and those arbitrary limitations will completely compromise the project. Either I never finish it because I can't get it to work, or I get it to work but it sucks because of stuff like "the amount of work to avoid memory allocation might be much larger than the work of just making a damn memory allocation".
But yeah, write the vertical slice and implement a print builtin are good tips.
For more complicated languages, there are tools like tree-sitter and ANTLR4. You can even extend an existing tree-sitter grammar to augment a language without having to re-write the base language parser.
I really wish Python 3 had taken that advice. ;-)
If we are being honest here, most people just want their own better Python/Go/Rust. I doubt your average engineer is truly interested in the subtleties of compiler design and memory layout optimization. A single pass compiler, built like the olden day of yore, is the best option for most.
https://en.m.wikipedia.org/wiki/One-pass_compiler
Leave the 3 stage design for homework (or if you are being paid to do so).
I see programmers, even experienced ones, making this mistake all the time when implementing larger changes. They try to get the whole thing implemented at once even when it's obvious from the outset that there are multiple sub-features that can be more easily implemented separately, so one can focus all their efforts on it, make sure it's well tested and clean, before moving on to the next sub-feature.
On the other hand I do really hate the task of writing parsers, so I can relate to people who think it's the worst/most difficult part. But I think other parts are probably harder, like getting type system stuff right.
To me there's a clear analogy with startups. The naive conception of starting a company is that you get a pile of money so you can hire a bunch of people and create important infrastructure. But with startups, you're trying something new, so the most efficient use of time is to find the riskiest hypotheses and test them as directly as possible. That often involves doing things that seem wrong if you proceed in the "logical" way. E.g., I knew a successful UGC company that didn't implement accounts and logins until like 6 months in. But that was fine, because actual accounts were not needed to figure out whether the business worked.
I don't know where I heard this, but the idea is so important to me that I saved it on my blog: https://dorotac.eu/posts/in_the_middle/
For a programming language? Maybe if you are designing your language by feature list.
What if you are designing a programming language for ergonomics instead?
Let's be real - the differentiating factor in any modern language design is the syntax, not the features. They all mostly support a similar cross-section of features, in terms of "getting things done".
What you are really designing is a competitor to the existing languages, in which case it is beyond the scope and effort of a lone developer to match feature-for-feature of modern languages.
My experience of lone-wolf programming languages is all the same ... namely ...
Even if you do have one, single, differentiating feature, people aren't going to adopt unless you have all the other features they want. Doesn't matter how good your feature is if your language is missing some feature that people like in current mainstream languages.
You should also be careful of thinking that a single good feature will cause a little adoption; if it's any good the existing languages will simply adopt it!
Another path into darkness is thinking that the batteries included is so different to current offerings that people will adopt it, such as that recent post on HN about a language developed for cloud by the author of CDK. There's nothing in that language that can't be implemented as a library for existing languages.
For programming languages alone, going feature-first is a good way to produce an obscure language that no one is interested in. Without even a small community, the original dev themselves won't use it.
Where a new language makes sense is in ergonomics, not in features.
Can you make the syntax such that people onboard quickly? Can your syntax support something complicated in a manner that the most simple-minded developer can understand? Is your syntax amenable to collaboration? Can it be easily parsed in pieces for IDEs? Will the output be package-distributable or module-distributable? Can you ensure easy GDB integration? What build mechanism can be used (for reading the sources and figuring out dependencies).
Syntax is the major difference between writing in Kotlin and writing in Java.
My new language project, I'm still trying to nail down what the syntax should look like. I have no problem documenting the tree to support features I want, but I find that settling on what good syntax looks like to the majority of corporate developers is really really difficult.
Compare with, from an AST, emitting code for some advanced language constructs. That's almost a mechanical effort that I think I can delegate a lot of to ChatGPT.
Designing universally acceptable syntax, on the other hand, is a lot more complex and requires actual human decision-making.
Actually, his criteria was exactly the same as mine when I first started writing my own languages - learning experience, bragging rights, whatever.[1]
At some point, though, even I didn't use my own languages, because without a community of users, it soon became pointless.
That's the thing about some projects - they take off even when the objective is not to build something that takes off. It's why I admire (and maybe slightly jealous of, as well) Andreas Kling's SerenityOS.
It's the only project I can think of in recent years that was totally superflous and unnecessary (any solo OS or PL project is, at this point in time), and yet its gathering steam and shaping up to be a real contender.
But that type of thing is like winning the lottery - the odds are against it.
[1] One of my comments on HN last week mentioned that I want to write a Forth type language, and telling one of the responders to my comment that it isn't because there doesn't exist a quality Forth implementation.
I prefer the analogy of painting. You start with collecting references, exploring ideas in a sketchbook, make color tests, draw outlines on canvas, use big brushes for colors, refine with smaller and smaller details.
The problem is, that programming is all details / only details. There is no easy way to use big brushstrokes, so you have to improvise and not loose the overview. It doesn't help that engineers love details.
(Esterel —IIRC— is the only language of which I'm aware that explicitly has two syntaxes, one traditionally parser based and one that, in principle, could be parserless)
Parsing something might be trivial but its still parsing
(in the examples above —which missed the prologs— I'm pretty sure the parsing is trivial on the order of "you can see everything that handles 'parsing' without needing to scroll a window", and in several of those examples it'd still be true even if your windows were only 25 lines long)
> in that case, pretend I wrote "grammarless"
But Lisp is not grammarless. Smalltalk is not either. The Lisp reader, notably Common Lisp, is a non-trivial piece of code. The Common Lisp lambda list, as a language construct, is not trivial either. (Dare I mention the CL LOOP macro?) Just because everything is "a list, symbol, or constant" does necessarily make the parsing problem trivial.
It SEEMS simple, but then you get into it, and you find you fall into the weeds.
OP wants to write a hobby language, not reimplement Common Lisp.
If OP wants sexprs, they can get quick-and-dirty sexprs by:
(1) substituting spaces around each paren in the input
(2) breaking input up into a list of whitespace separated words
(3) folding over the resulting list (by looking at its parens) into a tree
You know what it takes to implement a parser for s expressions?
E := atom | '(' *E ')'
With atom expanding to all data types you wish to support.
Let's presume using a parser generator is an insurmountable task for someone developing a parser. In a hand-rolled packrat parser, this whole grammar is implemented with a couple of functions.
Is writing two functions too much work for someone wanting to write their own parser?
Your ad-hoc trick is much more work than actually implementing a proper, maintainable parser.
You are always parsing a grammar. It seems you simply aren't aware of it.
Every single time you write code that expects to find a certain type of token after another type of token, you have a grammar.
If you have an idea for a format or a language and would like to quickly start hacking on the layer above the syntax, Jevko is an option.
It's meant to be even simpler and hackable than S-expressions.
It gets you from a string to a tree in the least amount of steps.
See here[2] if interested.
Happy hacking!
[0] https://jevko.org/ [1] https://djedr.github.io/posts/jevko-2022-02-22.html [2] https://gist.github.com/djedr/151241f1a9a5bc627059dd9b23fc74...
I was more directly inspired by Lisps, but I do prefer the original M-expressions and the syntactic choices that REBOL and Red make.
I think placing the operator before the opening bracket better emphasizes its special significance and can reduce nesting for constructs like `f[x][y]` (vs. `((f x) y)` in Lisps). Square brackets somehow seem more aesthetically pleasing to me. And there is a practical reason to prefer them, especially if your syntax uses only one kind of brackets -- square brackets are the easiest to type on an average keyboard.
So REBOL-like syntax is nicer. As were M-expressions. They probably didn't catch on, because they were not minimal enough, compared to S-expressions. And maybe because S-expressions were fully implemented first.
Benefits of this approach include there being no such thing as a syntax ambiguity, language extensions being as easy to make as libraries, the language being easy to write by non-programmers, and a JetBrains IDE for your language - at the cost of not being able to use any other editor.
A great example of such a language is mbeddr[1].
[0]: https://www.jetbrains.com/mps/
[1]: http://mbeddr.com/
I had hope even though I knew what the deal with projectional was going in and indeed I was disappointed. The number of proprietary "dsl"s you have to grok to be proficient at using MPS is just mind-boggling and that's coming from someone that designs dsls for a living. Like by the time you get to codegen you're already 3 or 4 deep. And it's only for codegen, where you're generating Java, that you get the full projectional editor treatment. Everything else is just a gui form for some small "dsl" with effectively dropdowns, so you might as well just call it an API for a configuration system rather than a language. Like there's zero composition for the structure "language" and the editor "language". You're literally just toggling forms.
It's just a complete turn-off because even if it is "powerful" it's completely non-portable - you cannot ship anything without shipping MPS to orchestrate the cascade of dsls.
I tried to get mbeddr to work but could not even though I can drive gradle and etc fine.
Overall really disappointing.
Sorry if this sounds stupid or obvious, but with this kind of thing I find that it's easier to cross the finish line if you maintain humble goals. Focus on just getting a working end-to-end MVP. Refine and enhance it down the road; don't get stuck trying to make version 0 an awesomely praiseworthy effort.
Hoping to hear a positive update from you in the near future!
The other reason is that a language like Common Lisp is defined in terms of the data-structures used by the language and the language has no unique text representation: the “default” reader uses a slightly extended version of s-expressions, but any data structure in the language can be evaluated and any transform from text to data structures can be a textual syntax for Common Lisp.
Lisp helps with this. You have the syntax settled, and can concentrate on the language from the get-go with all your steam.
Only to the extent that AST structure depends on syntax. For something like s-expressions defining new semantics never requires new syntax since arbitrary trees suffice to syntactically express any AST.
Well, 12 years later (ie. early 2023) I realized that I don't really have any kind of cool sideproject and started implementing the same idea in C (with the added goal of the VM being natively multithreaded with fine-grained locking along the lines of JikesRVM and WebKit). Well, I have stub implementations of classes needed for the AST representation and S-expression reader and writer…
That got me thinking that if I ever do any language design again I’m almost certainly going to just start by writing a Lisp and then layering any additional syntax that appears desirable on top of that.
Naturally there are limitations here. There's not much you can do to poke into the structure of Haskell functions other than evaluating them, so any sort of optimization and code gen work must happen by evaluating the function in Haskell, which gives you more of a challenge in designing the AST.
That said so far the author's language resembles simply typed lambda calculus and this is generally too simple. The meat of the language hasn't actually been designed yet in this post.
You could also try doing an embedded DSL in Haskell or even Kotlin if you just want to get your feet wet.
Course page: https://www.lix.polytechnique.fr/Labo/Samuel.Mimram/teaching...
Having nice syntax is a good thing, yet no matter what the syntax is, it ends up as an AST. And the structure of AST records and the ways they can be combined matters quite a lot, I would say more than text syntax. XML makes it visible and easy to experiment with without getting distracted by text syntax niceties.