Stem, a stack-based language with metaprogramming and a C FLI
ret2pop.nullring.xyz
ret2pop.nullring.xyz
https://concatenative.org/wiki/view/Concatenative%20language
I was interested in the C interface part but it never got there. :(
min does not have a C FLI essentially because it has a "Nim FLI". It is really easy to add new symbols to the language using a Nim API (which is also used internally to define all symbols of the language).
See https://min-lang.org/learn-extending/ for more info. Maybe I should generate some Nim docs for the public API though... but it's really easy to use, and you can check the lib folder of the repo for examples: https://github.com/h3rald/min/tree/master/minpkg/lib
I'm curious, why did you undertake the project? What are your goals for it, or what problems do you intend for it to solve?
I'm going to include a bit from the README here, which describes some of the functionality/design but not the why behind the language:
> Stem aims to be a minimal interpreted stack based programming language that allows for metaprogramming and a foreign language interface. It features a C API that is elegant and simple. Additionally, garbage collection is not needed in this language as the responsibility of writing memory safe operations is put in the hands of the maintainers of the builtin functions and FLI library maintainers. Therefore, the end user does not need to worry about memory allocation while the implementation remains extremely simple.
Goals -- I eventually want to write an emacs-like text editor in stem by writing a C library that does bindings for ncurses. I think it would be fun, for one, and I think it would not be that hard of an end product to make. I also had plans of writing a stem compiler, and I already pretty much know how to do that, but there are a couple of difficulties in doing that. Namely, right now, there is no way of telling the difference between compile-time and runtime operations, which will slow the language down considerably. My solution to this currently is basically "interpret once, run binary forever", but the details are still need to be worked out.
Problems it solves -- to be honest, you can really do anything in any programming language, but this langauge aims to be a higher level rendition of stack based programming languages, mostly for scripting and creating abstract representations of things.
Note that I am still trying to write in a couple things, namely, an actual include statement as a builtin that reads from some predetermined standard include directory, and an OOP implementation using metaprogramming in the standard library.
Not to mention the fact that I have been really interested in writing programming languages since I was in high school. One interesting thing about stack based languages is that the AST need not be generated at all. I taught myself a lot of linguistics, specifically generative grammar, as a result of attempting writing various programming languages in the past, and so these types of projects are just interesting to me.
If you ARE looking for one unique thing about this language, though, it is that `def`, the method by which you define functions, is itself a word. I don't think a lot of stack based languages do this (because it is much harder to compile languages like this as mentioned above), but it offers much more internal consistency within the language as a result.
But of course, that is the way of the Forth.
Hmmm… Have a look at JonesForth. I think it might be very rewarding for you.
Cool article posted to HN a little while back: http://ratfactor.com/forth/the_programming_language_that_wri...
The more common : word enters the newly defined word into the dictionary, which is Forth speak for the newly defined word now being considered 'in scope'. The :NONAME word instead returns an execution token on the stack, which is analogous to a function pointer.
See also [2], on Forth's execution model more broadly.
[0] https://www.complang.tuwien.ac.at/forth/gforth/Docs-html/Ano...
You can even define your own 'mirror' of the : word, which involves some fiddly management of Forth's state, but it's doable. [1]
See also [2] on Forth's execution model and [3] on defining your own defining words.
[0] https://www.complang.tuwien.ac.at/forth/gforth/Docs-html/Con...
[1] https://www.complang.tuwien.ac.at/forth/gforth/Docs-html/Use... (The specific code shown is specific to Gforth, there's no stats or LATESTXT word in the ANS Forth standard.)
[2] https://www.forth.com/starting-forth/9-forth-execution/
[3] https://www.forth.com/starting-forth/11-forth-compiler-defin...
So depending who your lawyer is, : and ; are "words". :=)
How would say the language performs at runtime vs other REPL languages. Is it compiling to native code behind the curtain or is there a VM back there?
Do you know of a language which is not based on stacks?
I would say that languages where values are assigned a ‘name’ in the program text are not ‘stack-based’ in the manner implied by the term. So while C and nearly all other programming language have a stack data structure allocated as storage for values used in procedures/functions, they are not ‘stack-based’.
This is a shitty and broken term. Hence my comment.
> is a language which there’s a stack threaded through successive procedure calls, where access to the values on that stack are accessed by popping them off and computing with them, then returning a result via a push.
You're just describing a procedural language that returns values. I suppose you could also describe a procedural language that doesn't return but what would be the point? You're just describing a form of indirect subroutines that revolve around jumps rather than the natural instruction progression. This has been a wildly unpopular method outside of entering a performance loop for many decades. It knows it will never have to exit/unwind the stack/call any kind of destructor or deallocation callback when entering a non-exiting loop.
Ok so why don't people (you!) just say "concatenative" given that other languages are trivially stack-based as well? How is it surprising at all that it's mostly syntax and a mediocre runtime (as the majority of languages can boast) that differentiates your pet language from other languages? Can we just admit that one of these options has a complex syntax and the other has a trivial syntax?
The main point you made that I can not agree with the characterization of other languages being ‘trivially stack-based’. Other languages in the Algol, ML, Lisp families have stacks that are an implementation detail of their function semantics. In those languages the stack is an implicit storage mechanism that merely holds values which are semantically associated with some ‘variable’ in the program syntax. In ‘stack-based languages’ the stack is, most usually, an explicit semantic construct which acts as the only available value for procedures to act on.
If your primary argument is against the term ‘stack-based’ that’s a personal position, but there are semantic differences between Forth and C, where emphasizing the explicitness of the stack in Forth is a viable point of divergence. As to using the term concatenative to describe a language, I have some opinions on that which fall outside the norm, so I tend to not use it to describe any of the ‘popular’ languages usually lumped under that umbrella.
I don’t know if the ‘pet language’ remark was towards me or in general, but it was certainly baseless if directed at my comment. If it is a general statement, I would only say that it is common for a community to settle on some terminology that they like and it’s rarely out-group pressure that forces a change in that language (see the ‘Alan Kay invented the term Object Oriented and it doesn’t describe C++’ argument that occurs frequently on forums like HN).
To the extent this is true, this is also true of all languages that anyone uses these days—C, JVM, Rust, Chez Scheme, GHC, python, ruby, ocaml, blah blah blah. The only difference is the syntax used to manipulate the stack. Acknowledging that syntax is the main difference from other languages seems key to evangelization. The idea of "implicit/explicit" doesn't really make sense. how do you refer to the stack in forth? You don't. How do you refer to the stack in C? You don't. Both operate with respect to the stack without ever referring to it via syntax. In both languages, the stack is invoked by referring to procedures/words.
Look if you want to keep up this fantasy that forth or factor is somehow more stack-oriented than any other languages, go right ahead. But you're going to fundamentally misunderstand why people use the language. It's the syntax! It was always about syntax. The stack is just a necessary implementation detail to deliver this syntax, same as in every other language that people actually use.
From what I've seen of other forths, I much prefer how this one does things. Using quoting as a sort of lambda is much nicer and more consistent than the weird compilation semantics you see in stuff like GForth.
If I had a choice of making and then using a DSL in Python, Lua, or Forth, it’s Forth all the way, no question. Now if you include Scheme, Ruby, or Tcl, then it might become a competition. (Haskell is somewhere in between.) It’s worth appreciating how much above his weight the humble granddaddy is punching here for the dozen K of memory that a running system normally takes. (Less relevant now than for the micros of old, as for interactivity that memory needs to be RAM, whereas the cheapest controllers today are usually ROM-rich and RAM-starved, relatively speaking.)
a priori "must haves" that eventually get in your way. If I had to choose one quote from Chuck Moore (the creator of Forth), it would be "Don't anticipate, solve the problem you've got" [1].
What you need on a daily basis it to be able to reuse names so that you don't clutter your dictionary. But at the same time you do want your Forth to warn you when you define twice the same name, because that could lead to vicious bugs. So all you need to do is to hack your Forth so that e.g. names starting with underscores can be redefined and won't trigger a warning.
> What you need on a daily basis it to be able to reuse names so that you don't clutter your dictionary. But at the same time you do want your Forth to warn you when you define twice the same name, because that could lead to vicious bugs. So all you need to do is to hack your Forth so that e.g. names starting with underscores can be redefined and won't trigger a warning.
The problem is already solved. You just spit out a warning message when something is redefined instead of an error.
I don't think I did such a thing.
> Do you have any concrete [proof] that the Forth implementation with compile-only words is better? What are its benefits?
Yes I do : a Smalltalk-style forces to have the code (to be executed if the condition is true) to be on the left side of the if. Additionally, if you don't want the address of the quotation to be in your way, you are also forced to adopt an argument order like condition-quotation-if, which is even less convenient than Forth's syntax. Oh, and the else part could be a lot of fun too. Most likely, you'll add some sort of exception to avoid all that, but exceptions are the archetype of unwanted complexity. You seem to think that Smalltalk-style is simpler, but Smalltalk is an infix language with, granted, rules that are simpler than most other languages - but still more complex than those of Forth. So it is not, actually, simpler when you consider the system as a whole.
Also, it makes it more difficult to build loops from that if and recursion like Scheme does (TCO). Chuck Moore's Colorforth does exactly that.
Moreover, and perhaps more anecdotally because few people care about this these days, it is more space and CPU efficient - basically the traditional Forth if is one instruction (bytecode or whatever) followed by one parameter (the jump target) that takes one argument on the stack. A Smalltalk-style if is also one instruction, but it has to call the "if-true" code, and this code has to have a return instruction as well. I can understand that people don't care about a few more bytes and cycles here and there - what's the point for a interpreter to be able to run on a bunch of kilobytes today? - but I wonder why those same people would care about a weird RPN thing that seems to be designed to make their head hurt, with or without quotations. It's only slightly less hostile for them than BrainFuck.
Computer games and computer languages have something in common: when you design and redesign them, you realize that they are a subtle network of interactions between different parts of the system - even if, in the case of programming languages you look for "orthogonality", which is the opposite; but this concept applies more to features/semantics than to syntax/ergonomics.
> The problem is already solved. You just spit out a warning message when something is redefined instead of an error.
You didn't get what I wrote. You want the system to keep quiet when you give an hint that you do something on purpose, like casting explicitly an integer into a pointer in C. You also miss the point: of course this problem is easily solved. But I argue like Chuck Moore that you'd rather solve the specific problem you stumble upon in practice, rather than trying to solve the more general problem that you imagine you will have. Sometimes when you try to kill two birds with one stone, you end up with one bird wounded and a broken window.
Yes, you were more subtle about it, calling him inexperienced rather than stupid. I see little difference between the two forms of ad hominem and that is what I wanted to point out.
> which is even less convenient than Forth's syntax
The two versions are syntactically the same. You can take a piece of Forth code and do the substitution:
s/if/[/
s/else/] [/
s/then/] if/
> You want the system to keep quiet when you give an hint that you do something on purpose
Why? No code that's saved in production should ever redefine variables, so it's a warning you'd get exclusively during interactive use at the REPL, where you'd be free to ignore it. It's easier to ignore a message than it is to tell the system not to give you that message.
Stick with this attitude and you'll go far.
Don't take it personally, as I was actually pointing what I think is a common mistake. I did a lot of mistakes in this domain, so I'm trying to share my experience here more than gratuitously criticizing a project.
I said that because that's how I see myself doing Forth a couple of decades ago. But there should not be any ambiguity here. Inexperience is the lack of knowledge, stupidity is a disappointing failure to use knowledge.
As for the ad homininem, that's a unfair because aside from that single remark, the subject of what I wrote is the feature, not the person who made it.
> The two versions are syntactically the same. You can take a piece of Forth code and do the substitution
True, but the if comes last and it is a significant ergonomic problem. For this to be not too unpleasant, you'll have to keep the quotations very short in practice, which is not a bad thing at all as "keep it short and simple" is a general good piece of advice which is specially true with "unfriendly" or "alien" languages, but one might be disappointed in terms of "usability": for instance nested quotes are something you will actively avoid because of that.
> The only exception I can think of is it being harder to compile, but as you said, that tiny bit of speed doesn't really matter on modern systems and additionally the code would likely be optimised to the same instructions in a JIT
You add complexity to the system in order to compensate for its deficiencies, exactly as I pointed out. Because it is more a glue language than a general purpose language as I understand it, a JIT shouldn't be needed.
BTW, the "modern systems" argument is a fallacy IMO, as many general purpose interpreted languages had to go the JIT route (JS, Python to make it viable for more demanding applications, Lua as well, Smalltalk, PLT Scheme/Racket to name a few). For glue languages, that's different because making up for the typical 10X slowdown of interpretation with the full speed native code libraries can go a long way.
Such is the issue with Forth in general.
> You add complexity to the system in order to compensate for its deficiencies
It is often true that to make the language simpler you must make its implementation more complex. I view this as a good thing. Having one slightly more complicated implementation is much better than many slightly more complicated programs.
True, although it is a last resort thing to do for me, or it has a very good benefits/cost ratio. IOW, where we disagree is the assessment of the worthiness of that specific feature.
Racket uses an AOT compiler these days.
The backend was rewritten and we switches to it early 2021 (since version 8.0).
I'm also confused about the "local variables are harmful" mentality. Why? They may not suit concatenative languages, but I find naming things serves as effective 'documentation'. Or is the point that these should be put in separate functions? But that also sounds wrong, because then you greatly complicate the control flow and spread everything to many places. Furthermore, isn't something like that A register 'just' a local variable (or worse: a mutable global variable)?
I find the dismissal of things like files and encryption disturbing. Many programs need some form of persistence. How is that useless? Why does 'needed data' not need to be encrypted?!?
FWIW (I don't think this answers your main complaint, but I do think it's fascinating), factor makes a lot of use of the pair "[" and "]" for control flow, but they're totally stack-based:
[1] https://www.1strecon.org/downloads/Forth_Resources/CM_ForthL...
[2] https://www.complang.tuwien.ac.at/forth/forth-77.txt
[3] http://yosefk.com/blog/i-cant-believe-im-praising-tcl.html
I understood why when I wanted to use it as a calculator. All relevant keys for simple computations are then on the keypad.
Well, Forth is flexible enough that you could define "00" to display results; the rule is that the interpreter looks for tokens in the dictionary first, and the try to convert to a number if a word of that name doesn't exist.
Keep on keeping on!
Edit: temporary fix of simply hiding the sidebar. lol.
Coroutines (which often carry their own stacks) can sort-of dodge stacks—effectively utilizing a sort of bespoke linear typing to avoid this—but I don't know of a language which doesn't have c-like, stack-based procedures or subroutines, which places arguments on the stack and registers and operates mostly with respect to that stack and foreign API calls.
A stack is certainly the most intuitive data-structure with which to implement a lambda calculus.
Really, what Forth and descendent languages seem to emphasize is a sort of a data-oriented execution rather than a symbol-oriented execution—C very much encourages you to name types, function signatures, etc, and the result if you don't is extremely difficult to parse and interpret manually, even if you can implement forth as a subset of C.
Forth emphasizes reasoning about the stack directly as data, seemingly forgoing any assistance outside of syntax. Arguably it forgoes the utility of syntax, too.
The most famous example is FORTRAN, until 1990 when they added RECURSIVE directive.
This depends on the evaluation order. I think you are talking about call-by-value (eager) or call-by-name orders.
But normal order evaluation with memoization (what human computer would do - write result only once) uses graphs and graph rewriting.
...which are still most simply implemented with a stack.
For isntance, in C:
int x(int a) {
int b = a >> 1;
int c = a + 34;
f(b);
return c;
}
This seems less meaningful than "syntax-based" . Certainly, a "variable" is not a fundamental I deal with on a daily basis in c-based languages—there are necessary exceptions cause by the limitations of the language, of course, like accumulator symbols in a reduction process or state variables in some specific algorithm. Nor do I understand any utility such a concept bings to the table—in fact, this concept brings mostly pain I'd like to forget and forbid from future projects. I refer to "symbols", not "variables".
Regardless, variables are commonly implemented with a stack and are not trivial concepts to differentiate from the equivalent usage of a stack.
Perhaps I should argue for a "syntax-oriented" language instead.
I understand that procedural languages often have variables, but any method of programming that blesses this code as "good" typically doesn't bless the usage of variables outside of looping-with-an-accumulator, state machines, or some other formal usage. I certainly wouldn't emphasize this as any positive characteristic of the language given that its unique attribute is the ability to cause bugs by interacting with it incorrectly (god forbid you have a "pointer variable"—the most curse-upon-humanity type if such a concept were to ever exist)
I write compilers. I don't know why anyone would willing make the mistake of creating the concept of "variables" again.
I'd just use "symbol" then. Symbols don't have much utility without a referent.
> Forth doesn't let you define these within procedures.
"within" vs "not within" seems like a distraction. Forth certainly allows defining your own symbols.
> Values are communicated implicitly on the stack in Forth, where other languages give them names and then use the names to say where the value goes.
Forth uses names, too. Other languages have unnamed expressions too. I just don't see the distinction you're drawing as meaningful.
It is a quite significant difference. A name bound at global scope exists with only one value. A name bound at function scope can be instantiated with different values depending on what the function is called with, and can hold multiple values at the same time if the function makes a recursive call to itself.
> I just don't see the distinction you're drawing as meaningful.
Yes, most languages are Turing-complete, so there isn't technically a difference between them. You are welcome to think the difference between Forth and other languages is not meaningful.