I must be missing some idea used there. Why does scripting need emulation of hardware?
The author is cleverly using libriscv for this purpose - and it is indeed a neat use of the library - but I fail to see the 'scripting' part, personally - isn't it scripting when you can just load a text file off the disk, containing program instructions, have it interpreted into some (hopefully optimized) interim representation, and then pushed through a virtual machine? I don't see the "text files" part here - but perhaps it gets more relevant in parts 2/3/4, which I haven't read yet, either...
EDIT: I heard you like ABI, so I put a foreign ABI within your ABI so you can ABI with your ABI .. actually, this started off as a joke but its kind of painful to think about. This isn't really "scripting" .. its more "embed a foreign ABI into your application so you can throw a foreign binary at it, any time, and modify program behaviour extensively "without needing to recompile" .. maybe the author should rename the article "how to embed a virtual machine into your app so you never have to write (your app code) again .."
I can see why they call it "scripting" since, this is what you usually do when embedding the Lua VM into an app - but the fact that I have to use a compiler disqualifies the term 'scripting' from being applicable here, imho ..
I'm not sure if this is a useful distinction to make.
Lua, perhaps the most used scripting language out there, runs through a compiler too and then runs a bytecode interpreter on the resulting code. Most scripting languages work this way. The compiler is still there even if it gets invoked at runtime. Most games ship the compiled bytecode files, not the script source.
The key part here is loading the "script" bytecode at runtime and then executing it in the host, allowing reloading and restarting the script without restarting the "host" process (often a game engine with long load times and lots of resident assets like textures, shaders, models etc that don't need reloading).
The host process could also include a mechanism to invoke the compiler at runtime, pass in the script as text, grab the output and then run it as it does currently. This would be identical to how Lua et al work, but quite a bit of work to set up for limited benefit.
Out-of-the-box, Lua runs through an interpreter - not a compiler.
A compiler produces native machine code from a human-readable programming language for the target platform - whereas an interpreter (in some cases) produces an interim representation (bytecode) for a virtual machine, not necessarily native - and in other cases, simply executes script code line-by-line, without the interim step.
In the case of Lua, this interim representation is in the form of bytecode which is then further interpreted by a virtual machine, pretending that it is a register-based machine vastly different to the actual hardware it is running on. This particular aspect is the only feature that the authors' libriscv usage and the Lua VM have in common - but that doesn't make it scripting.
And scripting means (to my jaded 40-years of development brain) to load text files from a resource (disk or otherwise), interpret it in some interim form, and then push through a secondary mechanism for final execution in the native environment.
>The key part here is loading the "script" bytecode at runtime and then executing it in the host, allowing reloading and restarting the script without restarting the "host" process (often a game engine with long load times and lots of resident assets like textures, shaders, models etc that don't need reloading).
So this is where our definitions conflict and go awry, because to me all you've described is runtime loading of a binary resource. Is loading and parsing a .PNG resource that appears in your executive bundle somewhere, also considered "Scripting" in your opinion? Because I don't think that is an accurate use of the term, personally. It would be scripting if the file (or memory blob) was somehow modifiable by the user, and then interpreted without further interaction required on the part of the developer - but in this case (the article) the developer (not the end user) still has to compile a binary blob, integrate it into their application, and then 'run it'.
This is more appropriately referred to as runtime resource modification and execution - for which the action of "scripting" is a superset - but the fact that there are two compilers involved in this project (native to build the .exe, and risc-v cross-compiling to build the binary blob resource for integration) means that we are far, far away from the typical scripting mechanic.
Its important to make this distinction. This isn't about scripting. It is about runtime resource modification and processing - but the different paths taken to the same fork in the road between scripting and compiling are vastly different and should not be conflated.
This isn't to say that integrating libriscv as a virtual machine in ones application execution environment is not a brilliant idea with a great deal of merit - just that the author is incorrectly using terms, and this must simply not be allowed. There is no interpreting happening here until runtime - when the binary risc-v blob is loaded and parsed by a virtual machine. If there were some mechanism to load code intended for the risc-v component at runtime, compile it, and load it for execution/report errors in the script/etc. - then I would say that the scripting workflow has been completed for this project - and it would be appropriate to use the term - but that has not happened here. It still requires a separate compilation (risc-v machine code) step, and there are still many, many aspects of the scripting workflow that are not implemented in this project ..
We can see, already here in this thread, the dangers of people mis-interpreting your technology.
Now, if you get the riscv compiler integrated into your projects such that we can indeed just load a text file containing C code, dynamically at runtime, (a la shaders), I'll revisit all of this and go along with your program. ;)
Anyway, thanks for the great series of articles - really gave me an interesting read on the way to work this morning, and I've put libriscv (as well as your sample projects) in my stack of Lab Todo's for the week .. now if someone produces LuaRISCV that targets the riscv VM instead of Lua Bytecode, that's gonna break a few molds, in and of itself .. ;) (In the context of your work so far, this seems like a low-hanging fruit, actually..)
The first thing the "interpreter" does is run the Lua code through `luac`, the Lua compiler and then feed the bytecode to the Lua VM. This may be transparent to the user, but it's still there.
Almost all interpreted languages have a compiler and a bytecode interpreter as distinct steps.
> Is loading and parsing a .PNG resource that appears in your executive bundle somewhere, also considered "Scripting" in your opinion?
Of course it's not, an image file does not contain any code that could be interpreted or executed in a host environment.
There is a distinction here on how code is executed, whether by compiling, interpreting, JITting or a combination of the above. But if you draw the line for "scripting" at consuming textual source code, that would exclude most use of Lua in gaming engines, as it is typically distributed as bytecode and may not even link luac compiler to the host executable. Yet it's commonly called "scripting" in the industry.
And conversely, if the code in this article included a mechanism to compile the source to risc-v would it meet your definition of "scripting"?
There isn't a clear definition of what's scripting and what's not, but drawing the line at consuming textual source code is not, in my opinion, a useful distinction (it's certainly a distinction) because it hardly matches what's the state of the art out there.
Interpreting has, also for decades, referred to the run-time interpretation of human-readable language into some mechanism which either a) immediately results in native execution at runtime, or b) produces errors reported to the developer for fixing, precluding the generation of immutable machine code being executed natively by the CPU (except of course with JIT, which is a form of compilation at runtime, since the interim bytecode is translated into real machine code instructions...)
Scripting has, for decades - not just in the gaming industry, which is a subset of computing enterprises - meant "loading a human readable text file containing a programming language and directly executing it if it passes validation - reporting errors, otherwise, preventing further execution. This is why we have BASH scripts and Python scripts and Lua scripts - they are not machine code, they are interpreted and either eventually produce machine code, or run through a state device pretending to be a non-native machine.
fwsgonzo is literally compiling C code into machine code to be executed on a pretend CPU (in the form of an embedded libriscv).
The code is not interpreted (from one language to another) - it is executed in a virtual machine environment. Bytecode is, on the other hand, always interpreted from one form to another prior to execution (except JIT, it is only interpreted once, compiled, and then exists as machine code always)
fwsgonzo's binary blob is executed by a 'fake' CPU, it is not interpreted.
If he had the execution environment set up, he could literally run that very same machine code on real RISCV hardware, unchanged, without translation or interpretation.
It is not, therefore, interpreted in any sense other than by his embedded RISCV emulator - and thus it is not scripting!
This is the point where the distinction is important.
>The first thing the "interpreter" does is run the Lua code through `luac`,
The first thing the Lua interpreter does is validate the inbound, human-readable script to make sure the program is correct, syntactically and otherwise - giving errors to the developer if errors are encountered. Meaning, at runtime, errors can be caught and fixed by the human developer.
Only then, once it has been validated, is it processed into an interim representation as bytecode. Sure, Lua can be told to produce bytecode in order to speed up runtime loading and execution, but its still an interpreted bytecode. It hasn't been compiled into a native form for direct execution - it still requires a secondary native program (a VM host) in order to provide any functionality.
From the Lua 5.1 manual, Section 2.4.1:
...
Chunks can also be pre-compiled into binary form; see program luac for details. Programs in source and compiled forms are interchangeable; Lua automatically detects the file type and acts accordingly.
The difference is, prior to Lua's internal compilation of the script into bytecode, validation of the programs correctness is performed, preventing the developer from producing incorrect bytecode if there are errors, and this is the process of scripting - distinct from compilation - because it is a more direct human/computer interaction regarding the correctness of the script language as it is being used by the developer.
>There isn't a clear definition of what's scripting and what's not, but drawing the line at consuming textual source code is not, in my opinion, a useful distinction (it's certainly a distinction) because it hardly matches what's the state of the art out there.
I disagree with you, there very definitely IS a clear definition of what is scripting and what is not - one must not call it 'scripting' unless runtime loading, validation (or not), interpreting, and subsequent execution (whether on an interim bytecode or otherwise) is happening - and that is not what is happening here in fwsgonzo's project. He is compiling binary blobs, and his eventual execution environment has nothing to do with the validation of the scripted code. The C compiler he uses in a prior step does, however.
Scripting is a development workflow where compilation/linking/building is not required - results are immediate, the program reports validity of the loaded script, or otherwise, and executes directly upon command by the developer.
Additionally, fwsgonzo has also, already confirmed that this distinction is correct (see elsewhere in the thread) - that in fact interpreting of a script is NOT happening in any sense other than the RISCV VM is interpreting pre-validated machine code (not bytecode!) in the form of RISCV instructions.
That machine code is not a human-readable script, and it is not bytecode! It is machine code, produced by a compiler.
This distinction is important because that same machine code could, theoretically, be executed directly, unmodified, on a RISCV machine - that is not the case with Lua bytecode, ever, which always requires interpretation by a VM.
fwsgonzo's project is entirely based around pre-compilation of native binary resources, not interpretation. A human-readable language is not being interpreted by his runtime.
There is no opportunity (at runtime) with his project to have the runtime report on errors in the human-readable script - this has all been done with a prior compilation step.
>but drawing the line at consuming textual source code is not, in my opinion, a useful distinction (it's certainly a distinction) because it hardly matches what's the state of the art out there.
I disagree with you. I think you are taking an aberration of the gaming industry, which is rife with such things, and applying it to computing as a whole - and this is where the protest over your position lays.
It is absolutely important to make this useful distinction, because in case a) INTERPRETATION: an interim representation is being generated after validating program correctness/conformity to a language spec, and in case b) COMPILATION: no such validation is occurring once the compiler produces runnable machine code.
> Yet it's commonly called "scripting" in the industry.
I have 40 years of experience with this subject, including working on game engines and realtime (scientific) use of Lua in high-performance environments, and the phrase "scripting" has always referred to the process of writing code that is immediately interpreted by some program before resulting in actual runtime execution if it is found to be valid code.
The interpretation may not even work as desired, but the host environment/program will still continue to be executed in a runtime context.
The fact that compilers "interpret" code before producing an interim representation that is then used to produce static machine code, is entirely why the distinction must be made: because there is a point where code can be incorrect and thus not runnable!
In the game world, "scripting" means 'writing some human-readable script and having it interpreted at runtime, without involving a native compilation/linking/building/packaging step" - and as this is not what is happening (key word: without) in fwsgonzos' project, it is incorrect use of the phrase.
He is compiling resources which can be loaded at runtime, bypassing the native loading/linking phase, and it is therefore more appropriate to refer to what he is doing as "runtime resource management" which, incidentally, feeds into a VM. This is not scripting, although scripting often necessitates the same activity - errors in the code are not reported at runtime, by the process doing something with the resource.
Please just use the right term and stop trying to justify an incorrect usage - don't re-define these terms based on a misunderstanding of the mechanics involved. There are literally decades of examples which are entirely counter to your misunderstanding out there, in the state of the art.
(tl;dr - Is /bin/bash a compiler? Is /bin/python a compiler? Why isn't gcc exclusively referred to as an interpreter? Is clang an interpreter? Is ld a scripting environment?)
Maybe I'm nitpicking, but to me scripting is just programming/extending a certain fixed piece of software to execute instructions not already programmed into it without having to modify that software itself. Usually using an API. Not unlike computer programming, which aims to make a piece of hardware execute some instructions without needing to alter the said hardware. The main difference is conceptual (the platform targeted: hardware or software). If the script happens to be run by a VM emulating a real processor because your software includes such a thing, I think the distinction becomes purely conceptual.
Now, the technology is obviously super cool, but what I don't quite understand yet is what is the best use case for this? Is it really game scripting? Or compiling C++ on the fly?
It's not exactly a simple drop in replacement for a Lua interpreter.