Template -> Code -> Template is one really hard part of this, and something this tool seems to take a really good approach to. If you're interested in related subjects (like transpiliation), I'd recommend this overview article as a great approach [2]. From this article: "Now, there is a single biggest mistake we see in persons trying to implement a transpiler without experience in this: they try to generate directly the code of the target language from the AST of the original language."
Question to the OP - you mention you parse the Ruby AST - do you also transform this into an AST of your template syntax before generating the template? Aka, do you avoid this sin, or is it not an issue for you?
With Mito, there is additional complexity beyond just going from Template -> Code -> Template, in that we also need to understand _which_ variables are being changed and in what way. This is necessary because a spreadsheet stores other data about your variables beyond just their current value. As an example, which of these columns in a dataframe are a result of a formula vs. being in the original dataset isn't something that is just stored in the dataframe itself.
I haven't tried too hard, but I don't think there's a general solution; it feels like it requires some sort of symbolic execution in the general case, and is tough to do well even in simple cases. Our "fail loudly and early" equivalent feels like it would be a lot higher than the 10-20% this tool can deliver!
Anyways, bidirectional spreadsehet code generation is low on the priorities... but it's a fun one to dream about :-)
[1] https://trymito.io [2] https://tomassetti.me/how-to-write-a-transpiler/
In this case the template is also ruby code, but with placeholders for the values. So both the template and the input get parsed into ASTs, then both of them are recursively walked and compared on the fly. During that walk it builds up the data structure with initial values.
In the code generation direction it walks one AST and does lookups and rewrites with values from the values.
Does that make sense?
That makes sense! Seems like a legit simplification to take advantage of.
Do you have any plans to make the template language more advanced/complex?
But if there are common situations where you end up writing the same matchers/replacers, those would be prime candidates for built-in placeholders.
As for the problem of which things of a data frame are generated from a formula and which are from normal form, Inflex compares the structure with the AST of the source, making it easy to tell that [{foo: ...}] are normal form and so editable graphically, whereas xs.filter(..) is not. The neat thing is you can still edit formulae that are deep within a normal form nested structure.
It helps that Inflex doesn’t have syntactic sugar, what’s parsed is what is in the final AST. It also has a symbolic evaluator, somewhat, so it’s fine to have a list of functions for example and edit the list. The evaluator produces the same AST, rather than an alien format.
This bidirectionality also applies to rich text editors. (https://mobile.twitter.com/InflexHQ/status/14923564133263360...)
Mito has the advantage of being a familiar language and ecosystem, but Python itself has a traditional runtime, it’s imperative and not expression oriented, and lacks sound static type information, so it’s inherently more difficult to achieve some things with it that are easier with a typed pure functional language, especially a custom one.
https://www.ibm.com/docs/en/rhapsody/8.2?topic=developing-ro...
I did a lot of code generation work in those years, working on the two dominant Mac-based generators (AppMaker and Prototyper) but was never ambitious enough to try round-tripping because of the horrors of parsing C++.
> Round-trip engineering (RTE) is a functionality of software development tools that synchronizes two or more related software artifacts, such as, source code, models, configuration files, and even documentation.[1] The need for round-trip engineering arises when the same information is present in multiple artifacts and therefore an inconsistency may occur if not all artifacts are consistently updated to reflect a given change. For example, some piece of information was added to/changed in only one artifact and, as a result, it became missing in/inconsistent with the other artifacts.
Source-to-source_compiler > See also > #ROSE, : https://en.wikipedia.org/wiki/Source-to-source_compiler
JS AST specs: estree [0] and babel's AST [1]
Parsers: babel [2], acorn [3], or espree [4]
Transformers: babel, recast [5], or jscodeshift [6]
Codegen: babel or escodegen [7]
[0] https://github.com/estree/estree
[1] https://babeljs.io/docs/en/babel-parser#output
[2] https://github.com/babel/babel
[3] https://github.com/acornjs/acorn
[4] https://github.com/eslint/espree
[5] https://github.com/benjamn/recast
Who's old enough to remember the Symantec Visual Cafe IDE?
I only used it after it was "remixed" into JBuilder. I actually used Visual Age from IBM, which I reckon was similar to Visual Cafe.
As for similar concepts, several projects by builder.io have some overlap. Most notably Mitosis[1], but I’d be shocked if TS-Lite[2] isn’t using similar techniques. Potentially Qwik[3] as well but I’m not sure, I would have bet that’s using Mitosis but it looks like that’s the other way around.
1: https://github.com/BuilderIO/mitosis
2: https://github.com/BuilderIO/ts-lite/tree/main/packages/core
I'd love to see it used for other projects/languages. At the bottom of the post I put a bunch of ideas that I'd integrate with it, if I had more time.
Feel free to get in contact (I’m easy to find) if you want to see if it makes sense to join forces/make this or something like it less of a solo effort. I have professional (open source, nothing will go to waste) and personal interest in this space and know other folks who do too.
Something like this would be amazing if it could be integrated as a generator. I'd love to do it if I had the time
I have OpenAPI pieces generated from my Go source code (comment, types, function signatures) as JSON.
I also have a manually-edited master YAML document that refers to generated bits via $ref links.
I then use openapi-preprocessor [1] (disclaimer: I wrote it) to produce a final openapi.json file which is committed in the repo.
When I want to extend the API in a spec-first process, I can add the new routes manually in the YAML file. When I do the implementation I replace the manual bits by the generated one when they are ready. When committing I can check the diff of openapi.json to verify I'm not losing in the process.
The concept may be portable - the devil is in the millions of details on which I've seen many promising tools bog down and die.
Also, please, don't say _first of its kind_ unless you've done enough research to be confident.
There are a bunch of ideas for dev tools in the original post. For example if you integrate with linters, you could define more complex code smells without all the AST juggling. Upgrading rails apps (or other frameworks that have a similarly well-defined structure) to new versions might work, by defining monocles for old the old and new version.
I find it incidentally related that you named your project "Monocle" while it does something quite similar to a concept there called lenses, so have a look at it if you have a chance!
Btw, really neat idea.
I'm not sure if it makes sense for monocle to support multiple languages, or if each language should have its own port. Someone writing monocles in a specific language probably wants to do custom scripting in the language they're used to.
Templating languages are trickier, because they usually don't have great tooling. In fact, we wrote our own tooling to convert from ERB to builder (a template system where you generate HTML through Ruby methods) and back. So for any templating language you would probably write a tool that converts from that language to builder, and back.
On the other hand, ERB is about as free-form as it gets. Templating languages that are more strict are probably easier to add. For more info on how ERB is supported, I wrote another post that goes into detail: https://blog.luitjes.it/posts/erb2builder/
I wonder it there’s a general code improver that tries this transform on every function looking for a shorter x2 than x1.
For those more experienced in Programming Language Theory, how does code generation slot into PL theory? Is there some kind of common formalism for it?