Dion 2020 IDE Research Demo

What I need for a metaprogramming tool is a code generator that uses CodeQL as input, then allows me to specify very terse AST manipulation/transformation functions, then generate my output code for me.

The one real flaw about CodeQL imo is that they don’t let me generate code..

Like suppose I found a very nice library whose only fatal flaw was that it only wrote its output to files as opposed to general buffers/streams. And I had another library that was godawful but it had a very nice, modular, network streaming routine.

I want this:

    from nice_library, awful_library
    select
    concat(nice_library_routine_upto_filwriting_entrypoint, awful_library_routine_networkIO_entrypoint)

It’s likely gonna need more information to make it possible but this is the sketch of my intent

In fact a gui like Scratch may be just the type of graphical user interface that would make this nice.. think colorful affordances that prevent you from writing a buggy codeql query and iterating pointlessly when everything could be visually typechecked. You could even call it Frankenstein :)

Would that become closer to reality with this?

I can see where they are going with things like Metadesk, but it's ultimately just another file format (text or otherwise) to represent rich structured data.

The endgoal is probably to allow neater refactoring opportunities and raise the level of efficiency by standardising on an easier-to-parse-into-AST-like-structure syntax, enabling generic tools to be built that will deal with such structures directly.

While I sense that there is something smart that could be done in that area, there are already plenty of de facto standardized languages for structured data (most notably XML), and already a bunch of real world programming languages that work.

So I think it'd be easier to grasp their goals if they started off of an existing language ecosystem (that language spec is the definition for their AST parser), and attempted to build the tools they want: this would have more quickly formalized what's missing in the source code format for it to remain human editable, but at the same time easier and richer to parse.

Again, it's all a bit vague since you can build a development environment based on text files where you are not seeing all of that generated text at all times (if ever).

As the goal seems to be to find improvements to the process of program development while being unconstrained by complexity of (re)parsing text files repeatedly, I would have started with trying to add those improvements and ignoring the slowness/cost of parsing (or simply used an existing language that's cheap to parse, like Lisp).

If you want to see their demo that made a pretty big splash it's available on Handmade Seattle. [0]

Usual disclosure that I am the organizer for these things.

[0] https://media.handmade-seattle.com/dion-systems

I think this is pretty cool.

I've been thinking about this "tree" in context of presentations which are structure + content/facts + layout/visualization. Basically all presentation content is just a big tree/hierarchy of facts (text and numbers) and some applied layout.

Many of the problems are similar:

- you want to look at the problem on different levels "high level" story line, low level drilling into section 4.3.2, seamless zoom

- you'd want to have different representations of the same information. During problem solving the information you are collecting is constantly growing and it's a back and forth between adding data, shaping it differently (lists become tables etc.)

- easy entity/renaming, search and replace for facts + audit trails "where does the number come from" etc.

- CSS / re-styling (why does it matter if important words are blue or red, that should just be a CSS theme you switch out)

- there also exists "structs" which are structured entities like say "these are the 40 offices" the client has, and we want to represent them in various forms

- ...

I wonder if notion could be turned into a modern problem solving / result presentation maker. It's very flexible in surfacing the same content in different formats.

More related to software development and their demo: Is any additional information in the tree, that is not /can not be in the text rendering.

I get that the text rendering can be "loss-y" meaning that if it's badly rendered, information can be lost / the text has errors.

But if you have the information, and you render it into a certain text representation that is error free. Then the program should also be able to parse it back into a tree?

What are the situations were storing it in "tree form" are actually superior?

Isn't the "tree" just a more compressed, less error prone and stricter text?

I like seeing raw ambition in these projects, my reading is that this is not "let's make a better IDE" but "let's fundamentally change how software tools interact".

There are numerous projects that I have worked on that would benefit from semantic code storage (and have used LSIF[1], which only goes so far).

I think the demos are neat but really don't hit on the massive productivity increases, this is the stuff that people would throw money at :

- Language-aware-yet-independent and polyglot build systems that don't need new frontends for new languages.

- A CI platform that doesn't need configuration

- Semantic fuzzing

[1] https://lsif.dev/

Sounds a lot like sexprs(/LISP), something they seem to avoid mentioning. Overall I'm not convinced about the need for new file format, I don't really believe that parsing is a bottleneck in IDEs (unless we are talking about C++...), especially with tools like tree-sitter or rust-analyzer. Furthermore parsing should be easily cacheable if performance is a concern. Going the other way, tools like gofmt etc already pretty much standardize a canonical serialization format for any particular AST.

That being said, I'm fan of the concept of structure editing etc, so I'm still interested to see what they come up with.

I'd much rather they do this with text and then use something like tree sitter to do the zooming and separation thing because I've always thought that would be useful, too.

Looks neat, not sure how practical it'll be.

it looks like they want a code representation better than text

tbh basic text is a fine storage tool, we just need better incremental analyses and tools to display and manipulate that text as more