Code as text is a problem

This sails happily along until someone realizes that they _already do that_. Code is structured. It compiles to an AST which can be worked on by any number of IDEs and editors.

You want a super rich structure in that regard? Something that makes refactoring a breeze? Well, you're asking for a strong, expressive type system.

Shrug.

Nothing to see here.

I think the problem is that for most languages, the AST is very complex, and writing a transformation pass isn't something you do as a one-off hack.

Lisp famously keeps things simple. I think we could do better than that on syntax, but designing a language with an AST designed for hacking still seems like a good idea.

But there are other ways a language can be easier to hack on. The gofmt tool can do a search and replace on Go expressions and also write the output in Go's standard format. This builds on the language's strengths: (1) having a standard format (2) mostly unambiguous references to imported symbols, so you don't need type-checking to disambiguate. This makes it much easier to find all references to a function.

I see several problems with this:

1) human behaviors: you want the ultimate "Source Of Truth" to be the code-as-data structure instead of the text but programmers would eventually just shift the SOT to the "rendered text" because it's easier to work with. Friction for editing is a big deal. A similar behavior happens with database Stored Procedures where programmers use external "schemachanges.sql", "tabletrigger.sql", etc for source code revision control. The db engine's "source of truth" of that code is an internal blob but that's not visible to Git/emacs/vi/Notepad++/etc.

2) work-in-progress code is often broken and can't be parsed: E.g. the parser stops at a missing semicolon or unbalanced quote and therefore, you end up storing a large unparsed string for text with parsing errors. If you have a 500 LOC source file where the first 20 lines was parsed but the last 480 was not, you might as well have left all 500 lines in the "rendered" text as the SOT. The ideals of maintaining a perfect round-trip from code-as-data --> code-as-text --> code-as-data ... is an illusion in this case.

Based on current macro trends of compilers exposing their parsing engine (e.g. clang, C# Roslyn), it looks like programmers would rather use "parsing as a library API to process normal text" rather than "code-stored-as-parsed-data to be rendered to text for editing".

Additionally, the tooling to work with ASTs for many languages is poor.

However I was pleasantly surprised when I discovered clang query(c++). Eli has a nice introduction: http://eli.thegreenplace.net/2014/07/29/ast-matchers-and-cla...

I wish someone would build more refactoring tools with proper semantic analysis

This is either hyperbole, or exactly the problem: It's no wonder that a behemoth of that size is hard to refactor.

The idea is to write code in blocks. So you have a raw file with and the following operations:

INSERT (create a new block of text, either top-level or nested inside another)

DELETE (Selected block or all nested blocks)

UPDATE

Each block would effectively be a node consisting of its text, optionally split into a LEFT block of text and a RIGHT block. Each line would be stored along with a line number. Then when an insert or update occurred, all line numbers would be updated based on which lines were already to the left and right of the insert, at which point the file would re-render.

I think if you added tagging and selection / editing by tagging you'd get the benefits of something structured without getting down to the AST level or breaking all of the other useful add-ins built on the assumption that code will be text.

The "IDE" that a number of people talk about, let's face it, is anything but integrated.

It is laughable to call them "integrated development environment". They are glorified text editors that go to extreme measures just to give you an ounce of useful information which is often unreliable.

Such as trying to "parse" your literal strings and use heuristics to determine if it's a SQL query. That is not integrated, it is a sad hack.

My point is we can set our bar of expectation from our programming languages and environments so much higher. Let's not just settle on what we have.

https://news.ycombinator.com/item?id=14282600

[1] https://github.com/nikic/PHP-Parser

If two languages have similar grammars and APIs you could potentially allow the programmer to choose the language they want and each programmer can work in their own language. Even one of those graphical block-based languages.

In fact I was thinking of writing a blog post about it along the lines of "The plain-text tragedy".

A program is so much more than a flat collection of lines of text stored on a filesystem.

Our tools force us to "reduce" a program and dumb it down to a bunch of flat files.

This in turn increases the cost of representing information (syntax clutter).

It also adds cognitive overhead for naming things and so on.

As a result we get ourselves into a fight with the compiler. On one hand we want to give it as much information as possible so it can help us solve the problem.

On the other hand we don't want to overwhelm ourselves so we awkwardly compromise somewhere in between (terse syntax, short weird variable names, "def" for define "fn" for function, symbols and so on).

As a result, and believe me I could write a book of rants about this, what we call "IDE" is anything but integrated.

Imagine an editor that allowed you to easily view and modify "objects" in your programs.

Instead of the project tree representing the files on a filesystem it would be representing components of your project.

You could view a function, select it in the editor and right click on its name and click "Write documentation", or "View entities calling this function", "View tests involving this function", "View authors".

In a way similar to, for example, how stored procedures are stored as objects in a database that you can fiddle around with.

And you could have custom attributes attached to these objects. You could tag them with domain-specific stuff.

Imagine the possibilities for version control. If the units of our programs where tracked and edited as first-class-citizen objects in a smart database you would automatically get "git on steroids".

We already "have" some of this stuff but they are not integrated. We first throw away information then come up with parsers that try to make sense of it again like fancy auto-complete intellisense stuff. As a result they become expensive to write and maintain and they become fragile and unreliable. You can't trust them blindly because their accuracy varies wildly. Good luck getting intelligent intellisense in a large project with dynamic languages and transpilers and so on.

We need a development environment that is not hostile towards the humans. It should actively encourage the programmer to provide and specify things about the domain.

We need programming languages that allow and encourage the programmer to provide information. The flat-file ASCII syntax that we are so used to punishes that.

In the "real world" there's so much meta-data associated to each object/entity/unit of the program.

Consider a simple function that adds two numbers together.

While from a mathematical standpoint it's nice to be able to write "function add(a, b) { return a + b; }". From a industrial real-world team-effort software engineering point of view that is incomplete.

Ideally we would want documentation (purpose, arguments, return value, etc), tests, history/version-control and some more context-specific things such as logging that can be turned on-off depending on the context.

We need languages that recognise these concepts and editors that allow us to keep these things in a way that doesn't get reduced down to meaningless text (like documentation crammed in comments that later needs to be pulled out and parsed awkwardly).

If we don't "throw away" the information suddenly so much of the stuff that we work so hard to do today becomes available for free (such as much more accurate documentation that stays up to date with the project, version control that operates at the program level not lines of text).

You could "tag" the components of your program with various attributes.

Then your IDE could auto-generate a report that says "Hey 57.2% of your functions related to authentication currently do not have a test case associated with them. Would you like to create a test case for checkUserLogin function now? click here".

As I said right now we have a lot of this stuff but they are not integrated and fall apart easily, they rely on fragile mechanisms to keep them accurate so you can never let your guard down and trust what you see on the screen you have to actively fight to keep things in sync with each other and so on.