Programming breakthroughs we need

Just to cherry pick a dramatized example. In the late 90s, one of my first programming experiences was in Delphi. It's a very visual way to program. You drag and drop a UI together from standardized elements. You could data bind things like input fields to a data source. You'd double click a button and it takes you to the already created "onclick" event where you do the actual coding. In a way, this is still similar to how you can "program" Microsoft Access today.

In this same period (I was in a computer science study) I was moaning to my teacher how tricky programming in C was. I made tiny memory management errors crashing the entire PC, with zero debug output.

His words: "Don't worry. By the time you get to work, you'll just do modeling".

Now I work on the web. It's been 30 years since Delphi and instead of productivity gains, we've taken steps in the opposite direction. There is no standardized UI library. JavaScript has no standard library. There's no single unified architecture. There's no standard tool chain. It's all very low level after which we stitch together fragile semi-abstractions.

I know it's an imperfect comparison, but I hope it gets the point across. It's all far too low level and fragile.

This doesn't just hurt productivity, also accessibility. It's simply not beginner friendly.

You want to do the hard stuff? Maintain existing code you're not familiar with.

The problems around maintaining unfamiliar code are huge, largely unsolved, expensive and risky. There's a little branch of computer science called Program Comprehension and no one pays any attention. Though most programming money is spent on maintenance.

My day becomes selecting libraries, integrating them, and testing integration. My business code is < 20% of our entire codebase. And I’m much more productive in 2022 than in 1992.

We need more "Excels." More and better tools that let "regular" people program.

Boilerplate is a thing that happens when libraries don't have sane defaults. It's one of the more unpleasant things about Docker et al. I wish we had a dev climate where something like Linux Standard Base could take off.

I find frameworks to be perfectly fine, as long as you break out of the "Perfectly fit the code to the vision in my head" model and go for "How do I make this product using only these high level blocks I have, without modifying them".

Frameworks are great as long as you don't fight them. Using a distro like Mint? Don't try to swap out core system services. Using a declarative web framework? Don't try to do some synchronous imperative stuff it was never meant to handle.

I get the feeling a lot of programmers really value creative freedom and don't want to feel like they are playing a rail shooter, just coding the one obvious thing the opinionated tools tell them to do, and then they have a bad time when they use tools that were specifically designed to remove any need for interesting code.

I disagree with the text criticism as well. The reason we spend time thinking and reasoning about code, is not because its text format. Its because that's the job. We aren't scribes just copying books. The hard part of programming is figuring out what to do, not writing it out. Well i am sure better visualizations or something could offer minor improvements to comprehending code bases, i doubt it will change things significantly. We will always spend most of our time trying to figure out what we need to do not doing it.

I think that the future of programming is more sophisticated static analysis. Programmers will write statements like, "every code path that writes to the Payments database must have called validate_user()." Then, the tooling will confirm that rule with every commit.

We kind of have this already (for example, Facebook's Infer tool [0]), but I think it will become much more important in the coming decade.

0: https://github.com/facebook/infer

This is something Rails gets very right. Bundler and Rake make Rails one of the nicest development environments there is.

But we can and should do better. Why is DLL-hell still a thing? Imports should specify versions, defaulting to a default if none is specified. Languages barely even consider versions today, they are almost always hacked on via magic text strings in filenames.

Why do tools like Dependabot and Renovate need to parse a million dependency formats to figure out versions? The language runtime or build system should produce that as a first class output.

Deployment and documentation are other places most languages ignore and delegate to a variety of crappy external tools.

Build Server Protocol is a good step in the right direction, but it barely scratches the surface. It would be great to see a language that prioritizes the parts of the developer experience that aren't writing code.

What's sad to me is that the modern follow-up to Ruby seemingly doesn't exist. Every hot "language du jour" which has come after Ruby has gone BACKWARDS. Lower-level, more systems programming oriented. Maybe even compiled. Static typing everywhere. It's utterly baffling to me.

"Why are you using Ruby? You should use…Rust! (Go! Zig! Fill-in-the-blank nerd hype!)"

Lol.

What I actually want is a new programming language/environment which makes Ruby look like programming pointer arithmetic in C by comparison. Something so advanced, so high level, that much of the time you're really just describing patterns and flows and data models and extensions, and then letting the computer determine the most efficient way to develop those code paths and execute them.

Unfortunately, I'm a bit cynical on this front. I believe the reason this doesn't exist is because it's at cross-purposes with programmer nerd culture. Many programmers enjoy the nitty-gritty of low-level coding. They fear abstraction. They fear "magic". They fear things like "implicit imports" or "duck typing" or "many ways to express the same method/function/algorithm" etc. because it's all nebulous and fuzzy compared to the safe confines of deterministic math & logic. "If I declare that this variable MUST BE AN INTEGER, then it MUST BE AN INTEGER. The idea you could pass me a string instead? UNACCEPTABLE !!!#%@!"

In other words, I don't have high hopes that great UX for forward-looking developers will come from present-day programming culture. For a quantum leap in DX, we probably need people who aren't die-hard programmers to engage in blue sky thinking. We need to talk to artists, philosophers, linguists, psychologists, and other experts in social & historical cultural dynamics. They can provide the insight we lack. Because for every "this is an integer, damnit!" type out there, there are probably many, many more who would see 123 and "123" and think THAT'S THE SAME THING. :-D

> We spend endless amounts of time bikeshedding the right syntax, indentation level, tabs vs spaces, or where to put code in the structure of files, but this all feels just pointless - these are all properties of text, but the text is just a tool to manipulate some abstract model of the program.

No we don't spend endless time doing this. We automate as much of that as possible. I haven't cared about syntax for whitespace for years (I mean, I care and I have opinions, but I almost never have to work on it).

One item from the list, which we can't automate, is a really important part of the job: "where to put code in the structure of files". This is an intrinsic part of the solution modelling. Often this is something you want to change as the project expands. What are the modules and objects of your application? What are their responsibilities? Does this new object now have responsibility for that thing over there? These are key questions to answer on a daily basis.

I agree with the complaints against boilerplate. Boilerplate can help decide where a bit of code should live, but it also gets in the way when changing that decision is the best thing to do. We don't teach or celebrate refactoring enough (at least, not in the places and projects I've worked).

Perhaps I should read more of the article...

Often the workflow in automating a work process is: - Worker has intuition on how things work - Specialist starts automating and running into exceptions, walls - Worker explains exceptions as they're discovered - Specialist adds spaghetti to the 'clean' business logic model they started with - This keeps going until the end result looks like what the worker expects - The worker then sees a datapoint they think is incorrect, but they can't check it because the business logic is embedded in software, code, database stored procedures, etc. - The specialist is then called back into debug and resolve the issue.

At my workplace we run into this issue constantly, where the ability to modify/adapt the business logic is lost through automation, and it becomes obscured to the point that the intuitive workers can't trust that the logical model is right.

There needs to be better separation of business logic from models so that programmers don't need to 'dig' into business processes and exceptions, and workers don't need to deal with part of their responsibility being shoved into a black box that they're not sure they totally trust or can adapt easily should a need arise.

Text is a seemingly basic, simple and even crude format. Ancient, outdated even?

My opinion is that our perception about the simplicity of text is related to Moravec's Paradox, and is an illusion.

Reading and writing text is something we take for granted, but this is a skill we all have to painfully learn at a young age, and we're so good at it that it seems transparent.

Text is very abstract, the fact that we, humans, are trained to easily parse and decipher it is the reason why it is so powerful for us.

I don't see any replacement in the foreseeable future.

I don't think so. In fact, I think this perspective is actively harmful.

A program is whatever the human beings who maintain that program operate against. I mean there are many possible representations of a program, and there are different ways to evaluate those models, but the model that most precisely expresses the AST of the logic of the program, or the call sequence of the von Neumann execution of the program, or whatever else? These models are incidental. The thing that matters is whatever humans read and interpret and mentally model. And for the moment that's source code.

Once upon a time, a lot of microprocessor code was written in assembly. Then C came along, and many of the assembly people said: "whoah, hold on, I use different calling conventions in different places for good reason; what do you mean every C function is going to use a whole stack frame even if it's never re-entered?" But now 99+% of programmers don't care about sub-kb stack frames, and probably most of them even consider C too low-level[0] to touch.

I've said elsewhere[1] that much of the ceremony of cloud sw dev reminds me of 1960's mainframes — so what is the equivalent of a "structured programming" toolchain for the cloud, that takes some higher level description of inputs, outputs, and logic[2], and then generates not-entirely-ludicrous boilerplate to mash it all together?

[0] in hindsight, C's big advantage back in the day was that it mixed much better with assembly than purer HLLs.

[1] https://news.ycombinator.com/item?id=32467919

[2] compare the "environment division" of COBOL with Terraform, and the "data division" with Swagger. I mean, we've definitely progressed on many fronts since then, but certainly not on the axis of "lack of verbosity"[3].

[3] OTOH, Conway's law suggests that multiple-team projects will always wind up with multiple configuration in multiple places.

There's a reason no one single alternative has triumphed over text. And many have been proposed!

Standards are hard. Proprietary tools lock you in. Text is universal (OK, restricted to plain ASCII, which is a big deal but has already happened). You can open and modify a text file with whatever editor of your choice, the simplest possible editing tool. Some weirdos (joking) even print source code.

That reading, not modifying, is the primary operation to be done on source code is an unchangeable fact of life. You won't be able to change anything without inspecting it first, and no representation is going to save you from that -- and it's undesirable to wish for this, in fact.

https://www.unison-lang.org/learn/the-big-idea/#richer-codeb...

Until these ideas go into mainstream, treesitter has resulted in some interesting improvements to e.g Neovim. I use it extensively for AST based selection/navigation and along with conceal I can hide/transform a lot of noise characters in text source code.

A quick explanation is provided in this video: https://www.youtube.com/watch?v=S4LbUv5FsGQ

And more details can be found at: https://github.com/kgrgreer/foam3#videos

FOAM is a modelling framework that generates cross-language boilerplate for you, but it takes a much broader view of what constitutes boilerplate than most systems. Typically, it can generate between 95-98% of a working cross-language cross-tier system.

FOAM helps you create features for modelled data. Features include things like a Java/Javascript/Swift classes to hold your modelled data, code to marshall to/from JSON/XML/CSV/etc., various GUI Views, and support for storing your data in various databases or file formats. However, FOAM models are themselves modelled, meaning they're afforded all of the above benefits as well. This lets you apply the MVC technique of having multiple views work against the same underlying data-model concurrently (say a grid and a pie-chart in a spreadsheet), so that you can choose the best view or views for your current need. When treated this way, your code is no longer text (but it can be, if that's one of your views), and you can easily view and store it in many different ways and more easily programmatically manipulate it.

The reduction to model and scope to algorithms are necessary and unavoidable. All of the criticisms made are objections to the selected model or limited set of algorithms.

That includes text as the standard "model". By contrast, IBM VisualAge in the 1990's had a true AST-based model of the code, and showed text as views. It did (some) amazing things, but people hated it because there was no way to make it do other things they wanted.

I would add the disparity between the granularity of commits and comments. Clearly, I should be able to describe specific changes in a file and provide high-level account of a long-running task without collapsing everything into one history.

Proposal: A single AST model, configurable for multiple languages, with JVM & LLVM back-ends, and automagic support for IDE's via LSP and program transforms like language conversion, slicing, and aspects.

The killer app there would be transforms for parallelism to the processor-specific vector instructions. That would capture more of the value of the oncoming onslaught of processor architectures.

the concept of the network should become tightly integrated - I should not have to jump through hoops to send network requests and evaluate their responses

handling errors should be extremely standardised in such a way that the programming language defines how it is done and all programming languages do it the same way

a programming language like Rust - inherently safe to do sophisticated multithreaded low level coding, guaranteed ot be memory safe and guaranteed not to have race conditions, but without the complexity of Rust

a programming language dramatically better at team coding - the idea of microservices is a terrible way of solving this problem

an internet that is type safe from top to bottom and all the type safety is the same typing system or at least compatible -all data and applications should talk through a ubiquitous type safe way of communicating

a much more obvious way, perhaps visual, of connecting data/functions/interfaces together - the typing systems I describe above should make it possible to just click code together

You're going to have to perceive and manipulate your program in some way. Maybe it will be great to get rid of code, but then what will you have instead?

There are some more advanced refactoring tools now available. These tools enable you to write code to detect bad code patterns and even automatically fix them. You can use them to write one-off transformations of code too. Rust has Dylint [1] and C# has Roslyn Analyzers [2]. Facebook has tooling [3] that helps writing CodeMods, enabling authors to generate changes for thousands of files at a time.

The thing I really would like to see is a smarter CI system. Caching of build outputs, so you don't have to rebuild the world from scratch every time. Distributed execution of tests and compilation, so you are not bottle-necked by one machine. Something that keeps track of which tests are flaky and which are broken on master, so you don't have to diagnose spurious build failures. Something that only runs the test that transitively depend on the code you change. Automatic bisecting of errors to the offending commit.

[1] https://github.com/trailofbits/dylint

[2] https://docs.microsoft.com/visualstudio/code-quality/roslyn-...

[3] one example: https://github.com/facebook/jscodeshift

> To be more specific - for most web projects I work on, I have a very similar yaml file for CI, Dockerfile,

Very similar, but not actually the same - of course there's always a tradeoff between control and convenience. Maybe you don't like where kubernetes, $CI_TOOL_OF_CHOICE, etc puts that tradeoff, but that's why there's a zillion tools and languages out there, because someone wanted a slightly different tradeoff

> Changing the program is the most common thing we need to do in our work, it's the reason our job even exist. And yet, most of programmer's time is spent reading or planning how to change the code.

I think I fundamentally disagree with the idea that "reading or planning" are second-class activities here. I think that one of the highest purposes of a programming language is is to communicate to other humans, unambiguously, what the program is intended to do.

I think most languages have a sweet spot for "good at communicating for $TYPE_OF_PROBLEM" - it'd be a nightmare to read the assembly code for say a AAA videogame and try to figure out if it's a FPS or RPG game, but if the intent of a program is to do a small thing in a super-architecture-specific way, probably assembly is a great choice - exactly because some other human will see _exactly_ what you mean.

If I have to work in a big complex codebase, it's great to use a very structured language like java/rust/etc where all the many, many pieces are (or at least can be) clearly named. I'll have no idea which CPU registers are twiddled when, but I don't care, and neither did the author of that code (presumably why they chose a high-level language).

On the extreme end - a tool like excel which is AWESOME for getting computations done and presenting the results, but is (IMO) not the best for communicating _how_ to do those computations. Do every single one of the cells in this column have the same formula? If not why? Is that intentional?

I thing programming languages are awesome exactly because of the extreme, explicit precision and repeatability that they allow us to communicate with, but certainly not all problems [that are currently solved with programming languages] require that kind of rigor.

This bit reminds me a lot of a concept Aaron Hsu mentions in his fantastic talk on anti-patterns in the Iversonian languages[0], a paradigm he terms "Idioms over Libraries". The idea being that because of the expressive power of APL's primitives programmers often opt for utilizing easily recalled snippets of symbols over the importing of library code to implement common functionality. This facet of array-lang culture can be seen with sites like APL Cart[1], which serve as indexes of these frequently used snippets.

I think a lot of people miss that this is one of the major advantages induced by the terseness of these languages (it's not just about code-golfing for the implicit joy of laconicism). By allowing for the direct expression of algorithms in the same context they are being used, APL and its kin enable one to tweak and optimize these idioms on a case-by-case basis: precisely the kind of specificity the author of this article is asking for.

[0]: https://youtu.be/v7Mt0GYHU9A?t=1027

[1]: https://aplcart.info/

The move from testing to static analysis is happening there, in the expressiveness of the type system, and its inferences. Functional languages are on the forefront of experimentation with new type systems.

yup.

> When you think about it this way, it becomes clear that using a textual source code is really inefficient way to manipulate this model.

Hard nope. All the attempts at different approach were a failure so far. Text so far is the best thing we have because text can be analyzed formally and computed with by computers. This is the result of the work that's being going for at least like 200 years in philosophy, logic, and math.

What we really need is to bridge the gap between programming and math as industry standard, not academical exersise somehow. But I highly doubt eliminating text as source code is it.

Testing is hard because the requirement for correctness is often absolute and inflexible. When software is allowed to be a little incorrect, then yeah testing gets much easier.

> Why not use a framework?

These 2 sections highlight exactly the problems I've been trying to solve with Molecule.dev. I agree with the author that software development has somewhat stagnated, and I believe something like Molecule.dev is the future. It isn't perfect and there's still a lot of work to be done, but I'm certain it's headed in the right direction. The codebase is currently in the process of being repackaged (it'll be an MIT licensed monorepo) so that developers can more easily play with it, and so that it can be more easily integrated into existing systems, as starter apps are not a frequent enough problem to build a scalable business from. This repackaging is taking longer than normal because all the investors I've spoken to apparently don't see the value in it (yet), so I've been looking for contract work to stay afloat. (Know anyone?)

> Testing and Correctness

> I want simpler testing

I built something else in early 2020 to address this exact problem as well, TestFront.io. I haven't touched it in a while (so don't sign up) but I may return to it eventually. It's open source. I've tried many testing tools/frameworks and none of them quite do what TestFront.io does. There is a pretty similar tool which someone turned into a very successful business (actually can't remember the name of it now), so there's definitely some value in it, but from what I saw when trying it, it's still not quite up to par with TestFront's granularity and ease-of-use. I'd like to return to TestFront some day, but for now there are bigger fish to fry.

No, it really isn't unsolvable. I saw the demo for DION[1] after following a comment[2] here, and it was close enough to some ideas that I also thought were unsolvable, until I saw their demo. They made a brilliant choice, use the model to regenerate source code. This transforms the problem into something much easier to work on.

I'm an old Pascal programmer, so I did what I always do, fired up Lazarus (the Free Pascal based IDE) and started writing code.[3] It doesn't do any actual work, except for showing off the main concept. Abstract syntax tree on the left generates almost usable Pascal code on the right. Selecting code then selects the right part of the tree that generated it. I've only got a few hours into it, and it's already doing things I would have thought impossible, trivially. There's no reason you, dear reader, couldn't do the same with your own programming language.

We're at one of those points in history where 5 people invent the same thing. The future is going to be a lot better once we can directly manipulate Abstract Syntax Trees instead of source code.

  1 - https://media.handmade-seattle.com/dion-systems/   (watch the 1st video!)
  2 - https://news.ycombinator.com/item?id=32372707
  3 - https://github.com/mikewarot/treehouse

See Kent Beck shaking with rage at the passing of Smalltalk.

This is why I always use black for Python, cargo fmt, prettier, etc. for my code.

But what is a "language"? It is a set of "legos" of different types like 'verbs', 'nouns', 'adjectives' which can be combined in many different ways, but only according to the lego-rules of syntax. Some lego-pieces can not be combined together.

The ways LEGOs can be combined is a "syntax definition" the designers at the LEGO headquarters came up with. LEGOS are a language.

So I don't think it is possible to program computers without a programming language. Even configuration needs some kind of language to give you the flexibility you need.

In the old days we had compilers linkers and loaders. They all did the wrong thing. Compilers did text translation to semantics and generated code. The linker combined code from several sources and produced a loadable image. The loader checked for compatibility and relocated an instance to an address.

Instead, the compiler should have stopped at semantics. The linker should combine semantic modules into an abstract execution model. The loader should generate code for the target machine at the target offset.

We have some of that in different paradigms today. But still we transmit far too much of lame abstraction in large wads, with all the important semantics erased.

Sigh.

There is nothing wrong with improving shovels, exercises are great.

But why are we digging holes? :)

What does using a modern subset of C++ prevent us from achieving, that we really want? Are programming tools the bottleneck of modern society?

Do we really need 100s of computer games released each year, 1000s of funded start-ups trying to become unicorns, a 'metaverse'? Do we need all that? Or would we rather have UBI, land value tax and enough sanity in society so that women choose to have children again?

You tell me :)

If you say 'we can have both', can we really?

[0] https://youtu.be/EGqwXt90ZqA

The #2 thing is documentation, which is in a very sad state right now: everyone writes boatloads of unstructured text that vaguely amd loosely explains something, and newcomers have to either parse this mess, or ask those who know for a digestible summary. This happens because text, the way we write it, poorly represents our thought models. We've learned to split text into words, words into paragraphs, and paragraphs into chapters, and we've even invented content tables, but all that isn't enough to deal with the enormous complexity of software we have today.

This is perhaps most clearly revealed here: "Changing the program is the most common thing we need to do in our work, it's the reason our job even exist. And yet, most of programmer's time is spent reading or planning how to change the code." Combined with the disdain for testing those changes, it seems the author wants a magic programming wand.

But constraints are what make programming languages great and easier to work with than machine code. If this boils down to a hierarchy of constraints the question that needs to be asked is “why are there so many different ways of constraining a program’s behavior” (via programming languages)? If there were only one way, much of the innovation wished for here, as abstract as it may be, would have probably already been worked on. Instead the base shifts constantly and the techniques for building changes everywhere.

Sometimes I wonder why all of programming can’t boil down to C code. Python would first be translated to C, Rust would be an analysis engine for C, etc. It is elitist to make programming a meta language and field of study just to understand and use effectively, even if not consciously done. The lack of standardization, compared to what could be, drives this complexity. You’ll know programming has revolutionized again when an order of magnitude more people can do it without dedicating chunks of their life to it.

There is SO MUCH CODE out there. To some degree that is a bad thing, but there are genuinely good libraries in other languages that would be nice to more seamlessly utilize in other languages.

Generally, there isn't a "best of breed" movement to actively track which implementations and approaches to things (json deserialization/serialization, http client, regular expressions, high speed I/O, algorithms and data structures.

Instead, good approaches organically move around, or are at the whim of the steward of a language recognizing and pushing for some new language, approach, etc. Often good approaches or libraries or interfaces are shut out from another ecosystem over nothing more than dogma or reactionary hated in the worst case.

Instead, if one language gets a good library, you either interface at the OS level or you have to migrate to that language if the library is that critical.

The JVM was close to this, with Jython, JRuby, Javascript, Groovy, Clojure, and lots of other languages runnable within JVM code. Ultimately, notably Ruby where the JVM was the fastest runtime for Ruby for a time, rejected that approach.

But I think it was the right approach fundamentally, if the politics got in the way. LLVM may have that potential, but it seems to tied to compilerland. OS-level interop seems generally a failure, either because of OS balkanization or other concerns.

> I feel like we will continue to run into these kinds of issues to the end of time unless can figure out: 1. How to precisely describe the properties we want the final system to have, 2. How to 'compile' from a higher-level language/layer into a lower-level language while maintaining all those properties. At the least this will require us to: 3. Define the precise semantics of each layer and enforce that every construction in and implementation of that layer adheres to it strictly.

> Layers/languages should verifiably maintain these properties through every translation via paths like: eula/privacy policy, requirements, UX, UI, state machine, source code, AST, IR, ASM, machine code. Every one of these layers should have a well-defined execution model that is isomorphic with other layers, modulo metadata. For example: source code === AST [+ comment/whitespace/filesystem metadata], AST === IR [+ name/desugaring/applied-optimizations metadata]. All undefined behavior at every layer must be exterminated by definition or avoided by construction.

[1]: https://news.ycombinator.com/item?id=25836398#25838252

Then when the software designer finds that the software does something wrong, he can simply add assertions. Its a little like TDD but without coding.

For GUI applications it would be cool to define assertions via natural language and the programming system drawing pictures to explain what it understood.

We largely build software the same way with the same tools (I'm looking at an Emacs window). One line at a time (or one boilerplate paste at a time). It's been decades since I sat down with Turbo C and discovered the ID, and decades since I drew my first UI on a NeXT Cube. Decades since Java, Python and C#. We've never found a way to program at a higher level - we're still stuck processing strings, parsing text, formatting text... all these little details that make it really hard to work at a higher level of abstraction. And all little details where each developer does it just enough differently that we can't be sure what our code really does...

I guess that is how job security works.

1. Less code. Operating systems have become obscenely bloated and restrictive. There is an enormous amount of power in the hardware we run our stuff on these days but you can't even touch it as a programmer anymore.

2. Powerful proof assistants and code synthesis. I love low-level programming but even the most hardcore, brilliant people I've met can easily introduce show-stopping security flaws into software without realizing it. It turns out that programming is hard and without rigorous models and proofs humans are really poor at understanding programs. We naturally hand-wave away details which get us most of the way there but these are discrete systems and details matter a lot.

As for testing... I agree, unit tests are often insufficient but probably the lowest bar you could convince typical industry software developers to hurdle to prove the correctness of their programs. Even then, as the author notes, they will come kicking and screaming with their opinions. There are far better options and strategies but as soon as you mention property testing or formalization you'll lose about 90% of the room so you have to pick your battles.

The trouble is that most startups are motivated to accidentally stumble upon a market niche and in this day and age that often means, "something with computers." They're not in it to write reliable, robust software that is efficient and performs well on a target platform. They're writing code that they can cash out on and never see again. Very different world.

The final breakthrough we need is to fund research labs again. Technology isn't inevitable and civilizations in the past have lost the ability to build and maintain it. We shouldn't let that happen again if we can help it.

A very good example of literate programming can be found in the fantastic book Physically Based Rendering [1]. It also has a preface discussing it [2].

The authors write a full physically-based rendering system in a declarative fashion. This approach has also led to a nicely adapted web-based version of the book [3].

Although we have successful examples like TeX and HTML, it would be very interesting to see this approach explored further by a more modern language or programming methodology.

[1] https://pbrt.org/

[2] https://www.pbr-book.org/3ed-2018/Introduction/Literate_Prog...

[3] https://www.pbr-book.org/3ed-2018/contents

I tend to write far more testing code, than the CuT, itself. Many of my various widgets have test harnesses that are, in and of themselves, full-fat, App Store-ready apps.

I find test harnesses are generally better (for me), than unit tests[1].

I was just thinking about this, this morning.

I'm what people refer to as a "crusty oldtimer." I wrote my first program (Machine Code) in 1983, so I've seen some changes in the landscape.

These days, the general posture of many shops seems to be "Our programmers suck, and can't be trusted to write code güt." Sadly, this is not necessarily an inaccurate stance.

It tends to result in highly defensive IT infrastructure and process, though, with heavy reliance on languages and strict, rote methodologies.

I used to write a lot of code in C++. Nowadays, people seem to freak out, at the thought of writing in C++. It's entirely possible to write safe, performant code, in C++, but it isn't enforced by the language. The same for PHP, which I've used a lot.

It's also quite possible for bad programmers to write truly nightmarish code in both languages. I know of which I speak, having written quite a bit of said "nightmarish" code. That's one of the reasons I write fairly good code, these days.

The conventional wisdom seems to be that we need to rely on the tools (languages, IDEs, VCS, CI/D, etc.), to assure good code.

That's fine, but it seems to stem from an unwillingness to invest in the most expensive component in a cubicle; the software engineer.

This is a long, tiresome discussion, with no one willing to change their minds, so I'm not going to follow up on it.

I did notice that this posture seemed to start in the mid-1990s. Before that, companies put ridiculous amounts of trust and investment in us.

[0] https://yoyo-code.com/programming-breakthroughs-we-need/#tes...

[1] https://littlegreenviper.com/miscellany/testing-harness-vs-u...

I don't believe relational databases are the end all or be all either. What has stood the test of time is the double entry accounting transaction journal entry. In this model, the journal entry is the system of record. And you can have multiple distributed systems creating these entries at the same time. You collect and aggregate these entries to display the "balance" at a specific point in time. To me, this is what Kafka distributed logs powerful. Distributed logs builds on a proven concept.

What is the problem? That not 100% of the code in the file is business logic or only exists in that project?

When I receive a letter, there is plenty of boilerplate there: addresses, greetings, sign offs, business hours... does that really get in the way of my reading the message, once I'm used to the format? The information is there if I need it, easy to ignore if I don't, and easy to copy-paste for the author who will update if needed (e.g. closed exceptionally or something).

Is the author running into a problem or just complaining about perceived inefficiencies that don't actually get in anyone's way?

A language is released too prematurely -> some companies start to use it -> design problems become clear, but the language is in a compatibility trap at this stage -> e.g. it becomes a swamp or it slowly becomes a monstrosity of design afterthoughts and crutches.

It probably even wouldn't require much more spending from corporations, only some initiative. They anyway already invest considerably in language&toolchain development, but often with wasteful goals.

In the same vein as the developments around LSP, TreeSitter, etc. I had the idea that one could create a scripting language dedicated to refactors. The language would make it simple to talk about the AST of your programming language and provide simple primitives for safely transforming it.

Doing this in a language-agnostic way is pretty difficult, though. There seems to be some discussion among the LSP folks here (https://github.com/microsoft/language-server-protocol/issues...) but no resolution. It's worth thinking about!

It's not perfect but defining your types so it's hard to get into error states is amazing. https://fsharpforfunandprofit.com/posts/designing-with-types... It's like having database constraints but at compile time.

I've approached building web apps as data models. It works really well for our team. This was how / why we started to develop Lowdefy [0], to express a web application with a data model. We've been experiencing the benefits of this - I really need to start writing about it.

Writing this app model in json or yaml might not be ideal start, but once you have that creating a IDE / GUI builder / next thing on top of that becomes really a simple task while reaping all the benefits of defining an program / app as a data model.

[0] - https://github.com/lowdefy/lowdefy

In mobile there are multiple tools that you can use to prototype UI behavior. They allow you to specify navigation, transitions, clickable areas, behaviors when clicked, swipe actions, etc. They're this way so that the UI behaviors can be seen before coded (to an extent - not every prototyping tool does everything).

For code, well, only formal specifications get to that level. I haven't actually seen one of those outside of software engineering books, but I presume they exist in more critical fields. Commercial software doesn't really use them because they're too much work.

It describes an idea called "wise computing" or "behavioral programming" which i'm not sure that i fully understand, but it sounds like the idea is to allow IDEs to help more, via additional formal reasoning about code in the manner of an expert system AI.

That and that everyone adopts it so there's plenty of libraries and a community :D.

Seems like Clojure will get there with beefed up REPL workflows ala Calva/Portal. Community is amazing rn but tbd how it 10-100x (if ever).

Currently GC for compiled languages like Go and D (optional) are considered as disadvantages rather than an advantages for high performance systems.

Perhaps if we can get AI enabled GC that has good performance as manual memory management in C++ and Rust it will be a genuine breakthrough for modern programming languages.

We used to write whole gui programs in an evening, compile and release them as an exe.

A while ago in a React app I needed to keep some state around so lifted it from component state hooks to a context. Very mechanical work. It feels like we should be to say “move these to a context consumed by A, B, C” and just have it all done automatically.

Is there anything inherent to React/JS that stops something like IntelliJ implementing that?

https://github.com/brennancheung/wasmtalk

Some key takeaways from the above link:

- The programmer's tool should be a tool for manipulating an annotated AST (not text)

- There should be many different types of UX's for different scenarios, each maps to and from an AST in a UX that is optimal for the developer for that scenario

- We must be conscious of human brain limitations and cognitive psychology and work within those constraints

- "Reading" and "Writing" code should have different UX's because they are radically different use cases

- Use RPN. It models the real world. Humans are designed to manipulate their environment in an incremental manner seeing the result each step of the way. When we have to plan out and write code for an extended period of time, trying to play compiler in our head, we overload our brain unnecessarily and highly likely to make simple mistakes.

- Testing should be a first class citizen in the developer experience and indeed baked into how we develop at a fundamental level that it seems strange that they are even decoupled to begin with.

* Solve a lot of these issues

* Exist

Okay, well, you just said it doesn't work, but you still want (simpler) testing?

I am working on the expression problem at this time.

Interlisp did this.

Regular users are allowed to edit their own books.

Editors are allowed to edit any books with the same account id.

The public can read about any book that is not in draft status.

I need a domain model where users can write books with the following properties: userId, accountId, title, author, status.

status is an enum supporting draft, published, no longer available

I know this seems like a completely different paradigm, but I think we were wrong about General AI needed to replace programmers - we just didn't know how "normal AI" would go without being General AI.

I can also see this extended into the front-end at least for rudimentary forms. I can imaging someone doing a transfer AI to make front-ends have a consistent and nice looking feel that can change pretty easily - perhaps to match the user making the request.

Amen.

On the backend, it's nothing new. Just use functions and problem solved.

1. Glue code and boilerplate waste

Yes, yes and yes! Glue code is the dark matter of software [1]

My hypothesis is that the reason we have to write so much glue code, and that it's always sufficiently different is that we don't have the right architectural abstractions. Specifically, our programming languages only support (essentially) procedural abstraction (procedures, functions, methods).

With Objective-S [2][3], I've been introducing language support for other architectural styles. It's been a slog, but recently things are starting to really click together, particularly the combination of "stores and streams" is very powerful and eliminates tons of boilerplate, often reducing it to the "connection operator" →

Which not only happens to be very brief, it's also generic/polymorphic.

3. Why don't frameworks work?

Also really very perceptive observations. Frameworks sort of work, but they tend to be limited by having to represent their domain in terms of procedural abstractions.

4. Non-textual code/modeling

I don't think this is the problem. Modeling is a problem, but a large part of that is because the systems/models we need to build are non-procedural, and so our textual code can only be a meta-program or meta-system that then constructs the actual system. Which is not visible/present in the code we write, but only exists in the running system and in our heads. If we close that semantic gap, having that model expressed textually shouldn't be a huge problem.

The problem is not the text, it's the wrong abstractions that our text expresses.

5. Testing

Not sure the idea of generic testing is that useful, because the key to testing (for me) is its concreteness. That said we can reduce test friction a lot, I think. [4]

6. UI construction

You don't need UI mockups and prototypes - you just program the real thing, because it's that simple. If the user doesn't like it, you can completely restructure it easily.

Yes. I've seen a lot of mockup tools, and it always seemed obvious that writing the real UI should be just as easy (or easier). With NeXT's Interface Builder, we had a large part of that, but sadly Apple has had it languish so horribly that nowadays even imperative code is often preferable, and the fluent APIs such as SwiftUI or React, Flutter etc. seem vastly preferable, despite their deep flaws.

[1] https://blog.metaobject.com/2021/06/glue-dark-matter-of-soft...

[2] http://objective.st

[3] https://news.ycombinator.com/item?id=32444300

[4] https://blog.metaobject.com/2020/05/mpwtest-reducing-test-fr...

RE: Boilerplate

A significant amount of the boilerplate he talks about comes from the web stack not being designed for what we use it for. That leads to a need for a complex three tier architecture, ad-hoc app-specific protocols layered on top of HTTP, and then lots of ad-hoc logic to convert that protocol to and from the underlying database protocols.

Consider how much less code you'd have in a two tier architecture that looks like this:

1. An app that runs on the user's machine, which connects directly to:

2. Your RDBMS, which knows about every user of your app and imposes security/privacy ACLs on them in the standard manner using views/row based security/etc.

In this architecture there is no web server, no REST, no JSON, no JWTs, and therefore also entire classes of security bugs are eliminated in one go (XSS, XSRF, SQL injection, non-transactionality triggered race conditions etc). There are also no load balancers because your DB driver already knows how to do client side load balancing, and a variety of other advantages. When you need more than just standard SQL CRUD operations you write server side function plugins for your RDBMS in a language of your choice and let the DB protocol and servers act as an RPC protocol that happens to support batching, transactions, large result streaming/paging all built in. In effect the RDBMS itself becomes the application server.

Today we don't build apps like this for a few different reasons, mostly related to the inconvenience of distributing desktop software outside a web browser. But that's solvable! And in fact my current project is a company that makes a tool that makes distributing desktop apps as easy as distributing web apps is [1]. With that and a technology like Kotlin Multiplatform + Jetpack Compose, or with JavaFX, you can write a single frontend app that runs on every desktop OS, Android and iOS (where you can write a custom SwiftUI GUI with shared business logic or re-use the Android UI code if you don't need pixel perfect native UI.

There are a few other issues that are all also solvable e.g. tunneling DB protocols through proxies, OAuth/SSO integration, streaming code to the client etc. And as a pattern it really benefits from a powerful RDBMS. But once you have the foundation of brain-dead simple app packaging+signing+notarization from your laptop, with smooth auto update to clients, suddenly you have lots of options to massively simplify up and down the stack.

RE: Text vs abstract code models.

Another big win if you go this route is you suddenly have way more language freedom. The author talks about wanting a database to store his code, but then talks about Rust, for which IDE support is limited. If you use a language with really good IDE support like Kotlin or Java, then your IDE is building a database like the one he wants already. He says it's painful to query that DB but that really depends a lot on what you know and what languages you use. IntelliJ has a structural search+replace feature that lets you do example based queries, and it also has a console and plugins that let you do on the fly queries and structural changes by writing code against their PSI API. You don't have to write full blown plugins [2].

[1] https://www.hydraulic.software

[2] https://plugins.jetbrains.com/plugin/7282-liveplugin

1. Writing glue code and boilerplate is a waste / Why not use a framework?

The author complains about boilerplate just to circularly complain about frameworks hiding "important details you care about". This, to me, denotes a failure to recognize some pretty basic trade-offs of any kind of development: you either create your own axioms and start something new or base your work on previous art by others. The more you want to customize, the less prior art you can use. He seems to want to "eat the cake and have it too".

2. Editing code in general doesn't work well / Program is not a text / Program is a model / Do we need a language?

This section makes no sense to me. Everything is a language, whatever format he desires to abstract away all formats will need a language to be coded into. Language is a tool to convey meaning and achieve communication, it does not matter if it's text, pictures, sounds, whatever. It just so happens that written language is the single most ubiquitous communication method ever developed by humanity, and the formalization of syntax and semantics are there to reduce the ambiguity which is natural to natural language. The author seams to believe that we should achieve a single natural language that conveys meaning without ambiguity, or that some other form of communication can better represent the programmer intention than text. I would love to understand exactly how, because to me it seems like a pointless argument in the way it's presented.

3. Testing/Correctness / I want simpler testing

Again, the author does not seem to understand WHY testing is difficult. To me, it's very obvious that testing is hard simply because 1. proving a complex system is hard and often impossible as we all learn in CS101 and 2. Testing itself requires conveying the meaning of the requirement correctly, and as with all communication there are always failures in this process. Once more the author seems to want a magic wand, ignore the trade-offs and, I don't know, fix all communication problems ever to have existed through human existence?

4. What is the vision?

Finally, the author tries to force upon the reader an argument that everyone's perspective is wrong, that some sort of revolution is required - and possible - to make all these desires come true. I think this misses the point by such a long shot, like the good old Product Manager that "just wants things to be done and doesn't care how". I may be the one incapable of changing my perspective and thinking outside the box, but I have a feeling that if such a revolution was possible - and most importantly - unanimous - we'd had some glimpse of it already.

Note how every single argument of the author is - to my interpretation - fundamentally tied to a communication problem. All the issues mentioned converge on difficulty to encode a message in such a way that it is interpreted correctly by the receiver, be it a machine or a person, in a unambiguous way. This problem is so much bigger than programming, so much bigger than engineering, so much bigger than Science. I'd argue that this is THE fundamental open problem of humanity, and believing that there is some sort of final solution is extremely naive.

In my day-to-day JavaScript programming adventures, I typically encounter issues around paths and how they're imported. There has been so much energy devoted to parsing a string and reading the file it (probably) points to, and we just assume that what we're importing from that file is actually what is in it. Maintaining all of my static imports in JavaScript feels like I'm doing something that a compiler or bundler should be doing, not a human being.

Go has the right idea, it considers every file in the current directory to be part of the same package. This is really useful because you basically don't have to think about file importing anymore. I love being able to compartmentalize my Go programs without having to worry about updating a bunch of path imports all over the place. It lets me just focus on the code and sort out the organization later on when the project gets bigger. Of course, when the project _does_ get bigger, that's when you start having to grok how packages work and how they're built. This isn't a huge problem but it's definitely a learning curve that you don't necessarily need to overcome when you're first starting out, so it's a bit of a skill jump.

I feel like there should be a programming language where the module system is deeply integrated with the package management system, and they are both baked into the language using syntax. This way, you can optimize those systems without disturbing how programs work, and the programs can specify what stuff they need from each package. Rather than deciding on some kind of folder/file convention for what a "module" is in your language, this one would put that responsibility on the programmer.

A package would be defined by specifying its unique name, and the code that can be imported out of it.

    export package "fs" {
      export module File {
        export function read(path: string) {
          // ...
        }
      }
    }

    import { File } from "fs"
    
    export package "my-app" {
      export module App
        export function start() {
          File.read("README.md")
        }
      }
    }

    lang --run 'my-app#App.start()'

    lang --compile 'my-app#App.start()' --output ./app

===>>> In computing, meaning is crucial. Computer source code doesn't mean anything. <<<===

Computer code obeys strict syntax rules, but those rules do not give meaning.

So far, sorry 'bout this, essentially, to date, there is only one good and significant approach to meaning -- writing in a natural language, e.g., English. Did I mention, sorry 'bout that.

In simple, blunt terms, in well written code, the comments which provide the meaning are more important, especially for maintenance, than the actual code that gets executed. Sorry 'bout that. Beyond that, beyond any doubt, far and away, in well done software, the most important part is the well written documentation. I should put this word documentation in all caps, ultra bold face, with flashing lights -- well written documentation.

Sorry 'bout that.

For my code for my startup, early on I wrote out some documentation in TeX, with the crucial core pure/applied math, data handling, etc. Then I wrote some notes about the code. The code is just awash in documentation, 100,000 lines of typing and 24,000 programming language statements. Right, on average 4 lines of typing for each programming language statement.

I had some external events pull me off the work of my startup, but now I'm returning to it and finding my old documentation just crucial, terrific. Even though it is all my project and code, I still need all the documentation, and it works great -- let's me understand again right away.

Don't believe me or argue with me. Instead, learn from D. Knuth who has demonstrated very well that he is really good with software. See what he did, and how and why, with his literate programming -- more like reading a book in English about something in engineering than just source code.

Right, long many software projects at IBM and the US DoD went through stacks and stacks of layers of planning, requirements, specifications, ..., before any code was written. Then for any change, might have to go through the hundreds of pounds of paper of the documentation to get it up to date again -- a real pain, bottleneck, boat anchor to progress, etc. Right. Need a better way. But apparently their documentation process was too clumsy or some such. On the other hand, if the software is for an airplane and it is you who is going to fly on that plane, then maybe ....

Net, the meaning is crucial, and the code doesn't mean anything. The meaning is in the documentation written in a natural language, e.g., English. Sorry 'bout that.

One more point: We can try, but I see little hope soon. We can try to design a programming language with syntax so expressive that the meaning is automatically obvious and no more documentation is needed. Did I mention I see little hope soon? Sorry 'bout that.