Hacker News Reader

I was thinking about code along the same lines: we are modeling, not writing text. This just happens to be the best way to express our models in a way a computer can be made to understand it, be formal enough and still be understandable by others.

What current languages are bad about is expressing architecture, and the problem of having one way to structure our models (domain models) vs. the actions/transformations that run on them (flow of execution).

I strongly disagree on the global variable side though...

> I strongly disagree on the global variable side though...

My thinking is that software has been terrible (over-complex) for such a long time, so its time to start questioning our most dogmatic principles, such as "global variables are bad".

Imagine you can instantly see all the dependencies to/from every global variable whenever you select it. This mitigates most of the traditional complaints.

I would argue that adequate tooling that allows for this would dramatically simplify all development. It's the only thing that matters and its so absent from every development platform/language/workflow.

If we could only see what was going on in our programs, we would see the complexity, and we could avoid it.

Another related bit of dogma is _static scoping_. Why does a function have to explicitly state all its arguments? Why aren't we allowed to access variables from anywhere higher up in a call stack?

What you realize is that all of these rules are so you can look at plain text code and (kind of) see what is going on. This is a holdover from low-powered computers without GUIs like most of programming. Even if an argument is explicit, if its passed down via 10 layers, you still have to go look.

> Another related bit of dogma is _static scoping_. Why does a function have to explicitly state all its arguments? Why aren't we allowed to access variables from anywhere higher up in a call stack?

E.g. dynamic vs lexical scoping. Dynamic scoping used to be more popular and you can still use it in some languages like elisp. In some situations it's a natural fit for the problem, but I think in most cases lexical scoping is simply easier to use.

Yeh, static = lexical.

> easier to use

With plain text editors for sure. You really need a mandatory re-imagined IDE to make it work.

You need to be able to see exactly where the variable is coming from...which I think would be a good feature anyway.

And for this you really need a live programming environment...which I think would be good too...but they are very rare. Everyone is obsessed with static typing these days, but runtime value tracing is more useful imho.

Mutable global variables online work for linear code with no concurancy.

Having code be tree like or at least a DAG is how you resolve what is probably most easily visualized as dependency hell.

This is why many modern patterns try and steer people into having their domain logic centered and compromised of idempotent functions.

And while they didn't know why back then, it is why accounting has used immutable events for over 500 years.

The notion of free variable is another lens to think about it.

> globals not bad

Globals are a tool with tradeoffs.

Even with excellent introspection and debugging tools, it's hard to prove anything about the state of a mutable global variable (since it's hard to reason about the effects of many interleaved instructions), so it's hard to prove anything about your program that depends on that state (like whether the program is correct enough), and code accessing that global must be more complicated to account for the fact that it doesn't know much about it.

Suppose you have something that's effectively a global anyway (logging configuration), isn't mutable (math constants), or for some reason you can actually only have one of (memory page table). Sure, you probably gain a lot by making the fact that it's global explicit instead of splattering that information across a deep, uncommented call chain.

Could other use cases exist? Sure. Just be aware that there's a cost.

> dynamic scoping not bad

It's not just a matter of visibility (though I agree with something I think you believe -- that more visual tools and more introspectability are extremely helpful in programming). No matter whether you use text or interactive holograms to view the program, dynamic scoping moves things you know at compile-time to things you know only at run-time. It makes your programs slower, it makes them eat more RAM, and it greatly complicates the implementation of any feature with heap-spilled call stacks (like general iterators or coroutines).

Implementation details vary, but dynamic scoping also greatly increases the rate of shadowing bugs -- stomping on loop iterators, mistyping a variable and accidentally using an outer scope instead of an inner scope, deleting a variable in an outer scope without doing a full text search across your whole codebase to ensure a wonky configuration won't accidentally turn that into a missing variable error at runtime 1% of the time, ....

Modern effect systems actually look a lot like a constrained form of dynamic scoping, and some people seem to like those. Dynamic scoping isn't "bad"; it just has costs, and you want to make sure you buy something meaningful.

I think the main problem is that we think of code as text. So the only way to determine if code is related is by parsing all of the text again. I'm not sure if a database representation is really the correct path to take, but I think we need some other way to represent code and give parts of code meaning.

Author here!

You may be interested in the programming language I've been working on :)

[1] https://scrapscript.org

reminds me of unison in some ways. did that provide some inspiration?

I was mostly propelled by a hatred for Ethereum and a love for Elm haha

I didn't find out about Unison until much later :) Cool project though!

Your ideas sound intriguing. Are they original, or can I read up on them somewhere?

As far as I'm aware, the Unison Language implements some of his ideas: https://www.unison-lang.org

I've heard the name Intentional Programming applied to this or a similar concept https://en.wikipedia.org/wiki/Intentional_programming

> Tight integration of the environment with the storage format brings some of the nicer features of database normalization to source code. Redundancy is eliminated by giving each definition a unique identity, and storing the name of variables and operators in exactly one place.

I think the key is making it feel exactly like writing plain text though with cut/copy/paste and the rest of it...which is kind of hard.

To me they felt very similar to Joe Armstrong's "Why do we need modules at all?" [0]:

---

Why do we need modules at all?

This is a brain-dump-stream-of-consciousness-thing. I've been thinking about this for a while.

I'm proposing a slightly different way of programming here The basic idea is

- do away with modules

- all functions have unique distinct names

- all functions have (lots of) meta data

- all functions go into a global (searchable) Key-value database

- we need letrec

- contribution to open source can be as simple as contributing a single function

- there are no "open source projects" - only "the open source Key-Value database of all functions"

- Content is peer reviewed

... ---

Whole thread's worth a read.

[0]: https://erlang.org/pipermail/erlang-questions/2011-May/05876...

Kinda sounds similar to unison.

How is versioning handled?

Sounds exactly what I have been thinking of.

IMHO the key is visualization of everything. Data dependencies, runtime control flow, and a centralized graph-based data store and schema.

I've worked them up from questioning the things about programming that seem most rigid and dogmatic over many years. But there is a lot of literature I have found along the way.

Intentional Programming is an interesting read as someone has already mentioned...from the guy who brought us `strHungarianNotation`. Storing code in a database but retaining the joy of the plain text cut/copy/paste experience is the key challenge, as well as all the unix file goodness.

Its quite fun to talk to ChatGPT about these topics and just question everything and delve back in the history of programming.

Also projectional editor...https://en.wikipedia.org/wiki/Structure_editor

They're both old and completely ignored. People occasionally reinvent them when they e.g. store code in DBs, or add scripting languages to their programs, or build new programming langauge because Hello World in Java is too verbose.

Unison plays with these ideas (I tried it, it's taking things in the right direction, though I still can't figure out how to write anything more complex than sorting numbers in the REPL with it; the examples are too Haskelly, IMHO.) Smalltalk language is, I believe, the original - built around the assumption that code is in the database, and coming with a built-in IDE for this. Glamorous Toolkit is trying to push this further, to give programmers better ability to create ad-hoc problem-specific views into their programs.

I've seen a few other articles written about this over the years, but I don't have any link handy.

I am working on storing code in SQLite: https://mailchi.mp/62e9b4a81f16/cosuz

this is sort of what modern ides (e.g. jetbrains stuff) already do in the bg. when im working on stuff, i almost never navigate via text or the file explorer, i use things like "goto usages or definition" and navigate via what is essentially data tracing. this only works well with statically typed languages ime, though.

the indexing step is basically building this db in the background, it's just kept out of view / hidden unless you're building ide plugins or whatever.

> via what is essentially data tracing

Value tracing is at runtime. JetBrains cannot trace how values flow through your code.

To do this, you need to instrument all your code, and track all the transformations that occur for each value. It's really difficult to do if the language is not designed for it and there are a lot of performance implications.

If your code is written in a functional paradigm it becomes much easier to trace...such as with rxjs.

At this point I'd settle for making the background DB a read/write interface to the codebase. Value tracing is nice, but there are many, many lower-hanging fruits that would improve the coding experience by orders of magniutde, and that can all be handled statically, or without global code instrumentation.

> All the issues with global variables can be solved with better tracing tooling

I would argue this problem is solved in most current languages with strict types.

Stop making all the things strings or abstract base classes.

Easy example I've worked on recently. An IPv4 address is an IPv4address in code. I don't care if it is just represented as an uint32 or string in Memory, in your code it should be an IPv4 address and if a function expects an IPv4 address and you pass it a string that is a compilation error.

Here's what I have to say about locality: https://loup-vaillant.fr/articles/source-of-readability

TL;DR: locality is likely one of the most important concepts in all of knowledge work. Of course we're a little obsessed with it.

> Our screens offers only a small window, and even the smartest IDE can’t give us instant access to everything

This is the real problem that needs solving.

> code that is read together should be written together

Code is a database of functions. This approach is like trying to design a database in denormalized form.

The "problem that needs solving" as you put it, is I believe fundamentally not solvable. Not at the human level, not at the computer level, not even at the we-made-a-Dyson-sphere post-human utopia level. Because the speed of light.

No matter how efficient we are overall, what we can access the fastest is fundamentally limited, because what we can keep closest is limited. If you want to access information in less than 3 nanoseconds, a copy of that information must be stored less than a meter away.

More prosaically, the reason our screens offer only a small window, is because our eyes only offer a small window. The problem would not really be solved even with infinite resolution VR googles.

> Code is a database of functions. This approach is like trying to design a database in denormalized form.

I believe I disagree for the most part.

How would you do the diff/apply?

Those tools are essential and they basically rely on locality (context).

Semantic diffs.

I have actually started to think that, against conventional wisdom, everything should be a global variable. All the issues with global variables can be solved with better tracing tooling.

This is an interesting premise, and actually I think we have quite a lot of examples both of successfully applying the idea up to a point and of where it starts to break down in practice.

Modern distributed applications mostly have a back end with some kind of database and front end UIs that depend on that back end via an API. Those databases are often global data stores, accessible from anywhere in the back end implementation. A lot of work is done to design and manage them, probably modelling some real world system that our application is concerned with, and there are varying degrees of abstraction/isolation used to preserve that design intent.

If the data model is simple then this works OK, particularly if you have a SQL database that can enforce some basic constraints and relationships to make illegal states unrepresentable.

What we usually see as the data models and the actions that update them become more complicated is the introduction of some business logic layer. The rest of the system isn’t allowed free access to update the state any more; it’s required to go through some defined interface that provides specific actions that guarantee the state remains valid.

That’s the writing side. On the reading side, aside from security/privacy issues, we generally don’t have the same concerns with allowing free access to the whole database from anywhere. However, often we need some form of derived data that isn’t directly stored in the database itself but instead can be constructed from other state that is. So again we end up with some kind of abstraction/isolation layer between the rest of our system and the database.

In each of these cases, there is probably data that we’re working with that is not the state that ultimately persists within the database. So the question immediately arises, if we only have global data in our programs, where does all of this transient, intermediary data go? If we put it into our database as well then all the usual problems with concurrency and integrity immediately appear, so we are back to needing something that is local to our immediate logic and can’t conflict with any other logic or indeed any other instance of the same logic that happens to be running in some other context at the same time.

We see analogous issues in the front end UI code for those distributed systems. If there is a relatively simple model then maybe the front end can effectively just fetch/cache the state from the back end API. As things get more complicated, maybe you end up with a front end data store analogous to the back end database that becomes the central, authoritative store of your front end state. And again maybe this provides some defined interface for accepting valid updates to the state and/or for accessing derived data. And again the questions arise about where the intermediate data generated by all of that logic should go if we have only our global store to hold state, and the answer is likely to be some form of more local data.

On top of the persistent state and anything acting upon or derived from it, we also have other kinds of information we work with in front end code. Many UIs will have state that is used purely to control the user’s view into the inner world: the sorting criteria and current page of a table, the current position and zoom level over a map, the last item we’re currently showing in an infinitely scrolling list, look and feel settings like whether we’re using a dark mode theme. Some of this data might apply across the whole UI while other aspects might only apply to, say, a specific table, with each table needing its own instance of that “UI state” data. So again, if everything were global, that would mean we’d need to include every possible piece of UI state for the whole application in the global store.

This comment is already far too long so I’ll just quickly note that there are other recurring themes. One is how to synchronise “global” stores in a distributed system where you might have multiple front ends running with their own copies of the state, or perhaps multiple microservices on a back end that have duplication in their databases because everything is supposed to be independent and denormalised. A related issue is how to represent temporary clones of significant parts of the data during user interactions, like building up a transaction with several changes before atomically committing it or rolling it back (think dialog box on the UI side or an internally consistent batch of changes sent to the back end), or supporting an undo facility that needs to reconstruct a previous version of the persistent state one way or another.

I do believe there’s a lot more we have to learn about different types of state and transient data and how we can model those cleanly in our systems. There are certainly common patterns we touch on in a lot of different contexts. And I think both extremes of having too much data trapped too locally and having too much lifted to global storage have their own difficulties and probably there is some sort of structure in between that would be better than what we typically write today. But it’s not an easy problem or we’d all be solving it by now…

The comment was mainly related to in-memory variables within an application process...focusing on scoping/syntax...but the thinking was definitely inspired by the fact that most apps center around an external database without realizing its essentially a global variable.

In application code, when I talk of global vars, I mean that every function has access to all data...as opposed to access being abstracted and modularized into various services which are exposed via being passed through a chain of function args, or some kind of dependency injection system.

But this global variable could actually be an abstraction (a store) allowing data integrity checks on writes.

> there is probably data that we’re working with that is not the state that ultimately persists within the database

If you think about all your data in one big graph, this transient data still has a relation to the final persisted state. There is a data flow of transient values into the persisted data values. And separately, your intermediate data structures might also contain relations to your persisted data structures.

Most dev tools don't track these relationships, and you have this tangled ad-hoc mess where data is dumped from one structure to the next.

> “UI state” data...we’d need to include every possible piece of UI state for the whole application in the global store.

Yep, this is what you should do.

If I am sorting something on a webapp and I refresh the browser, I probably want to have see the same things as they were sorted. This might vary between use case, but adding the functionality should be easy to do if necessary. So therefore it is good practice to allow all local ui state to be persisted by default.

The ui state is being persisted anyway, inside a component or inside the HTML document. Somewhere in the heap this data is stored. And if we think about our one big graph again, this data is related to other things...its just that we lose these relations.

> A related issue is how to represent temporary clones of significant parts of the data

All rendered UI values should be in their own _ui_ models (view model is similar), separate from the source of truth models.

These ui models basically allow all rendered UI to be editable without immediately committing changes to the database. This allows for optimistic UI updates. They get notified of any incoming changes from the source of truth, and can decide what to do with them.

If you want to batch them up, you just create a Batch entity, and add a relation to these ui models. The main thing is to treat the ui models like any other models. Whether they are persisted or not should simply be flipping a flag in your code.

For UI, everything should be in one big graph. Code is data.

I find with modern programming, all of the popular programming languages, frameworks, libraries, databases, platforms, really get in the way of being able to do things simply.

In this context when people say "global variable" it is effectively shorthand for "arbitrary accessible shared mutable state" and the associated risks of mutating something which other code expects to stay the same, and also often unintentionally coupling things without it being clear that they are coupled.

I'd argue that an external database is not necessarily such a thing - it can be and sometimes is, but in many (most?) applications there generally is a somewhat well-defined layer that manages that state mutation and enforces any constraints needed, isolating the "global variable" from the rest of the program, just as a singleton class with constrained setter functions could do for an OOP approach to a shared state that's arguably fixes most issues with "naked" global variables; and standard data flow analysis in specifications can make explicit any coupling between components that happens "through the DB".

Again, I like a lot of the thinking behind this. I’m personally a big fan of having clear data models and data flows, and I think there is a lot of potential in ideas like reactivity that clearly define sources of truth and relationships in single places so derived state gets updated automatically when its underlying source(s) of truth change. I also believe realising the full potential of some of these ideas will, as you’ve mentioned yourself, need better tools than we currently have. However, we can already get pretty far with these ideas, and when they work well, it certainly does make for a nice, clear, easy-to-follow design compared to having state and relationships scattered all over the code base.

Where I suspect we might disagree is the idea that everything should fit into this model. Sometimes the data models and relationships get more complicated and the one-way flows implied by the basic idea of reactivity are no longer sufficient. Sometimes the shape of the data flow graph changes on the fly, because we need multiple instances of certain parts of the state. Then we need a way to distinguish between those instances and often some way to connect the repeated parts of the graph back to the wider graph so that we can make changes to the duplicated parts but still enforce relationships with other parts of the data model.

I have never so far seen a reactivity-based approach to modelling state that handles more demanding scenarios like these gracefully, nor a convincing theory for doing so without introducing more than just a general global data store and simple reactivity. This is where I believe there is still a lot of potential for finding better ways to model the data and then better tools to implement those models. The trick, as again you said yourself, is to find the essential aspects and make it simple to describe them without the accidental complexity from the tools becoming excessive.