Orthogonal Persistence
github.com
github.com
The object-orientation was actually pretty unimportant, (except for those products that brought in persistence via inheritance -- so not really orthogonal). No, the point was adding a new storage class to programming languages.
I worked at Object Design, and we had (IMHO) an incredibly elegant approach. In our approach, persistence really was orthogonal to type, for C/C++. If you want a FooBar, you would write "new FooBar(123)". That gives you a FooBar in the heap, disappears at process end (or on deletion), etc. Or you could write "new(db) FooBar(123)", and then on commit (we had transactions of course), the FooBar would be in the database, and accessible by other processes.
A page-faulting mechanism would bring in pages containing locations that your program referenced. That itself was very elegant.
But the really beautiful thing about this architecture was getting it to work in a 32-bit address space. We did some clever things about mapping portions of the address space during the faulting process to make things work transparently. (This problem pretty much disappears with a 64-bit address space.)
Separate from all that, we had a collection library, integrated with an OO query language. E.g., you could have a collections of widgets in your database, write "widgets[: weight < 0.01 and !strcmp(color, 'red') :], and get back a set containing the qualifying widgets. We also supported 1:1, 1:n, and m:n relationships, which would maintain pointers and sets of pointers in both directions.
It was a "database system" because our VCs wanted it to be. But it really wasn't. It really was a new storage class for C/C++, and later, for Smalltalk and Java.
Object Design also had a spectacularly talented group of engineers, many of whom came from MIT AI Lab/Symbolics.
(the data you'd like to keep is more volatile than you'd wish, but the data others keep on you is much less volatile that you'd wish)
when we express state directly in programs, we gain a lot, but our notion of trashy disposable execution goes away and now we have to think a lot more about how that system evolves.
Volatile memory is at this time merely an outgrowth of the uptime of the system. Back when people routinely turned their machines "off and on again", it became part of that convention. But now uptime can be measured in years, and even personal laptops can enter and exit suspended state for weeks on end without clearing volatile memory.
What we have developed in software systems to accommodate this on long running processes is garbage collection.
If the volatile/non-volatile distinction had never developed, all that would have happened is that R&D into garbage collection would have been more intense, and earlier.
In fact Lisp had garbage collection from day 1.
Systems like Smalltalk were also built from the ground up on an image-based model where all reachable state was persistent.
In other words: transient data does not necessitate volatile memory. It necessitates garbage collection, though. (And likely also a distinction in programming between "performant" memory areas and non-performant, assuming our NV storage is the latter.)
In a way, programmers having to deal with their garbage upfront and not relying on "have you tried turning it off and on again?" could have created better software engineering practices earlier? Maybe?
despite this, virtual machine checkpoints in qemu work well enough for many purposes
https://news.ycombinator.com/item?id=32314814
But the most recent one is also interesting: https://news.ycombinator.com/item?id=38527437
> ...
> Based on disk latency, we may target say a millisecond as duration before which to commit the current transaction. When the timer is reached, the transaction is delayed until all current atomic sections are completed; and (possibly after a grace period) new atomic sections are blocked from even being started, until after the transaction is committed.
Maybe I'm reading this wrong, but the limitations on transaction duration seem to be disqualifying for real usage? If it's not possible to run an atomic transaction for longer than a few milliseconds without bringing the system down?
Less ambitious than TFA overall, I grant you.
"We present the Aurora single level store (SLS), an OS that simplifies persistence by automatically per- sisting all traditionally ephemeral application state. With recent storage hardware like NVMe SSDs and NVDIMMs, Aurora is able to continuously checkpoint entire applications with millisecond granularity. Aurora is the first full POSIX single level store to han- dle complex applications ranging from databases to web browsers"
I have some experience with a custom data runtime where the persistence is orthogonal to the data model, with silhouettes reminiscent of the described solutions in many of the features of my system, including multiple orthogonal/model-agnostic persistence backends, automatic data synchronisation, persistable executions, automatable schema changes, automatic reactivity.
This direction can indeed bring about great savings in various parts of development; however, it seems to me that more subtlety than indicated in the post is required.
The programmer must be provided with ergonomic means to give denotations for things like when and where to persist, in order to reduce data movement, and to keep the system performant (this does not violate orthogonality; we may specify e.g. to persist at the logical location, say, in the cloud, without having to specify the physical persistence). For instance, considering the case of schema changes, unless the system bundles its language inside the database, for performance sake, to perform such changes in an "Orthogonal Persistence" system external to the database would take an completely disproportionate amount of time relative to using SQL in the database. The data runtime I work with uses the idea of lenses (where valid lenses would necessarily be reversible) to allow for coherent, undoable schema changes, but I still resort to SQL for regular (eager) migrations (the lenses system for schema changes can still be useful for migrations applied lazily).
To be honest, such experience as I've had with automated persistence has generally actually strongly convinced me of the opposite, that it is a positive good that we do not get persistence everywhere. Consider the understanding that we get from functional programming that state is generally dangerous and to be carefully managed. Pervasive persistence fights hard against that careful management. Now state is not just in your program up until the OS process is terminated, but it's all permanently and automatically persisted. You get a huge new class of bugs involving path dependence on what bits of code were running across what bits of state when, and who ran which versions, and you hit them all the time, and they are nightmares to debug. At least when the program has the courtesy to completely cease existing and leave some particular concrete bit of state behind for the future, and then run through your code to load it back from that location, you have boundaries, and procedures for minimization and reconstruction. I actually shy away from too much automated persistence, and also have a very skeptical eye on the ever-present promise of memory that is as fast as RAM but persists like SSDs... I rather expect the computing world will discover that "rebooting" is not just a crutch, but actually a pretty fundamental and useful tool. However much in theory your software should never need it, in practice it's just too useful.
That said, best of luck to those jousting with this windmill. I'm not saying don't joust, people in general probably don't joust enough, I'm just saying, learn the history of why this hasn't worked before and learn the challenges. Success is at the very least more likely if one learns from the previous efforts.
Your homework, should you choose to do it, because of course I can't make you and don't really care :), is to write a "Haskell theme" for Python. The theme must retain *args and **kwargs capabilities, as well as all other Python capabilities, though those two things will be one of the first major issues you hit. On the flip side, write a Python theme for Haskell. (This one based on my own stabs at it doesn't have such problems with the capabilities of the language, but it sure does take all of Haskell's elegance and wrap it in the grace of an elephant with a sprained leg.)
Also, bear in mind my claim is not that it can't be done. My claim is more like, nobody would want to use the resulting language with the resulting "skin". Languages are not all the same. Even dynamic versus static imperative languages don't really "skin" very well; compare idiomatic Ruby ActiveRecord-based code with Rust code. The differences are not just skin-deep.
I fail to understand the section about why transactions are unmodular. I've never encountered transaction code where the initiator of the transaction would affect the computation; could anyone elucidate this?
If you start a transaction when the calling code already started a transaction, then either you get an error because nested transactions are forbidden, or a reference counter is incremented for the transaction, so that when you close your inner transaction, no commit is done at that point, and instead the commit is only done when outermost transaction closes.
This latter case means that you don’t know when your inner transaction really commits, and also if you perform multiple inner transactions and the later one fails, the earlier one will implicitly also be rolled back, because they are all really just one shared outer transaction.
Of course, you could use separate database connections with independent transactions, but then you get into deadlocks or other problems when you really work on the same data.
So you can’t have modules that build on each other, while each being able to use transactions independently from each other. Transactions don’t compose in that way.
You would basically have to “color” every function based on wether it may perform a transaction or not, and within a transaction block you would only be allowed to call functions that don’t themselves perform a transaction. It becomes more complicated when you have transactions that are not lexically scoped, but for example live in an object.
do $$
begin transaction;
update page set change_count = change_count + 1 where page_id = 1;
if (select change_count = 100 from page where page_id = 1) then
rollback;
commit;
$$
(quoted from the main readme)
I bet there's some fun attacks waiting to happen, related to watching for specific access patterns. Avoidable I'm sure, but I imagine it'll require awareness from application developers.
1. Please use an AEAD!
2. IIUC, the current design exposes the hashes of the data values. This seems undesirable and I think you can avoid it.