Persistent virtual memory is the future.
When my SymbianOS phone started dying, I found out that there is no way to bulk-export all of my notes -- only way was exporting them one-by-one via laggy user interface. I subsequently spent lots of time analyzing backup file format, and then analyzing notes file format, and then finally extracting the text portions of the notes.
Modern apps often use standard libraries for serialization [0], so it is much easier to get to your data back.
But persistent virtual memory? We will be back to Symbian world again, but this time, you won't have to reverse engineer unknown on-disk format -- you will have to reverse engineer unknown object model, and chase the pointers to reassemble the data from the tiny pieces.
[0] https://jvns.ca/blog/2018/03/31/reverse-engineering-notabili... https://news.ycombinator.com/item?id=16726189
> Phantom doesn't have files. Well, there are no files in the sense that a developer opens a file handle, writes to it, and closes the file handle. From the user's perspective, things still look familiar – a desktop, directories, and file icons. But a file in Phantom is simply an object whose state is persisted. You don't have to explicitly open it. As long as your program has some kind of reference to that object, all you need to do is call methods on it, and the data is there as you would expect.
So in today's OSes, your editor may store data in the complex structure (ropes? ArrayList of Strings? b-tree?), but when saving, it serializes data to the nice format, using capnproto, json, xml, RIFF or something similar.
In Phantom, there is no "save" step. If your editor stores text as a list of extents pointing into set of global buffers which contain strings in UCS-4 representation, this is whats gets written to disk, with pointers, vtables and all. Phantom OS magic ensures that pointers and vtables can be restored later.
This is nice for application developer (less code), but very bad for third party trying to extract the data -- you either have to run the program and ask it nicely; or reverse engineer the class structure of the program.
[0] https://www.theregister.co.uk/2009/02/03/phantom_russian_os/
From CS perspective, it's an impressive feat of engineering. They used persistent data structures to optimize in-memory operations, and essentially a write-ahead log of these same structures to optimize commits to disk. They also created "OLE" technology in the OS+libraries to support embedding live documents one in another - remarkably not unlike Phantom's goals. The trouble is, they did these in document interchange formats! Office succeeded, and the insane complexity of reverse-engineering its formats actually became a great business "moat".
But decades later, after switching the surface structure from binary insanity to XML, it still contains attributes like `autoSpaceLikeWord95` and `footnoteLayoutLikeWW8` :-( http://www.robweir.com/blog/2007/01/how-to-hire-guillaume-po...
A lot has been written assuming malice in OOXML, but you can also argue it'd require unbelievable competence to evolve their codebase to anything other than this! If you never needed pervasive dump/load abstractions, your formats map to 1:1 to your semantics, and the only way to evolve is keeping old semantics around, forever...
EDIT: I can't help copy pasting more from the last link:
- lineWrapLikeWord6 (Emulate Word 6.0 Line Wrapping for East Asian Text)
- mwSmallCaps (Emulate Word 5.x for Macintosh Small Caps Formatting)
- shapeLayoutLikeWW8 (Emulate Word 97 Text Wrapping Around Floating Objects)
- truncateFontHeightsLikeWP6 (Emulate WordPerfect 6.x Font Height Calculation)
- useWord2002TableStyleRules (Emulate Word 2002 Table Style Rules)
- useWord97LineBreakRules (Emulate Word 97 East Asian Line Breaking)
- wpJustification (Emulate WordPerfect 6.x Paragraph Justification)
- shapeLayoutLikeWW8 (Emulate Word 97 Text Wrapping Around Floating Objects)
So, build default data structures as if everything is swapped out, and add structures designed for use in memory for caching that data.
Alternatively, only use structures optimized for in memory use, and trust your OS to keep the active set paged in (easier, but can harm performance if you have much more data than RAM)
For maximum performance the application can still be aware of the different memory types. You may still have to copy memory around (from volatile to persistent state) but it's possible to avoid the cost of serialisation.