Hacker News Reader

Comment by TazeTSchnitzel | Hacker News Reader

Blender (the 3D modelling software) is a fascinating case of this. To make loading and saving files fast and lossless, it uses identical on-disk and in-memory representations. In other words, everything is in an arena, and saving and loading is just a memcpy of the entire arena. Considering the massive potential complexity of a Blender project and the myriad issues you could have with serialisation and deserialisation, this seems like a great design to me.

The trade-off of course is that your data structure design is kinda sticky, insofar as you need to be able to open files for older versions.

View on HN

> The trade-off of course is that your data structure design is kinda sticky, insofar as you need to be able to open files for older versions.

The missing piece is a way to evolve those data structures over time, in a similar way to database migrations. Only when loading data from an older version of the app those operations would need to be issued, and then the app could save the updated version in disk immediately to avoid incurring that cost again.

An example implementation of this concept is this project [1] that was created in the context or CRDTs. If it is not directly applicable, at least it should be a good inspiration.

[1] https://www.inkandswitch.com/cambria/

I once worked on a very large commercial application that was based on a home-grown I/O framework which operated on similar principles.

It was a complete pain in the ass. You were constantly future-proofing your data structures because you knew you were going to be stuck with them for all eternity because the I/O framework was going to serialize them verbatim whether you liked it or not. Those were dark days...

I don't get it. Surely if you ever needed to, you could write an interpreter for loading tbe older files if you decided to abandon this 1:1 representation?

I think the effort involved disincentivizes against it, and nudges people towards defensively adding spare fields or whatever.

The other comment pointed out that you can make a fall back migration code path that migrates over older file versions. That's the escape hatch if you have no other options

You ever hear of .doc files?

https://learn.microsoft.com/en-us/openspecs/office_file_form...

Yeah. There is a reason XML was seen as the future back in the 90s.

Custom databases often get extended into custom filesystems. And these systems on top of systems get obscure features (like embedding excel sheets inside of Word) and... Thing get hairy.

Wasn't that also the way Word used to dump structs onto disc in the "doc" days? I think I read that way also a major problem in writing converters, you baobab needed to decode Word's internal data structures which were of course undocumented.

I remember Spolsky saying Excel .xls worked like this, but I guess this should apply to .doc files as well.

This is usually orders of magnitude faster than serializing to/deserializing from a different storage format.

I don't think so. Serializing is IO-bound, if your format is "simple" (e.g. CBOR) you can't measure the difference.

All this look like BS to justify laziness to me, and makes loading/saving operations fragile and setting in-memory structures in stone.

> Serializing is IO-bound

Once upon a time, DMA did not meaningfully exist in the x86 world, so an IO-bound task was also CPU-bound.

Yes of course, but the original example was Blender, running on today's hardware. IMO this is just sloppy engineering in this case.

Don't forget that Excel file format is literally 30 years old. When you have 4MB of RAM in total for your OS, Excel and whatever part of the file you can fit into the rest and your CPU is an i486 if you're lucky, minimizing parsing starts to make lots of sense.

the cbor implementations i've had the misfortune to have to use are ridiculously slow, as slow as json parsers, perhaps because everything in cbor is variable-length

just because it's unreadable in a text editor doesn't mean it'll be fast

It's very common in old software and video games. Pre-97 office did that though with more formalisation than most (the binary format is now called Binary Interchange File Format, or BIFF, and is relatively generic, at the time MS almost certainly had utility libraries for working with BIFF).

This was a common source of portability issues for game saves (as well as blowing your saves on updates), as they'd commonly just blit internal data structure, with no formalised interface.

Yep, pretty sure Valve's brief experiment with a 64-bit Source engine in late 2005 resulted in save games that didn't work with 32-bit Half-Life 2.

And potentially photoshop.

This is exactly how a PlayStation game I worked on worked and because everything was signed they were in no way worried about attackers. If you were able to alter the files on disk you had already owned your device. It meant loading was absurdly fast we pulled everything in the memory and then just patched pointer locations to where they actually got loaded

I believe Microsoft Word originally did this, and it was a major pain as the file format evolved.

It also means you can't share files between little endians and big endians.

Is this a real problem though? It seems to me that any system you would realistically want to run Blender on will be little-endian.

It isn't a real problem. Blender supports only little-endian targets and the code byte swaps data files as needed. I guess someone could port it to AIX, z/OS, or Linux/zSeries, but that's unlikely at best.

There's almost certainly byte order+version markers in the header, a "happy path" that just memcpy's the structs, and a "safe path" that does byte order swapping, migrates the old structs to the current format, etc.

You can version the structs/loaders by keeping the relevant header files in their own version subfolders, and copious usage of sed. I've done this in a Django project to maintain support for ancient untouchable clients using an old version of the API, it's pretty manageable.

I checked the repo. Blender detects files with a different endian and byte swaps if necessary.

It doesn't, you just need to be careful to always read and write properties via accessor functions.

You can have the benefits of fast save and load without having the same representation. Stream changes to disk.

Storing changes and the originals blew up badly for Microsoft Word in the late 1990's due to privacy concerns.

That is of course not Kosher at all. Compilers are generally free to layout and pad structs in whichever way they prefer, meaning that two compilers for the same platform might very well use incompatible layouts. It can lead to major problems if structs saved by say a 64-bit version of the program is to be loaded by the 32-bit version.

Compilers (for C and C++) typically provide mechanisms to control padding and other layout requirements as necessary to make this work.

It's not that unusual in C and C++ code to define a struct that has a specific and well-defined memory layout. It's kosher, as long as you accept that you're working with a specific set of real compilers and use the appropriate #pragmas or other controls as needed to avoid undefined behaviour.

Yep, but then you force the compiler to choose a specific struct-layout which may be suboptimal if, for example, coding for simd. You also can't control the values of the struct padding bytes which may cause very interesting bugs.

Reply on news.ycombinator.com