> When your data doesn’t fit in memory: the basic techniques
Uhm... Isn't it called a database?
Jokes aside, one of the core use cases of things like databases is random access to a part of data that's too large to fit in RAM.
Uhm... Isn't it called a database?
Jokes aside, one of the core use cases of things like databases is random access to a part of data that's too large to fit in RAM.
I realize that relational databases are not the right box to fit certain kinds of data into, but you have to put your data somewhere that allows it to be efficiently manipulated. What is that if not a “data base”?
Next thing you know, grandma is typing:
SELECT * FROM GOOGLE.COM WHERE TITLE LIKE "%CAT PICTURES%";
That's just an analogy, of course. But perhaps you can imagine your own list of reasons why MySQL didn't replace Apache.
The answer to your question is that we could not possibly use any off the shelf software to provide the interfaces we needed (we were writing our own). By the time we had bytes that we needed to store somewhere, our data was about a GB of complicated structure that we accessed as memory-mapped files. (Look at https://capnproto.org/ for a rough analogy to the kind of access we needed.)
And what I didn't say was that the DBA was literally recommending MySQL BLOBs, storing a bit more than 0.5 MB (512 * 512 * 2 bytes of image data) in each row, having a thousand rows or more per CT scan. The performance of that would have been absolute crap. It made literally zero sense.
However, if you are simply "pulling blobs up" of compressed (or uncompressed) volumetric data... well yeah. Don't put that in a sql database.
In-memory databases are a thing, but that's a very specialized use case.