Basically it all exists because of storage devices:
- large transfer factors (spinning platter hard drive track access, flash erase blocks)
- latency (spinning platter track to track seek, and rotational latency)
Once you have fast random access, it's data structures and algorithms: forget the database cruft.
For once: you _always_ have to deal with memory hierarchy as there is no way to fit all data to the fastest memory. So you need to design with this in mind.
Secondly: the point of databases is not to implement a single specific task efficiently but:
a) provide means to maintain data so that it is easy to implement many different tasks (both current and future)
b) to safely _share_ data between various tasks/programs/users
Nobody ever rebooted their machine because it became unusable due to their L1 or L2 cache thrashing.
You really have to care more about that particular hierarchy level transition, RAM to external storage, more than the others, because traditionally the transition is very steep, like suddenly hitting a brick wall of slowness.
The higher levels hierarchies can be left to generic management mechanisms, like the hardware itself or compiler register allocations. The ram to storage transition is managed at run-time by software. Databases have to bypass the operating system version of that management, and provide their own, because it is so critical.
One of the key motivations for relational was to have a standardized system for data storage so data consumption was simpler.
Prior to relational every system had it's own unique method of storing data (e.g. your pointers and structs, and other things etc.) which required unique custom programming for every new data consumption use case.
With relational, there is a simpler (not simple, but much simpler) method of access that no longer requires a developer write a custom program to get the data for each new request.
I mean, yeah, but in practice you have to provide ACID or other invariants, you have to provide distributed processing and storage, you have to provide replication etc., etc.
Also, "once you have fast random access" sounds like "once you have gajillion dollars". I too want a pony!
"Once you have fast random access" happens just from moving up a step in the memory hierarchy; that's what I mean.