Hacker News Reader

Comment by amelius | Hacker News Reader

Yes, but data on the order of TBs is usually stored on platters (as opposed to SSD).

In that case, a fast data access library will not buy you much, because the disk will be slow anyway.

(By the way it is common that an in-memory database is combined with on-disk storage for large blobs; those blobs are just flat files, and the OS already offers the fastest way to access them without any library)

View on HN

Multi-Terabyte SSDs are very common. An i3.16xlarge instance on AWS, for example, has 15TB of SSD storage.

Ok, but it doesn't take away the point that very often the database can still be stored in memory, except for large blobs which are stored in flat files. Access of those files is usually sequential, so you will not need a special library.

It happens sometimes. Very often? Not in my experience

Yes, video streaming and management is a use case as you describe. Metadata is kept in memory, and the video files, themselves, are stored on an inexpensive media - usually as file system. So queries about "what can i watch" are quick, but loading the video can take seconds. But any use case where you want to keep state over time - sums, averages, maxes, histories - requires you to keep all the individual points of data in the database. Those are the (increasingly common) use cases that I run into where very fast disk storage like Scylla are important.

Reply on news.ycombinator.com