Hacker News Reader

Comment by t0mbstone | Hacker News Reader

Loading data from cold storage (HDD/SSD, etc) into RAM only has to be done once on startup, though. After that single slow startup, all of the additional queries happen super fast.

Your complaint only makes sense if you only intend to perform one single query against a dataset.

View on HN

My complaint makes sense because I might have 100B records per day to process, and the delay to load them up is significant

Could you stream the data into memory as it is created so that loading data isn't in the critical path?

Does this software do that?

Yes, that’s common use case for kdb+

Common for any analytical setup... you'd integrate with many of the other stream processing engines available (Storm, Flink, Kafka, Spark... all depending on what you want to do).

That assumes that we have enough RAM to hold all of the data we could ever want to analyse and that we have this data at startup. Neither of which has ever been the case as far as I have personally experienced.

My friend works for $financial_subcontractor and he told me they have an XmX of 1,46TB on their servers so that they can load the data needed for market analysis.

The response time is quite fantastic (less than a minute), but there are drawbacks: - GC takes in the order of minutes to complete - loading the data takes 90 min from spinning rust

So yeah, it's possible, but your bottleneck becomes the storage driver.

I've seen that setup in a number of hedge funds

It's not uncommon for in-memory database servers to have installed RAM in the order of multiple terabytes.

Source: I work at SAP's internal cloud team. For our internal customers running SAP HANA, we offer baremetal machines with up to 6 TiB RAM.

Reply on news.ycombinator.com