Hacker News Reader

Comment by jakozaur | Hacker News Reader

Servers grow much bigger. 256GB of RAM is pretty much a standard rack server on a major cloud. On AWS, ultra memory with 24TB of RAM is one API call away from you :-). The article took note of that, but tooling on one node puzzles me.

So, multi TB can still be a single node with DuckDB. It's also rather a small ClickHouse cluster. It sounds easier to use DuckDB than the proposed tools.

In many use cases, if it's not 24/7, you can use a data lake (e.g. Iceberg) and query it if needed. Databricks seems a way to go, since the author uses Spark.

View on HN

Iceberg is working hard to support pure python[0] / rust[1] workflows without Spark. Following Tabular's acquisition [2], I hope it still moves in this direction at the same clip (very impressed with the speed of development on pyiceberg).

We're using iceberg + duckdb to power analytics in our app[3] and I'm really happy with the combo.

0 - https://github.com/apache/iceberg-python

1 - https://github.com/apache/iceberg-rust

2 - https://x.com/thisritchie/status/1800522255426072647

3 - https://www.definite.app/

You can already have it in Delta with Delta Rust and Python bindings: https://github.com/delta-io/delta-rs

yes, we're evaluating Delta. We went with Iceberg out of concern that Delta was too closely tied to Databricks.

Following the Tabular acquisition, the decision is murkier.

What do you mean by “not 24/7”? Why would a data lake go offline?

not sure, but maybe it's possible only to update the model in a specific time? are there other uses to the data apart from learning and validation?

Probably in the sense of "you're only running the really heavy analytics queries once a day/week/month", and thus don't constantly need the 24TB instances.

It's pretty common to run analytics workloads on a fixed schedule. I've setup anlaytics workflows that ingest data every day/week/quarter and generate a report once a day.

With workflows like that something like DuckDB can be amazing

It is very common that the majority of work happens during business hours. In some setups it's easy. It's better to get beefy instance during that time and avoid having it running all the time.

Reply on news.ycombinator.com