Hacker News Reader

Comment by hipadev23 | Hacker News Reader

> Moving 1 billion rows is no easy task

This isn't an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, including on a single server. Exporting "select * from table" from an OLTP such as Postgres or MySQL into an OLAP is trivial and quite fast, and if 100M rows/sec on commodity servers isn't fast enough, there's always performance tuning [1].

[1] https://altinity.com/blog/loading-100b-rows-in-minutes-in-al...

View on HN

That sentence is more in the context of the blog of moving a billion rows across data-stores. We will edit it to make that more clear. Thanks for the feedback.

I can't speak for Snowflake, but on Clickhouse it's quite literally [1]:

    insert into new_table select * from postgresql('postgres:5432', 'db', 'table', 'user', 'pass');

I assume it's similarly easy on Snowflake, Databricks, SingleStore, and the rest.

[1] https://clickhouse.com/docs/en/sql-reference/table-functions...

Yes, and there’s loads of opportunity to over engineer every step of the process.

data stores or data lakes with data rivers?

You’re forgetting the data dams on those data rivers

Please can we have data fish ladders next?

Here in Oregon we are removing the data dams to have more natural data flows.

Data dams adds to the visual metaphor of data breaches...

gotta stick to the rivers and lakes that you're used to

Or wouldn't COPY from CSV be much faster?

CSV based export/import would involve writing to disk, sending over network then reading back from file. Select is just one copy over network, no intermediate CSV.

You can stream CSV without writing it to a disk.

What would be the point in that?

COPY is way faster than INSERT, at least in PG.

Fwiw you'd lose type information. Parquet / arrow are much more appropriate here

Reply on news.ycombinator.com