qoega

Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be.

By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves

Is it really the case? Assume you have column with schema name and one with json object. And your materialised view/JsonExtract can be dispatched by schema name for a row.

I see the only suboptimal part if different schemas have different types for the same field and it has to convert it to String.

Interesting! As ClickHouse is not the fastest in this scenario, we can improve it! Probably initialisation of clickhouse-local process takes more time than jq.

ClickHouse has no such optimizations for reuse of columns.

Have you tried ClickHouse for dataset above? Interesting if it is faster or not

Looks, it is not

Hi! Why do you share it today? You have updated something recently in it?

You can add even without mentioning. I did almost nothing - it worked almost without diff.

The rest of the queries https://gist.github.com/qoega/32c4f54da6b0324aef8cd8d910dd7c...

It will probably be even faster: > clickhouse-client --time --query "INSERT INTO rides FROM INFILE 'nyc_data_rides.csv' FORMAT CSV;"

You can use clickhouse/clickhouse-server image. It has 21.10 version: https://hub.docker.com/layers/clickhouse/clickhouse-server/2...

You can enable fsync in ClickHouse. And it will not decrease bandwidth.

Why use FDW if you can work with ClickHouse directly? What is the use case?

By the way I can see there are some already but I do not use them https://github.com/Infinidat/infi.clickhouse_fdw/ https://github.com/adjust/clickhouse_fdw

Nice to hear! We are looking for cases where ClickHouse is not as fast as it could be. And we you can share your experience with ClickHouse on GitHub issues or by mail. If something was not intuitive or worked not as expected. Or lack in functionality

ClickHouse has no limit on sting data length. MaterializedPostgreSQL engine is just very recent feature with not large adoption rate. I believe if community will use it frequently it will become more bulletproof and more edge cases will be supported. TOAST in replication protocol is just not trivial to implement.

From docs it seems they use forked ClickHouse code.

Have you tried generateRandom? It supports almost all types. https://clickhouse.tech/docs/en/sql-reference/table-function...