HNHacker News
TopNewBestAskShowJobs

gingerwizard

7 karma · joined July 14, 2022

submissionscomments
gingerwizard··on Loading a trillion rows of weather data into TimescaleDB
Few in this thread have suggested ClickHouse would do well here. We tested 1 trillon rows recently, albeit much simpler data - https://clickhouse.com/blog/clickhouse-1-trillion-row-challe...

This is a good dataset though and the level of detail in the post is appreciated. I'll give ClicKHouse a go on the same...

Disclaimer: I work for ClickHouse

gingerwizard··on One billion row challenge in ClickHouse SQL
Author of the post here, would love to hear ideas for optimization to improve this.
gingerwizard··on ANN Vector Search with SQL-Powered LSH and Random Projections
Hey author, here, so yes its not the most recent technique, but when coupled with rescoring like we propose it can be a simple speed up for linear scans. It also a pure SQL solution and requires no indices. It benefits from being easy to update and not being mem bound. We recognise all the issues with it and pros in the final section. We're defn not claiming this is ground breaking - more a useful technique which is easy to replicate in SQL.

Pinecone https://www.pinecone.io/learn/series/faiss/locality-sensitiv... seem to promote as a viable approach.

We defn don't consider this to be the final solution for ANN and hence are investing in other graph based techniques - https://clickhouse.com/docs/en/engines/table-engines/mergetr...

gingerwizard··on GQL – Git Query Language
Some good examples and docs on this data https://clickhouse.com/docs/en/getting-started/example-datas...
gingerwizard··on Show HN: ClickPy – OSS project and service for Python analytics using ClickHouse
Its about a billion rows a day. Nice idea, we could probably add a visual for possible malicious packages.
gingerwizard··on Big data is dead
Just use ClickHouse :) https://sql.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTS...
gingerwizard··on How do you quickly search databases that are hundreds of millions of records?
I'd use clickhouse-local, ideal for this workload without needing to load into a full ClickHouse instance.

https://clickhouse.com/blog/extracting-converting-querying-l...