Hacker News Reader

Comment by jnordwick | Hacker News Reader

So these ingested files are JSON or something semi-structured? I saw how the table was created, so maybe is was doing the schema-less key-value pair thing on table creation and running queries against that? I'm sure this is all in the docs somewhere, but while I have a lot of big db experience, I have about 2 hours of AWS.

View on HN

So in this example the image says "lineitem_1t_parq" which makes me think that's probably parquet files (optimized and perhaps compressed columnar file format).

But it does the same stuff with CSVs too, perhaps a bit less efficiently. JSON is also definitely possible, but a bit more tricky because of the inherently nested structure and more complex format.

Take a look at this for the details on a similar idea: http://stratos.seas.harvard.edu/files/stratos/files/nodb-cac...

That paper discusses PostgresRAW, which can be found here; https://github.com/HBPMedical/PostgresRAW

I wonder if there are any similar projects using psql as an engine?

I think it's pretty common - it's open source, nice and modular codebase, reasonably advanced execution engine with stuff like MVCC. But also redshift itself, citus, hadapt and greenplum are examples that come to my mind.

The files are anything separated by line. Log files, CSV exports, anything. No specific format required.

So this is like a distributed awk, reparsing on every query?

Reply on news.ycombinator.com