Hacker News Reader

Calcite is actually pretty damn fast, the overhead is surprisingly minimal.

I am using it as the backbone for my hobby project that auto-generated federated GraphQL API's:

https://github.com/GavinRay97/GraphQLCalcite

The experience has been incredibly positive and the community has been incredibly helpful & supportive. It's one of the coolest technical projects I've ever seen and has sparked my interest in query engines and relational databases.

I posted some JMH benchmarks of an app that parses a GraphQL query, converts it to Calcite relational expressions, and then executes it against an in-memory DB and it ran on the orders of milliseconds:

https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxb...

Something very similar to Calcite but much lesser-known is the "Teiid" project:

https://github.com/teiid/teiid

Highly recommend checking the code out. It's got a brilliant query optimizer/planner tailored to cross-datasource queries, a cache system, and a translator architecture that can convert a generic SQL dialect to dozens of SQL/NoSQL flavors.

Also integrates other data sources like REST API's, S3, flat files etc as queryable data sources.

I was moreso referring to the pieces of queries that Calcite may have to execute itself, often because they are not supported by the underlying data source. That piece of Calcite hasn't been very highly optimized. There's definitely ongoing work in that area though.

For example, I have a former student who's been working on an adapter to Apache Arrow. Beyond having a fast execution engine, this should also eventually enable Calcite to work pretty seamlessly with other systems using Arrow without serialization overhead.

Is this student in contact with Jacques Nadeau?

He's one of the only people I know that's a committer/PMC of both Arrow and Calcite, super nice and brilliant guy.

Maybe he'd have good feedback or ideas around this?

https://twitter.com/intjesus

Indirectly. I know Jacques. I'm also on the Calcite PMC :) Thanks for the pointer though!

If you've responded to any of my emails, no thank YOU!

I've never heard of Calcite before and dug a bit deeper into this project and found it to be quite active. Only taking into consideration changes to Java files, this is the repo activity for the last 12 months:

     month  | authors | commits | files | churn 
   ---------+---------+---------+-------+-------
    2022-02 |      21 |      62 |    84 |  4387
    2022-01 |      26 |     109 |   243 | 32429
    2021-12 |      33 |      99 |   198 | 10461
    2021-11 |      19 |      49 |    77 |  6960
    2021-10 |      26 |      64 |   371 | 13626
    2021-09 |      18 |      41 |    68 |  2258
    2021-08 |      11 |      17 |    25 |  1924
    2021-07 |      17 |      30 |    51 |  2704
    2021-06 |      14 |      31 |    28 |  1708
    2021-05 |       9 |      17 |    35 |  1606     
    2021-04 |      11 |      46 |    99 |  4224
    2021-03 |      16 |      36 |   143 |  8471

What do you use to produce this analysis?

My product (https://gitsense.com) moves most of Git's history into a Postgres database and from there, you can execute the following SQL statement:

    select
        commit_ym AS month,
        count(distinct(author_email)) as authors,
        count(distinct(commit_id)) as commits,
        count(distinct(path_id)) as files,
        sum(total) as churn
    from
        z1_commits_422 as commits,
        z1_changes_422 as changes,
        z1_code_churn_422 as churn
    where
        commits.id=changes.commit_id and
        changes.code_churn_id=churn.id and
        lang='java'
    group by commit_ym
    order by commit_ym desc
    limit 12

By having most of Git's history in SQL, I can slice, dice and cross-reference code history, which is how my product works.

The community is also great. Honestly, the main thing preventing things from moving faster is lack of time to review PRs. There's a lot of interesting things in the pipe!

Yeah I can see what you mean. There are currently 20 open pull requests [1] that hasn't changed in over 28 days, which accounts for 53% of all open pulls that are less than 4 months old.

1) https://oss.gitsense.com/insights/github?q=pull-age%3A%3C%3D...

Calcite is definitely some cool tech. I can see why it would be attractive for bigger teams, but it seems like a big lift for smaller teams. Our goal is to make it easy for devs already familiar with Postgres to be able to use add databases without learning new tools or adding software... besides adding Hydra, of course!

That's a fair point. Although if you're already using a JVM language, it's incredibly easy to integrate. Just another JDBC data source with some JARs to add to your classpath :)

What's the difference whit regular ORM?(genuine ask)

ORMs map relational databases to objects. Calcite does not do that. Calcite takes different data sources (potentially with different data models) and presents them all in the relational model. You interact with your data entirely in SQL while ORMs typically have some DSL which uses whatever object model the ORM defines. ORMs are also typically designed to connect to a single data source at a time. I'm personally not familiar with any ORMs that allow combining data from multiple sources within the same application.

The primary advantage of Calcite for connecting multiple data sources are 1) easily joining data sources that use completely different APIs (assuming there is a Calcite adapter available) and 2) supporting more complex queries than the original data source supports without having to write code (other than SQL) to do the processing.

Don't forget having two industrial-grade academic query planners/optimizer implementations that took collective decades of engineering effort!

And the project being founded by someone who has written multiple RDBMS.

Calcite is a wonder.