Relational Operators in Apache Calcite (2021) | Hacker News Reader

Relational Operators in Apache Calcite (2021)

querifylabs.com

60 pointsyeesian15 comments

I'd recommend looking into Substrait[0] for anyone interested in this mid-/low-level database/query engine space. It tries to establish a vendor-agnostic set of relational operators (and all the things surrounding it) to make it easier for query engine implementations to share individual components.

[0]: https://substrait.io

That's cool, thanks for sharing! Do you know how close they are to their example use cases [1]? So far I've only been able to find a tool for calcite SQL parsing [2] but not the portion connecting to Arrow C++ compute kernel yet.

[1]: https://substrait.io/#example-use-cases [2]: https://substrait.io/tools/producer_tools/

I'd check out the Slack, which is where I've seen a few projects integrating it coordinating.

I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet.

I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).

I recently had to parse source code of views to produce a mapping between the view columns and the underlying table columns - it basically worked with some caveats and was pretty painful to do (most databases expose this as information schema/metadata but that was not an option for us for various reasons).

For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.

This should be pretty easily doable in Calcite if you have access to the query which defines the view. Calcite has a pretty rich metadata system. Take a look at RelMetadataQuery.getColumnOrigin.

I once heard that Apache is the graveyard for open source software.

Is that really the case?

Yes and no.

A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest.

But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard.

Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move.

It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.

Calcite is relatively unknown, but it is the base so many other projects are built on. Among them is Apache Flink, which is quite alive and top of its category (low latency distributed stream processing)

https://calcite.apache.org/docs/powered_by.html

It depends. There are a large number of dead projects. There are also a large number of libraries and tools that are active and widely depended on, but you may not have heard of them.

Then there are quite a few large projects donated by corps with active communities. See this page

https://projects.apache.org/statistics.html

and scroll down to "Largest/Busiest repositories". There's a list of widely used enterprise software in there which are definitely not dead.

It is and it isn't. There's some good stuff there, but there's ever the risk that you'll get bushwhacked by something that you need to pay an expensive consultancy -- who just so happens to be the team behind the product -- to fix.

Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!

I'll just note that the risk mentioned certainly does not apply to Apache Calcite. (Disclaimer: I am a member of the Apache Calcite PMC, but not terribly active at the moment.)

A lot of open-source Apache projects were backed by commercial companies, and clouds (primarly AWS) forked those projects competing directly with the companies backing the Apache projects. The end result was that the commercial companies mostly released new versions of the software with more restrictive licenses that could not be sold for free by the cloud companies.

That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.

It's certainly not the case for Apache Calcite. Calcite is pretty widely used by a number of companies and there are a number of highly active developers.

to me it feels the same, plenty of great projects that essentially died once they got into apache. It has that java feeling.

Reply on news.ycombinator.com