Hacker News Reader

Comment by gavinray | Hacker News Reader

  > Hydra automatically picks the right DB for the right task and pushes down computation, meaning each query will get routed to where it can be executed the fastest. We’ve seen results return 100X faster when executing to the right database.

This is really interesting. Could you talk a bit more about query pushdown and planning/optimization?

Is this through FDW's? Would love to hear more about the technical details.

Shameless plug -- I work at Hasura (turn DB's into GraphQL API's) and this seems incredibly synergistic and useful to get access to databases we don't have native drivers for at the moment.

Any chance of an OSS limited version?

View on HN

Hi! JD here, Hydra's CTO.

Hydra does not use FDWs except for Postgres-to-Postgres communication (for now). What we found was that FDWs do not do pushdown very well, even when Postgres has full information. You can get FDWs to push down aggregations, but complex queries with subqueries etc quickly get slow again. In short, our goal is to have your queries take full advantage of the power of each datastores, and we found that FDWs do not accomplish that goal.

We want to support GraphQL at some point, so same goes for us!

We are thinking about an OSS version, I think how we do "limited" is a big part of what that means. What would you like to see in an OSS version? What would you use it for?

Thanks for the explanation =D

  > What would you like to see in an OSS version? What would you use it for?

I think that's a difficult question to answer because it's hard to do data-access partially. How do you gate that, so it doesn't give everything away for free and incentivizes people to still pay you?

Read-only access might be one way, but I'm unsure how popular that would be.

  > What would you use it for?

Generating GraphQL API's for other datasources by funneling them through Postgres

Parent comment is absolutely right that FDW as a general query router is still under heavy development. It's very likely that we'll see further improvement in forthcoming Postgres releases, which will come with additional benefits since FDW are used for a lot more than just "high-level" query routing in Postgres.

This would be great.

I know that EnterpriseDB is heavily invested in FDW development and core Postgres stuff, so maybe we'll see some more neat stuff come out of that team that makes it upstream.

In that case how does it work? Are the connectors wrappers around the other db drivers that expose a common Postgres wire protocol?

We have a similar product at Splitgraph, where we do use FDWs in the routing layer (along with some PgBouncer magic). We recently blogged about adding aggregation pushdown to our Snowflake FDW. [0]

[0] https://www.splitgraph.com/blog/postgresql-fdw-aggregation-p...

Big fan of Splitgraph and I know some other folks at Hasura are too

Just a suggestion, your home page's one liner is super vague and confusing. I had to scroll down to really figure out what is it that Splitgraph does...

"Splitgraph connects numerous, unrelated data sources into a single, unified SQL interface on the Postgres wire protocol."

Just my 2 cents.

Nice 2 cents! We just launched this LP recently so we're still testing it – we've also got a lot of pages to add this month that will hopefully clarify things.

The basic pitch is for a "Unified Data Stack – an integrated and modern solution for working with data without worrying about its infrastructure." Connecting unrelated data sources is one part of the product, but it also includes a data catalog, modeling, integration, warehousing...

By integrating the discovery, access and (optionally) storage layers, we reduce the friction for a lot of common workflows, kind of like GitLab does by bundling CI pipelines and version control. Even if each layer has some tradeoffs, the benefit of integrating them has a multiplicative effect on the platform itself. And if you need a more specialized provider for one layer, that's fine too – the "data middleware" model makes Splitgraph incrementally adoptable.

But yeah... marketing is hard, especially in the "bag of tools" stage of product/market fit when the optimal messaging can differ so much by use case. Thanks for your suggestion!

Steampipe [1] is an open source [2] project that uses Postgres FDWs to query 67+ cloud services (e.g. AWS, GitHub, Prometheus). The plugins [3] are written in Go similar to Terraform. We've found this approach very effective for DevOps data and pushdown works well for most (simple) cases. (Disclaimer: I'm a lead on the project.)

1 - https://steampipe.io 2 - https://github.com/turbot/steampipe 3 - https://hub.steampipe.io/plugins

Whoa, how have I not heard of this before?

If it's OSS, I am definitely interested in experimenting with this. Maybe I can write a blogpost or something?

Yes, the CLI, FDW, plugins and mods are all open source. Please let us know how you go - we thrive on feedback :-)

Incredible! This is SUPER useful!

This is awesome! I just have to ask, what's your monetization plan?

(I see your company has some pretty big costumers, but I suppose you're not just selling them a cloud version of Steampipe)

Turbot [1] is a bootstrapped company since 2014. Our namesake product is a cloud governance platform with a real-time CMDB, identity suite, policy engine and thousands of automated operations for tagging, security, deployment, etc.

Steampipe Cloud [2] is in private preview providing a hosted version of Steampipe (and more). We're iterating fast and would love your feedback :-)

1 - https://turbot.com 2 - https://cloud.steampipe.io

I was thinking the same thing - FDWs As A Service

Reply on news.ycombinator.com