Cloudquery: Easily ask questions about your cloud infrastructure | Hacker News Reader

Cloudquery: Easily ask questions about your cloud infrastructure

github.com

75 pointsyevpats18 comments

Damn this looks really good. Data-first SQL integrated interfaces always get a +1 from me, managing resources without unified abstractions is always painful, especially when you have to do this at scale, standard CLIs always end up falling short in some way

Also, relevant tool for similar applications (albeit not SQL based): https://github.com/cloud-custodian/cloud-custodian

Thanks! This is definitely one of the use-cases to have an SQL abstraction of many resources. Also the idea is to be able to define rules in cross-account, cross-apps - For example: making a rule/notification that depends on information in Okta and in GCP/AWS (This is pretty simple with SQL join).

Regarding Custodian - This is definitely a great tool with it's own advantages. Some of the dis-advantages that lead me to start cloudquery is that it has a proprietary query language that you have to learn and as such is more limited than SQL (The example above won't be possible out-of-the-box).

Obviously a quite popular project given the number of times it's been submitted this month: https://news.ycombinator.com/from?site=github.com/cloudquery

This is very interesting. If you're managing hundreds of accounts pulling data even from Config from the master payer level is tedious and time consuming. If that data can be pulled out and queried with a real query language (config ql just doesn't cut it) it would make mass management so much easier.

Looks like there are two steps to this tool:

  1) Fetch data and put into a sql db
  2) Query sql db

Wouldn't it be better, performance aside, if there was just one step, i.e. querying the cloud with SQL, essentially an SQL-to-cloud-APIs interface. Otherwise you're always operating on out-of-date data.

It's a cool idea for a feature!

The idea/advantage to have the data in regular SQL database - is to be able to work with all the standard tools and analytics dashboard you have and familiar with (AWS quicksight, Google DataStudio, sqlite shell, etc...)

The flipside is that performing joins in a process across many document databases (which is what this approach would resemble) is wildly non-performant compared to doing it in a relational database.

But now every time you run a query without a where clause you're firing API calls to all AWS services. And then maybe recursive API calls on top of that.

This is fine if you're a personal account, but then you wouldn't need this tool in the first place.

I work on a product named http://vantage.sh/ that provides something similar as a managed service specifically for AWS. I respect the effort that went into this!

Unsolicited feedback: your homepage could use a few more screenshots for us to see what vantage really is.

Very fair. We're about to ship a new marketing site with a features page that has more screenshots and feature explanations as well as a video demo on the home page.

Excited to see the launch

Really cool.

If I could create resources with INSERT statements, even better.

Thanks! This is on the roadmap, I'm not sure it will be via Insert statements but there will be an option of action/change via some level of abstraction that is not via API/writing code.

Ouch. That's a tall order.

There's a lot of stuff that needs to happen for that, and error-handling would be quite tricky.

This would require either having a logic to "converge" the infrastructure – meaning implementing a big chunk of Terraform and adding some features that would make this thing look like a Kubernetes cousin – or having the insert statements blocking for minutes at a time.

Yes, a complete support for insert will definitely look like k8s/terraform cousin. As mentioned in the comment below we probably will support some specific pre-defined actions. Anyway, currently we only support query and as we gather more feedback we will add the action functionality which has it's set of nuances some of which you described very well.

I get the request, but it seems like the outcome is better achieved with Terraform or Pulumi.

Yup, you can create mock resources with TF/Pulumi

Reply on news.ycombinator.com