Hacker News Reader

Comment by dmix | Hacker News Reader

I haven't used one of these products yet. Are these analytics platforms able to connect to pull APIs? For ex: situations where you have to query it for data periodically or use webhooks?

Or do you have to upload all the data as CSV/XLS or some other direct data source?

View on HN

Typically these platforms will query on a data warehouse that you provide: redshift, bigquery, a big postgres instance, or the like.

In terms of getting data into that warehouse, you can write ETL (extract, transform, load) scripts yourself. There are also a number of ETL-as-a-service providers like Stitch or Fivetran that cover common things you might want to sync into your data warehouse and query like salesforce, google analytics, quickbooks, etc: https://www.stitchdata.com/integrations/sources/

Of course there's ANOTHER company to help me generate the scripts to popular 3rd parties :P

I guess what I want is some powerful individual-use product that has some easy analytics automation on spreadsheets / APIs. I believe this is designed for much larger companies with integration teams and high-paid help consultants.

Maybe I should build one although I'm sure an app like that exists already somewhere.

I'm not 100% sure if this is a fit for your needs, but Redash[1] (open source, but there is a hosted version too) you can query spreadsheets and APIs (within limits).

Our JSON data source is not merged yet [2], although available already on the hosted version.

You can see details here: https://redash.io/help/data-sources/querying/urls#JSON

[1] https://redash.io/ [2] https://github.com/getredash/redash/pull/3805

(I'm the creator of Redash, now CEO of Redash the company)

This actually looks a lot like something I need. I'll give it a shot.

I agree that these BI tools are not a trivial investment and are not trivial to use, but in defense of building on a SQL data warehouse:

> Of course there's ANOTHER company to help me generate the scripts to popular 3rd parties :P

> I guess what I want is some powerful individual-use product that has some easy analytics automation on spreadsheets / APIs.

The point is that everything speaks SQL, so vendor lock-in becomes much harder and there's no data source that "is not currently in our plans to support" — if you can get the data into the warehouse, it's supported. Definitely a higher barrier to entry for an individual, but I wouldn't do business BI without it.

We've done some insane things, joining across data from 4+ different systems (CRM, email open tracking, analytics, primary app database, email to a blessed account) in ways that would be impossible for an "off the shelf" solution.

There are tools such as Klipfolio and Grow but you're only limited using their UI to build the reports. That's always limiting no matter how flexible their language & UI is and larger companies tend to build their own data-warehouse to be able to query the data using SQL without any limit.

That's why Periscope, Looker and Mode Analytics are quite popular. There are also open-source tools that help you to ingest the data to your database such as Singer.io. Once you have all your data in raw format, you can model your data and power your BI without any limitation.

Reply on news.ycombinator.com