Hacker News Reader

Would it be easy to pump Cloud Datastore into Bigquery and then read from that? (genuine question)

You can load *.info Datastore backups into BigQuery tables natively. You could probably write up a quick apps script to automate data refreshes it if you wanted.

Still not the same as reporting directly from a live datastore though :)

That's exactly my issue. I have a crawler that collects data and stores into Datastore. I need to periodically scoop up the snapshots and export them into BigQuery, then run the visualizations. I want my visualization to be live and also don't want the overhead of scooping the data.

Using App Engine, I schedule task queue tasks to insert rows into a BigQuery table on the fly [1].

It only works for inserts, though.

[1] https://cloud.google.com/bigquery/streaming-data-into-bigque...

Thanks! And yes, as you mentioned, the problem is that it works only for inserts. I use Datastore as my primary database. So requesting for supporting Datastore from Datastudio.

I would try a Cloud Dataflow streaming job. Both Datastore and Bigquery have "IO" connectors in Dataflow.

Reply on news.ycombinator.com