Hacker News Reader

Awesome! Datastore connector, please!! Cloud Datastore is the ideal NoOps and Serverless storage solution for projects from small to large. It would be great if you can build a connector for it.

View on HN

Would it be easy to pump Cloud Datastore into Bigquery and then read from that? (genuine question)

You can load *.info Datastore backups into BigQuery tables natively. You could probably write up a quick apps script to automate data refreshes it if you wanted.

Still not the same as reporting directly from a live datastore though :)

That's exactly my issue. I have a crawler that collects data and stores into Datastore. I need to periodically scoop up the snapshots and export them into BigQuery, then run the visualizations. I want my visualization to be live and also don't want the overhead of scooping the data.

Using App Engine, I schedule task queue tasks to insert rows into a BigQuery table on the fly [1].

It only works for inserts, though.

[1] https://cloud.google.com/bigquery/streaming-data-into-bigque...

Thanks! And yes, as you mentioned, the problem is that it works only for inserts. I use Datastore as my primary database. So requesting for supporting Datastore from Datastudio.

I would try a Cloud Dataflow streaming job. Both Datastore and Bigquery have "IO" connectors in Dataflow.

Reply on news.ycombinator.com