Hacker News Reader

Comment by iskander | Hacker News Reader

If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.

View on HN

Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR.

Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there.

http://stackoverflow.com/questions/11990839/is-there-a-way-t...

This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.

There is publicdata.samples.trigrams in the public datasets.

Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.

The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_...

@iskander - do you have other approaches for uploading which you'd prefer?

Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?

Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.

Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.

FYI - you can technically upload directly to BigQuery and not go through cloud storage. It just tends to be a bit more error-prone in normal implementations. We're working on documenting resumable upload to handle error cases better.

Reply on news.ycombinator.com