Hacker News Reader

Comment by spookylettuce | Hacker News Reader

I tend to utilize bigquery's external connections and scheduled queries to periodically clone my "hot" PG data to an BQ table for long term storage and analysis.

It seems so much easier to go PG=>BQ than the other way around.

View on HN

Is the idea something like a scheduled:

   INSERT INTO bq_table SELECT * FROM EXTERNAL_QUERY('');

I'm guessing you're on the hook for keeping the schema up to date with the Postgres schema.

I also use the external data connector, the problem is when you have two projects in different locations. Then I go for psycopg2 and uploat it to bigquery using bigquery magic lines

Same here. We previously used Stitch until they dropped the free tier.

I second federated table connector (and scheduled queries).

Reply on news.ycombinator.com