Logical replication and decoding for Cloud SQL for PostgreSQL
cloud.google.com
cloud.google.com
1) No way to force SSL connections without enforcing two-way SSL (which is a huge pain and not supported by all the clients we use). This is literally just a Postgres config option but they don’t expose it. RDS has had this feature since 2016.
2) No in place upgrade. This is again a feature built in to Postgres and RDS has had it for years. Instead the upgrade story for Cloud SQL is cumbersome and involves setting up a new instance, creating and loading backups, etc.
We switched to Cloud SQL from running our own Postgres and it is a huge improvement, but the feature set is disappointing compared to RDS
However, we have some third party data analysis tools (such as Tableau) that also connect to one of our databases. They are hosted in their own clouds and have to connect over the databases’s public IP address and can’t use cloud_sql_proxy. I of course manually confirmed that these connections use SSL but I would feel much more comfortable if I could enforce it from our end.
Here is a direct quote from google support when we contacted them about our database going down outside of our scheduled maintenance window:
> As I mentioned previously remember that the maintenance window is preferred but there are time-sensitive maintenance events that are considered quite important such as this one which is a Live migration. Most maintenance events should be reflected on the operations logs but there are a few other maintenance events such as this one that are more on the infrastructure side that appear transparent to clients because of the nature of the changes made to the Google managed Compute Engine that host the instances, this is a necessary step for maintaining the managed infrastructure. For this reason this maintenance does not appear visible in your logs or on the platform.
Here "transparent to clients" means that the database is completely inaccessible for up to 90s. Furthermore, because there's no entry in the operation log, there's no way to detect if the database is down because of "expected maintenance", or because of some other issue without talking to a human at google support: so really great if you're woken up in the middle of the night because your database is down, and you're trying to figure out what happened...
The Cloud SQL failover only occurs in certain circumstances, and in all our time using Cloud SQL the failover has not once kicked in automatically (despite many outages).
In fact, one of our earliest support issues was that the "manual failover" button was disabled when any sort of operation was occuring on the datbase, making it almost completely useless! Luckily this issue at least was fixed.
Either way I agree that some full-stack integration is needed on GCPs part to at least get that into the maitnaince log. It would also be nice to make most of these happen during the maitnaince window but IIUC they don't always have 24h notice of a machine reboot.
They have to do that when hardware fails (if you're lucky), but that it's happening so often suggests it's part of regular software maintenance or something like that. Which is pretty unacceptable to me.
Also, it enables a really cool pattern of change data capture, which allows you to capture "normal" changes to your Postgres database as events that can be fed to e.g. Kafka and power an event-driven/CQRS system. https://www.confluent.io/blog/bottled-water-real-time-integr... is a 2015 post describing the pattern well; the modern tool that replaces Bottled Water is https://debezium.io/ . For instance, if you have a "last_updated_by" column in your tables that's respected by all your applications, this becomes a more-or-less-free audit log, or at the very least something that you can use to spot-check that your audit logging system is capturing everything it should be!
When you're building and debugging systems that combine trusted human inputs, untrusted human inputs, results from machine learning, and results from external databases, all related to the same entity in your business logic (and who isn't doing all of these things, these days!), having this kind of replayable event capture is invaluable. If you value observability of how your distributed system evolves within the context of a single request, tracking a datum as it evolves over time is the logical (heh) evolution of that need.
Not only does it allow publishing events as part of an ordinary database transaction, it also provides a nice buffer if kinesis ever goes down. During the Nov. 2020 kinesis outage, our services using this mechanism kept chugging with no immediate issues.
Have been shopping around for a good way to do this, ideally with the ability to capture deletions and schema changes.
Have looked at Fivetran but it seems expensive for this use case, and won’t capture deletions until they can support logical replication.
DMS is awesome. No code is the best code. Big query is great but not having a “snap-your-fingers and the data is there” connector makes it a PITA for me to maintain.
As a result… we are using more redshift.
It seems so much easier to go PG=>BQ than the other way around.
INSERT INTO bq_table SELECT * FROM EXTERNAL_QUERY('');
Debezium => Kafka => Parquet in Google Cloud Storage => BigQuery external queries.
Disclaimer: working on Debezium
CREATE TABLE xx AS SELECT * FROM EXTERNAL_QUERY('postgres.db', 'SELECT * FROM my_table'));It's decent.
In fact, the Datastream documentation has a diagram showing Postgres as a source, as well as custom sources — but, disappointingly, neither is supported. Only Oracle and MySQL are supported.