1,776 karma · joined May 10, 2009
I reported it as a bug but Google said it was by design. More specifically they said: "You are correct, if the versioning enabled in your bucket then the object metadata is saved as an archive object in the bucket [1].This is the reason you are getting 200 for your HEAD request."
So Google claims that it does not have control over its own DNS servers and is therefore not to blame if the DNS is pointing to the wrong IP. Not very reassuring.
Unfortunately the problem is still occurring after I got this message (although less frequently).
Don't you agree that it's odd to only include HTTP 500 errors in the error rate? Let's say someone hacks your DNS servers and points storage.googleapis.com to 127.0.0.1. Then the entire service would be down completely but according to your SLA you'd have 100% up time.
This is not just a theoretical issue. In the past week we've been doing a bit more than 5 request/sec to Google Cloud Storage and according to NewRelic the average response time was 8 seconds! I.e. the service has been down and not been responding at all for large periods of time. I've been in contact with their support team and they've refused to reimburse us anything.
A couple of weeks ago I contacted their support because our newly created Google Cloud Transfer Service jobs were failing with "UNKNOWN ERROR". Turned out that it was a permission issue. I pointed out this must be a bug on their part since the page were you create transfer jobs says "Creating a transfer grants a Cloud Storage Transfer Service account the necessary source, destination, and project permissions to complete the transfer." Oddly enough they didn't agree that it was a bug.
Instead gave me a gsutil command to update the permissions. To my big surprise there was no way to set permissions to certain prefixes of the bucket. The command had to loop over ALL items in the bucket and update each permission individually. I pointed out that we have >300 million items in out buckets so this is going to incur large I/O costs (changing permissions is a Class a Request) and take ages to complete. Apparently t there was no other way. The command has now been running for 10 days and it's not even half way through. It has also crashed on several occasions, forcing me to restart it.