How Cloud Storage uses Spanner for consistent bucket listing | Hacker News Reader

How Cloud Storage uses Spanner for consistent bucket listing

cloudplatform.googleblog.com

75 pointsdeesix14 comments

This is Google's dig at AWS S3 which is eventual consistency for object creation/listings.

If you want strong consistency for S3 consider something like JuiceFS: https://JuiceFS.io

It wraps your S3 in a strongly consistent and fast POSIX API (aka regular mountable filesystem) and its way less expensive than EFS.

> This is Google's dig at AWS S3 which is eventual consistency for object creation/listings.

Correction: S3 is strongly consistent after new object puts, but not lists, modifications, and deletes.

Definitely not as convenient but you can get strong consistency if you only rely on new objects.

That being said, I am curious as to how available/consistent S3 is versus Google cloud storage.

Additional correction: S3 is consistent after new object puts only if you haven't already asked for that object. Otherwise you may get a stale "no object" response.

So wish this was true but unfortunately it is not.

Spanner is great, but I really wish there was a way to use it for less than $600/month. :(

Yep. I also wish that you didn't have to use a special proprietary RPC API for writes...

These are connected because it means that it's tough to prototype on something cheaper and then switch over.

Yeah, it makes Amazon aurora far more attractive

GCP also offers Cloud SQL with similar features to AWS Aurora.

Cloud SQL is more like plain RDS, no?

The Spanner paper is one of my favorites. I love how it took several different things to come together to make possible.

A great example is Google needed a determinate network so standard TCP/IP would not cut it. But then other things in the infrastructure was needed.

This is the ultimately full stack solution.

https://research.google.com/archive/spanner.html

But I still can't set a bucket-wide policy for caching to be applied automatically to new files. So in other ways, consistency is inconveniently eventual.

This is pretty easy to do with a cloud function. I was a little apprehensive at first, but have been using it across many buckets and haven't seen any issues. Here's a version of the script I'm using: https://gist.github.com/justinph/acb6301d63b407e1fb205cbb42e...

Google recently announced some changes with cloud storage and pubsub triggers that potentially means that you could only have this trigger when a file is newly created, not every change. I haven't looked into that yet.

The code I've picked up from a predecessor uses a Concourse task to achieve a similar end. But it's just annoying that I can't set a policy on the bucket.

As an aside, it looks as though Google's golang API for GCS has a way to set the policy when you provide the file, so maybe at some point I can close this issue: https://github.com/concourse/semver-resource/issues/61

While there is not an option to set a default cache policy on a bucket, you can specify a cache policy as part of object uploads, although I admit that this can be inconvenient if you upload from multiple clients.

In addition, caching only applies to objects that are anonymously readable. Private objects are never cached.

Reply on news.ycombinator.com