Hacker News Reader

Comment by boulos | Hacker News Reader

Disclosure: I work on Google Cloud.

This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors.

I'm now much more inclined to make my GCS FUSE rewrite in Rust also support S3 directly :).

View on HN

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop

It's been a few years, but I lost customer data on gs:// due to listing consistency - create a bunch of files, list dir, rename them one by one to commit the dir into Hive - listing missed a file & the rename skipped the missing one.

Put a breakpoint and the bug disappears, hit the rename and listing from the same JVM, the bug disappears.

Consistency issues are hell.

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors.

Yes!

S3guard was a complete pain to secure, since it was built on top of dynamodb and since any client could be a writer for some file (any file), you had to give all clients write access to the entire dynamodb table, which was quite a security headache.

If it's been a few years, it was likely before the migration to Spanner:

https://cloud.google.com/blog/products/gcp/how-google-cloud-...

We posted that blog in early 2018, but stated:

> Last year we migrated all of Cloud Storage metadata to Spanner, Google’s globally distributed and strongly consistent relational database.

So maybe pre-2017?

Had no idea this happened. Very cool to see. We use both GCS and S3.

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop

Is this public?

Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors

disclosure: I work at Google as well.

Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release:

> Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already.

If folks are actually interested in these connectors, I'd also recommend this blogpost from last year:

https://cloud.google.com/blog/products/data-analytics/new-re...

because even with consistency, GCS and S3 still aren't filesystems :).

Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.

(Phone reply, sorry for the brevity)

Just comparing to the previous release:

https://github.com/GoogleCloudDataproc/hadoop-connectors/com...

there are some big deletions like in:

https://github.com/GoogleCloudDataproc/hadoop-connectors/com...

and

https://github.com/GoogleCloudDataproc/hadoop-connectors/com...

Igor would know more :)

In the case of Hadoop's S3 connector, this could eliminate this entire directory, plus its tests, plus a bunch of hooks in the main code: https://github.com/apache/hadoop/tree/trunk/hadoop-tools/had.... There's an argument in favor of keeping it in case other S3-compatible stores need it (though you'd still need DynamoDB or some equivalent) and because it makes metadata lookups so much faster than S3 scans, which helps with query planning performance. But I imagine even fewer people will take the trade-off now that Amazon S3 itself is consistent.

Is your rewrite available?

Not yet, it’s too crappy :). I only get to it every once in a while and it shows.

As an example, until last weekend, I would just hardcode an access token rather than doing an oauth dance. Luckily, tame-oauth [1] from the folks at Embark was reasonably easy to integrate with.

I also got a little depressed that zargony/rust-fuse was stuck on an ancient version until I learned that Chris Berner from OpenAI had forked it earlier this year to modernize / update it [2].

[1] https://github.com/EmbarkStudios/tame-oauth

[2] https://github.com/cberner/fuser

Well when it becomes production-ready, feel free to open a PR or issue to https://github.com/ofek/csi-gcs

I'd prefer to use a memory-safe version :)

Reply on news.ycombinator.com