Gcloud storage: Faster data transfers for Cloud Storage | Hacker News Reader

Gcloud storage: Faster data transfers for Cloud Storage

cloud.google.com

166 pointsingve42 comments

From the article:

"The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."

It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.

We made the same decision for 's3cmd'.

All of that to say:

The biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky auth workflow for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.

[1] rsync.net

[2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.

[3] https://rclone.org/

Rclone is clunky because it works with everything, even things like OneDrive on Linux, which is not supported officially. Love rclone. Also just a great copy / move utility.

How do you freeze python tools into binaries?

par- I assume this is the open sourced version https://github.com/google/subpar

"How do you freeze python tools into binaries?"

I think there are several workflows for doing this - I know in the past we used something called 'cx_Freeze' but I think the person in charge of this is now using 'py2exe' ?

I think the 'borg' project packages up their single file .exe distribution with 'pyinstaller' but I may not fully understand that part of their process...

I use py2exe with pretty good success in deployments at my company. It's bloated and such but sometimes it's just the easiest path forward.

Pyinstaller is the most actively maintained and in-use, but nuitka is on the rise as well. There are less-active but still serviceable ones like py2exe as well

https://pyinstaller.org/en/stable/usage.html

https://www.nuitka.net/

I use s5cmd, it’s a fast parallel s3 cli client: https://github.com/peak/s5cmd

Haven’t heard of rclone before so can’t compare yet but s5cmd is supposed to work against GCS and other s3 compatible apis.

I'd also be curious about how rclone's parallelization compares to gcloud storage's in practice:

> Second, it utilizes a new parallelization strategy that treats task management as a graph problem, which allows more work to be done in parallel with far less overhead.

> ... In gcloud storage, all operations happen in parallel. As an example, parallel composite uploads are enabled automatically based on bucket configuration. This is a vast improvement compared to gsutil, which requires the -m (parallel operations) flag to improve the performance for uploads and downloads.

how faster is rclone compared to s3cmd ? We're using s3cmd to sync some stuff for backup purpose (many many small-ish) files and performance has been generally mediocre

We often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link:

https://github.com/mishudark/s3-parallel-put

How are gcp and aws different in terms of auth required by rclone?

I have also had persistent issues with gsutil and have used rclone instead, rclone is clunky but it has been an ever increasing part of our toolkit and I don't see that changing soon.

"These tests have been performed on Google Cloud Platform using n2d-standard-16 (8 vCPUs, 32 GB memory) and 1x375GB NVME in RAID0 in us-east4."

Am I missing something? Is there some configuration of RAID0 that allows you to use a single drive? Or is this just a confusing way of saying they have one logical drive available, because the two attached NVME drives are configured in RAID0?

You can create a RAID-0 volume on top of a single drive. But it seems this was a typo in the article.

This could just be an ambiguous wording thing, i.e: it's one 375GB effective volume, backed by NVME in RAID0. I dunno how it works, but depending on how it's sold (e.g: you choose the size you need, then the redundancy/performance you want, rather than having to define how that is constructed at a physical level), it may make sense to talk about it that way.

This was inevitable it seems. If you look under the hood gsutil is a wrapper around a vendored version of boto2. boto3 is a big departure and never got support for Google Cloud Storage.

Also the fact that it was separate from the rest of the gcloud CLI was a weird quirk. This fixes both of those issues.

Connect it up with Skyplane and you’ll really be speeding along

https://medium.com/@paras_jain/skyplane-110x-faster-data-tra...

I'm interested to see more benchmarks. The posted numbers in that blogpost seems too good to be true, is tooling really so far behind that a single tool could surpass the transfer time by that much?

The architecture goes through some of the methodology: https://skyplane.org/en/latest/architecture.html which is what I'm currently reading through and trying to understand.

It “cheats” by setting up a VM in GCP and another in AWS and does a direct transfer between the two instead of having to upload and download on whatever local connection you’ve got.

They're comparing against AWS DataSync which I assume also does not go through your local computer either.

I actually worked on gsutil a while ago! I added in flight compression -J/j. Glad to see gsutil is on its way out.

How did you choose the letter "j"? were all the more obvious ones just taken?

Maybe the inspiration was `-j` for `tar`, which enables bzip2 compression?

Now we just have to explain why -j means bzip2 for 'tar'...

The original intention for most of the flags was to keep them aligned with rsync. The feature only compressed the data in flight, as opposed to -Z/z which stored the compressed version. This didn't directly align to anything in rsync, so I picked an unused letter in rsync (-J/j).

In retrospect, it might have had a better home as some option on -Z/z, or if I had the tool figure out if you were bottlenecked on bandwidth, spare compute, and your data compressed well, and apply it automatically.

I don't know how long this flag has been there, but -z does mean compression in-flight in rsync.

> -z, --compress compress file data during the transfer

You're totally right! If anything the gsutil's -Z/z doesn't align with rsync's -Z/z, and gsutil's -J/j is a better match for what rsync's -Z/z does. I added -J/j well after -Z/z was there so it's something that we have to life with unfortunately.

Does anyone know if the "gcloud storage" vs "gsutil" chart in the article uses "gsutil -m" (parallel mode of the tool)?

I initially misunderstood the article's explanation of the gsutil shim: namely, I thought it was "turned on" by default, but that is not the case. One either needs to apply the shim, or to start using `gcloud storage` instead of `gsutil`.

Looks like they are missing the useful "rsync" command from gsutil (Synchronize content of two buckets/directories).

Would be nice getting these speeds on Google drive!

gsutil hardly ever finished when trying to download a couple-hundred google workspace takeout, preferring to consume all available memory instead - so that's a pretty easy thing to beat.

does it outperform rclone?

I have not worked with gcloud storage command as I am only familiar with gsutil, but is the syntax the same? Also, is there a comparison with Terraform?

It's in the article but the tldr is that they split the file into 32 chunks, upload the chunks in parallel, and reassemble when uploading is complete.

This triggered a memory from my distant (12+ years ago) past: Globus Grid FTP had this capability, and it was awesome for bulk data transfer [1].

[1] https://docs.globus.org/globus-connect-server/v4/man/globus-...

Also a lot more convenient to use with WIF

How long until it's discontinued?

Gcloud has been around a long time and will probably be there as long as GCP is

Wow, feel good to see good news from GCP

94% faster than what? and "up to"? We already know the "up to" trick...

Faster than the old gsutil utility. The speed increase is real and easily measurable, but will depend on your files (lots of small ones vs a few huge ones etc).

Reply on news.ycombinator.com