Hacker News Reader

Comment by tpetry | Hacker News Reader

Sadly the article never describes HOW they transfer the data. Simply sending 3PB on the wire will cost a fortune. Maybe Google does have some non-public service like AWS Snowball?

View on HN

Nothing against the author, but it sounds like another story that Google PR pitched and influenced. It is normal these days. I am sure AWS does the same. That is why the technical details may be light. The submitter is likely a Social Media Manager for Google (nothing wrong with that - kudos for not obscuring it!)

I, too, have wondered about the technical details when I see stories like this. If the data was originally stored in S3, they probably used the Cloud Storage Transfer Service (https://cloud.google.com/storage/transfer/). The secret sauce appears to be parallelization of the transfers.

When I copied hundreds of GB of data from one bucket of to another AWS bucket, it took me days to do! Then I realized it was because I was running a single threaded process. The slowest part was getting the list of objects to copy. If I had a separate, faster index of the objects, then I could have done a copy in parallel much more quickly and max out the bandwidth available to me. (Maybe that is part of the rationale of Netflix's S3mper?).

I'm curious to hear other thoughts on accomplishing fast transfers of S3 data.

Edit: theoretically speaking, if you had 1 TB of data to transfer, and a paltry 100 Mbps (12.5 MB/s) connection, you could do this in under 24 hours if the entire network was fully utilized. Not a network engineer so go easy on me if I am wrong!

As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before any other discounts.

Hi! Author here. (Nice to meet you.) I can tell you that Evernote, not Google, pitched this story to me. While the Google Cloud PR team was aware of its existence, they did not contact or attempt to influence me when it came to its content. (Literally, all that they did was confirm the on-site presence of their engineers, and that an engagement of that sort wasn't a one-off favor to Evernote.)

Unfortunately, a technical deep dive on the exact parameters of the transition wasn't really in scope for this piece. I tried to include detail where I could, and Evernote has a really wonderful 5-part blog series that goes into many of the details of the transition. (It's now linked in this post.)

To provide some additional details on the transfer: Evernote was going from data centers in California that it owned and operated to Google, not from S3. Technically speaking, the migration isn't 100% complete, as Evernote is still finishing the transfer of its attachment files to Google.

Evernote did not disclose the exact details of how that transfer was executed on the record, and thus it did not end up in my piece. Thanks for reading!

Thank you for your response and adding clarity around how the story came to be. You must forgive us if we are inherently suspicious.

Happy to add clarity where I can -- it is my job, after all. Suspicion and skepticism are healthy, and I totally hear the criticism that the piece feels marketing copy-esque.

For what it's worth, the Evernote team seems genuinely happy with the way the transition went, along with their interactions with Google.

The most straightforward is running aws s3 command but as you noted it's single threaded, slow as hell. Imagine you have 100TB of data to migrate from one bucket to another. If the command encounter an error (which is very likely to happen), you have to start again.

I think the only other option left is use Spot Fleet EMR. I believe the S3 API takes a marker for pagination so you can probably keep paginate to list the next N keys (of course assume your current bucket is no longer accepting new objects... otherwise create a Lambda function and let new file to be copied to the new bucket on create event).

I don't remember if aws s3 cp --recursive bucket1 bucket2 requires local copies first, or it is a remote copy-only operation.

I have always wanted AWS to allow S3 bucket to be transferred from one account to another with simply transferring ownership.

I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to copy was about 400 GB of small 5-20 kb files - not ideal for fast copying. I gave up and used a different method.

I suspect Evernote, if using S3, would have a similar data profile of many small objects.

Do you think you can share with the public this different method? If not, I completely understand.

No secret - I used s3s3mirror. Multithreaded copying.

>As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before any other discounts.

1TB out would be closer to $0.09 * 1024 == $92.16 (still expensive) -- think you may have misplaced a decimal there.

The Evernote engineering team posted a five part blog post [1] that goes deep on the detail of the migration and work to maintain consistency.

TLDR: Shard at a time was replicated over the network, then live traffic was cut over to the Google Cloud replica.

[1] https://blog.evernote.com/tech/2017/02/08/part-1-evernote-se...

the timing of these blog post is actually just perfect.. probably they knew that the "geeeks", after reading the PR article, wanted to have more insights.

You may have seen this story a few days ago, where a summer intern project at Google saved 1.5PB per day:

https://students.googleblog.com/2017/02/intern-impact-brotli...

Since this would be a simple connection from Evernote's existing location to Google, I don't think the cost would be a significant factor.

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

Yeah, this is actually AWS Snowball. But the Google Cloud does no have a competing offering. It has none publicly, so it would be of real interest if they only have it for big customers handing over their first born son or for everyone on request but not advertising it like AWS.

They had something like this, but now outsource it to iron mountain: http://itknowledgeexchange.techtarget.com/storage-disaster-r...

To the best of my knowledge, all of Evernote's data transfer was done over a network, not using an undisclosed AWS Snowball-like service.

-Author of the piece first linked here.

3PB over a fast, peered connection isn't unreasonable (and again, this would all be ingress to GCS which we don't charge for, but your provider would). The limiting factor would be how much you want to attack your storage layer at home.

Disclosure: I work on Google Cloud (but wasn't involved in this migration, nor even particularly aware).

Nope.

First, getting data into the cloud is cheap, it's outbound bandwidth that's expensive.

Second, that's peanuts for a big company. Most wouldn't notice a 100k bills for a one of transfer. Not to mention it's negligible compared to the size of storage and processing.

Reply on news.ycombinator.com