Hacker News Reader

Comment by aketchum | Hacker News Reader

I must not interact with big enough data to understand, but what is the point of this? I have watched/read several of the Snow product descriptions from AWS and I am still not quite sure I understand the point.

It seems like a way to sync local big data to AWS cloud storage for data that is too large to realistically transfer via the internet. So is this simply a sneakernet external harddrive because physically shipping an 8TB hard drive has better bandwidth than using the internet?

I would love it if someone could explain a couple of use cases for the Snow family of products to someone that has never had to handle 100 GB of data much less terabytes.

View on HN

I can speak to part of the rationality around Snowball:

The old method of customers sending in hard disks and AWS importing them turned out to be _incredibly_ high touch.

* Customers would put the wrong labels on the wrong drives. Given they were encrypted, and the label details were critical to matching the right disk with the right encryption key, that was a big issue.

* Imports would routinely fail due to all sorts of driver bugs both on the AWS side, and on the customer side. The number of fringe NTFS quirks was phenomenal, let alone all the wide range of other formats that had to be handled

* Encryption tooling still really isn't all that user friendly. Too many opportunities for mistakes.

* Drives would get damaged during shipping.

* Stuff would go missing in shipping.

It just didn't scale as a solution, and caused unending customer pain and frustration.

Snowball solves _all_ of that, by taking out of the equation almost anything about the process that customers could get wrong. It handles all of the encryption stuff, is extremely ruggedised, has basic tracking. The label is an e-ink display that automatically contains all the right information. It becomes a "just works" solution.

It's just sneakernet with bells and whistles. As it should be.

Those are all really good points. I'm interested in hearing more about the NTFS quirks - what kind of issues were cropping up?

Now I'm really curious what filesystem Snowball wound up standardizing on, and how/if file metadata/ACLs/alternate data streams are maintained.

I helped a client do a transfer to S3 via snowball once for a music studio client with a smorgusboard of random hard drives.

The software client for doing the transfer is some sort of command line JVM tool from my memory. You locally create S3 buckets on the snowball and copy from the local disks into the buckets.

Once the snowball is returned, the buckets appear in your online account.

The main issues with NTFS would be the system either wouldn't be able to see the drive, or would fail to be able to read part of it. They were building up quite a repertoire of tricks to get things to read correctly, but it was almost impossible to safely automate.

Do those even matter if the final file destination is S3?

It's mostly a technical curiosity to me. Although maybe they would if you're using it in a system backup/archive workflow?

great examples of why to use this over "just" a large external hard drive in a case. Thanks for the insight!

Doesn't solve * Drives would get damaged during shipping. * Stuff would go missing in shipping.

It does. The Snowball was designed in a ruggedised fashion. It can take some extremely large shocks while in transit without putting customer data at risk. I forget the precise tests carried out, but I believe they even included "dropped from roof of multi-storey building".

Stuff going missing in shipping is solved by very low powered tracking chips in them. AWS knows where those devices are and could tell shipping firms if needs be.

sure it does. the drives are specifically designed to be rugged enough to handle transportation (and are tamper resistant and tamper evident), and I imagine amazon probably uses their own delivery network to transport it, which means if it is lost in shipping, it is amazon's fault (and presumable can be corrected). It also wouldn't surprise me if at least some of the product contain GPS to make it less likely to get lost.

Pretty sure the E-Ink on the snowball devices are set up for UPS tracking label format.

Lots of potential uses:

Imagine Amazon was bidding on a big Department of Defense contract ($ in billion). DoD let's say wants to send drones / UAV's whatever to do big sensor passes out in far off areas / imagery, wideband radio etc). There is NO fast internet for the TB's generated. You pop this out of the drone, put it on the next resupply mail flight out, and ingest into Amazon cloud on far side for analytics.

Let's stay a store wants to collect 30 days of customer video related to all traffic in their 100 stores to do some sort of AI analysis on how folks use the stores. Again, some poorly paid store staff person can unplug this after a week and ship it off.

Let's say I generate data but instead of wanting to do a snowmobile every year, I want to do more granular moves. Again, I could do a weekly or whatever ship, and again from maybe a few locations.

I think especially in places with lower high speed internet penetration (africa, china etc) there is room for something "snackable" like this.

Let's say I distribute movies to movie theatres. Again, do a weekly ship in/out.

It's $60/job. Not bad. No data IN fees.

For a sense of scale, my company used an AWS Snowmobile to transfer 100PB of archived satellite imagery to AWS.

More details here: https://blog.maxar.com/earth-intelligence/2017/digitalglobe-...

For a sense of the world, I went to Manila in 2018 to find out why there are several distributor can only sent their data weekly.

One of the distributor show me a 64 MB usb flashdisk that’s travelled every week with the delivery truck to get 4 MB of data from their branches (8 hours truck driving from the capital)

Went to Myanmar in 2017. The nation is just upgrading to 4G in 2016. But not every places have 4G coverage

2019 in eastern Indonesia (Seram island) you’re lucky if you can get a reliable 5 KB/s internet. It’s a remote area, but we still have a branch there :(

Why not satellite internet then?

$$$

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

For real! Reminds me of this article...

"A carrier pigeon with a cargo of microSD cards can transfer large amounts of data faster, and more cheaply, than just about any other method"

Link: https://spectrum.ieee.org/tech-talk/computing/networks/pigeo...

This almost got me and then I looked at the date of publication.

If you're new to joke RFCs, I suggest reading up on RFC 3514, a suggestion that malicious IP packets should be required to set the high order bit of the IP fragment offset field, which is to be designated the "evil" bit. This significantly simplifies implementing effective network monitoring.

It's still true. A pigeon can easily carry 10 microSD cards, 1TB each. Even if you're on a gigabit connection, it will take you over 22 hours to copy 10TB.

They meant that it was posted on April fools, not that it's an old article.

Or these days, micro flash cards.

What was the premium of storing the data in S3 as opposed to keeping it in your own tape warehouse? Definitely curious about the cost benefits!

S3 Glacier Deep Archive is probably the closest to a tape warehouse, at the standard rate it would cost $99000/month to store 100PB of data.

$0.00099 per GB per month * 100,000,000 GB = $99000

You need to account data transfer OUT if you need to get your data from S3. _If_ you could download 100PB data, it'd cost you $1M.

Hopefully it is not stored as one giant object and hopefully your retrieval would be only specific parts (small parts.)

You’d want to do some research on getting data out of glacier. I’ve read some nightmare stories about the bill from trying to get a small amount of data out of glacier. I can’t remember the details, and some of the large cost was people sort of doing it wrong. But some of the tooling makes it easy to do wrong.

When I was using glacier, we treated it sort of like a write only storage, with the understanding that if we ever needed data out, we’d have to think long and hard about the data’s value, and probably would have gotten an AWS rep on the line to make sure we didn’t do anything stupid.

Is your tape library stored redundantly across multiple data centers?

Unfortunately I don't have access to that info, sorry!

Just FYI your company has a cert mismatch on the following link from that blog post: https://platform.digitalglobe.com/gbdx/

The cert is limited to *.wpengine.com

And the link 404s so.. that's a shame. Was interested in learning more about the platform.

Thanks for the heads up! I'll notify the team.

You can learn more here: https://www.digitalglobe.com/products/gbdx

Like everything from DigitalGlobe -- it doesn't work.

LOL. What products have you had bad experiences with?

That was an excellent read. Thanks for sharing.

I work in healthcare, sometimes we have to go to places with no internet, and sometimes with no electricity. We gather a lot of HIPAA-protected data. This might be useful for those events.

Sometimes, even when we're in the middle of a big city, there's still no secure or reliable internet connection at the location we've been given, so this might be good for those cases, too.

I'm not on the infrastructure side of things, so I can't say for sure. But I've already forwarded the page to our mobile IT team to make sure they've heard about it.

Edit:

They might also be good for television and movie production. In the early part of this century, rushes were sent from remote filming locations to the studios loaded onto iPods. With everything being 4K and higher now, this might be useful to move those rushes to a cloud location where they can be accessed by multiple people who need to see them in multiple cities.

Lord of the Rings was almost stolen this way: https://www.youtube.com/watch?v=Ndge8WlM9q8

So yea, pretty much. I had to do this with a project from a couple years ago where we were moving a ton of images from a legacy colo'd server to S3. I can't remember how much data it was but it was enough to where it didn't make sense and would've taken for-fucking-ever to move it to S3 over public internet.

When we did it, we bought a drive, went to the colo, offloaded the files, and you have to do some stuff to kick off a job to let them know you're sending a drive and what kind and all that sort of thing. Then you mail it to them and they run an import into an bucket.

My guess with these drives is controlling the hardware makes the process significantly more efficient for them, and for the enterprise customers they're trying to move off colo and onto AWS.

One use case I know of is in a mining operation. They collect a ton of data about their equipment and their mines as they work, but they are often out in very desolate areas with no internet.

The mining equipment runs greengrass right on the machine to do real time inference to help make mining more efficient as well as flag potential equipment failure to the operator.

So they would strap this device onto the equipment, it would collect data and do inference, and then they can swap it out for a new one and send the old one back so the data can be added to their data lake to update all of the AI models.

The speed is questionable in many countries where consumer internet connections already have 500mbit upload speed. But in the US I think bandwidth can be quite limited if you're not in a major city with fiber.

The other big part is pricing. AWS charges $ 0.09 per GB for outbound transfer, so getting 8 TB of data out of AWS via the internet costs over $ 700. Doing it through Snowcone costs around $ 300 + shipping.

Reasonable use-case would be transferring video material or high resolution sensor data.

It never ceases to amaze me that people pay that much for transfer. It's one tenth the price on DigitalOcean and it isn't like this is comparing two different types of things.

If you have data that you're unlikely to need to access (i.e. backups), not many providers can compete with Deep Archive storage pricing.

4TB on S3 Deep Archive: $3.96/mo to store, $360 to restore and download over the internet 4TB on B2: $20/mo to store, $40 to download 4TB on Digital Ocean: $80/mo to store, $30 to download

Yes, it feels almost extortionary to have to pay that much to get your data out of AWS. But depending how long you store data for, it can end up being the cheaper option: you're just gambling on how likely you are to actually need to download it.

DO is incredibly limited in terms of products compared to AWS.

Yes but this is just data storage. I get it if you need to be on network, yadda, yadda, but so many people just senselessly use S3 and rack up bills that are easily avoidable.

Digital Ocean Spaces doesn't have the same durability designs of S3. Or performance in terms of TPS. It's not apples to apples.

DO is great for small personal projects, but, I wouldn't run a fortune 500 company on it.

What does that have to do with the price of bandwidth?

Can you transfer data out of AWS with Snowcone? I thought it was all transfer in, which is free.

You can transfer out, but you pay the similarly insane egress costs. Somewhat cheaper, but still bonkers.

Snowball/snowcone, US East

$0.03 per GB ~-> $300 per 10TB

If you're buying disks at scale that's almost the cost of the disk.

Network egress

$0.15 per GB

$0.08 per GB (>150TB/month)

It makes sense for the occassional transfer so you're not paying for expensive EBS storage for stuff you don't need cloud access to.

A screenshot in the article implies that you "Export from S3" https://media.amazonwebservices.com/blog/2020/snow_job_type_...

I would assume that this would be for gathering information from remote data collection nodes or from computers that don't access networks. Instead of hooking up with a laptop and some kind of portable NAS, you could connect a snow product to slurp up the data and then mail it to Amazon to upload. I think their selling point is the reliability of the snow family, and not having to worry about the data after the initial upload which can be super critical for some business models.

I've used AWS snowballs fairly frequently and it is definitely a specialty product.

For our use case it was genome sequencing. Each individual genome was a terabyte or so depending on what types of data we sent. We can crank out a few hundred genomes per week at full production so when our collaborators need to receive their data they usually neither have the capacity in bandwidth or storage for cohort of thousands. So it gets shipped to AWS.

We've made improvements to get more direct connections to cloud services so we don't use them anymore but for awhile we were filling a snowball and shipping it every week or two. We also used the Google Transfer Appliance once or twice (480tb) to transfer projects all at once instead of piecemeal.

It's sneakernet optimized to deliver direct to S3, with full end-to-end integrity checks. Depending on the amount of data you have and your internet connectivity, this can be significant optimization — for example, say, you work with professional photography or video but live in the United States where only a small fraction of people have an uplink speed greater than 10-30Mbps even if their download speeds are 500+Mbps. If you can generate 8TB of data in less than a month, this is faster and doesn't interfere with all of your other network usage.

We used them in a B2B setting when signing an enterprise data contract for nationwide real estate MLS data, including photos. The photo's needed to be uploaded to our S3 bucket and the only method available to download was via FTP. It made more sense to ask the third party provider to download the images to these disks, send them to us, and then we shipped them to AWS.

The video [1] quickly lists a number of use cases.

> Explore the new AWS Snowcone with Bill Vass and Jeff Barr (2:46)

[1] https://www.youtube.com/watch?v=8RNRssCiR_E&feature=emb_rel_...

I recently listened to an interview with the editorial team of Deadliest Catch (Art of the Cut #51). I could see this product being used by similar media productions where thousands of hours of film footage is generated in remote locations with low uplink speeds. Instead of shipping the drives all the way to the post-production house, the drives could be ingested at the nearest AWS facility.

I imagine if you're already integrated into the AWS ecosystem, there's a convenience in treating your on-premise node as "just another node". You can run EC2 instances on it, and manage it using the same tools. But with better latency and more robust local connectivity due to the proximity.

The ability to build AMIs and run them on Snowcones gives you the power to build applications that do all sorts of interesting filtering, pre-processing, and analysis at the edge.

To me the product feels like snakeoil and I'm skeptical it will take off in a big way, but I think more than enough customers will use it to justify their investment.

Remember, AWS is selling a platform. The more claws they can sink into their customers, the harder it is for them to switch to a competing one.

I don't get this one, but for the larger Snow products, yeah, it's for transferring data too large to realistically sending over the internet.

I'm with you. I think this is really cool but I have a hard time finding a use case for it.

The use case for the other Snow products (Snowball, Snowmobile) are clearer, as they are for transferring petabytes of data to/from AWS without having to use slow internet connections. Snowcone could be used for this indeed, but at only 8 TB, the value proposition seems a lot less. Personally, I'd probably just suffer through having to spend a week doing a slow internet upload/download rather than paying for Snowcone.

Also, the articles talks multiple times about using Snowcone to upload the data for you, over your own internet connection. Why would I pay for Snowcone to do that when I could just upload directly to S3, without using Snowcone as the middle layer, and get the same speeds?

The compute aspect of it seems cool from a "im a tech enthusiast and this is cool" perspective, similar to how a Raspberry Pi is cool, but I don't see the real-world use case for hosting compute workloads on a rented Raspberry Pi. I would just buy a Raspberry Pi instead for cheaper.

What this does make me interested in is something like this, but purchaseable outright without renting, a la AWS Outposts but the size of the Snowcone. It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services, and it would open up some niche use cases like having a tiny server cluster in places where I otherwise wouldn't have infrastructure, like a remote research camp or rural community.

Disclaimer: I work at AWS, totally different team, and had never heard of this product until this announcement. This is 100% my personal opinion and I'm not operating in any official capacity.

>Personally, I'd probably just suffer through having to spend a week doing a slow internet upload/download rather than paying for Snowcone.

Well, I think there's two things here.

1) A lot of businesses probably won't be willing to spend a week with reduced internet capacity to upload stuff. Things we as single users might be okay with might not always translate to being a good fit for a business overall.

2) My reading is that some of the use cases for this are areas where you are likely to have limited or no internet connectivity.

From https://aws.amazon.com/snowcone/

>AWS Snowcone is built for edge computing and data storage outside of a data center. It is designed to meet stringent standards for ruggedization, including free-fall shock, operational vibration, and more. When sealed, the device is both dust-tight and water-resistant, protected from water jets on all sides. Snowcone has a wide operating temperature range from freezing to desert-like conditions, and withstands even harsher temperatures in storage.

and:

>AWS Snowcone deploys virtually anywhere you need it. It features 2 CPUs, 4 GB of memory, 8 TB of usable storage, Wi-Fi or wired access, and USB-C power using a cord or optional battery. You can put it in a messenger bag, run it in an autonomous vehicle or an airplane, or even attach it to a drone.

So, ruggedization and the ability to run this totally off battery points me towards thinking about use cases where there's not existing infrastructure to take advantage of. I guess this supported by the 'run it in an autonomous vehicle or airplane' bit I'm quoting as well.

>1) A lot of businesses probably won't be willing to spend a week with reduced internet capacity to upload stuff. Things we as single users might be okay with might not always translate to being a good fit for a business overall.

Perfect for us. We used to ship small amount of data in the scheme of things on external drives to Amazon for long term storage in Glacier. Worked great. That program was dropped and replaced by Snowball.

We tried Snowball and never could get it to work properly in our location. Amazon support couldn't get it to work, either. It was really overblown for what we wanted to, anyway.

Sending over the wire isn't an option for us.

This is a better solution for us as long as the networking issues are resolved and the pricing works out.

> Also, the articles talks multiple times about using Snowcone to upload the data for you, over your own internet connection. Why would I pay for Snowcone to do that when I could just upload directly to S3, without using Snowcone as the middle layer, and get the same speeds?

Most or at least many households in the UK I think max out at my speeds (FTTC) which puts me at about 20Mbps upload. At 2.5 meg/second that's over 4 days per TB if I've done my maths right. 8TB is over a month.

I'm looking at a house where I'd have an upload speed about 1-2Mbps, so well over a month per TB uploaded.

> It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services,

You used to be able to manage things on opsworks outside of AWS if you installed a client. Maybe you still can do this kind of thing - possibly AWS Systems Manager for non-opsworks things? I get a bit lost in all the services these days and their restrictions.

Disclaimer: AWS employee, don't work on any of the Snow products

My guess is for situations where you don't have a decent internet connection. Like some remote research base way out in the middle of nowhere. The other major application I can think of is for transferring data to air-gapped networks. You could use Snowball for this also, but that would be overkill in a lot of cases. I think for air-gapped networks this is meant to fill in the niche between a USB drive and something like Snowball.

EDIT: expanding on that off the top of my head I can think of a couple other applications. If you're using long range drones with lots of sensors (think RQ-4 global hawk, but also weather monitoring and the like) you're generating way more data than you could stream over a satellite uplink. So you could put a snowcone (or maybe several) inside the drone and use them for storage of all that raw data. On landing you can remove the snowcone(s) and ship it off to Amazon and all that raw data is available on S3 the next day. There's a bunch of other defense-type applications. Embassies could transfer these in diplomatic pouches for top secret information. In fact, if you wanted to be particularly secure but wanted lower latency, you could use the snowball to transfer a bunch of random bytes to use as a one time pad, which the embassy or intelligence center could use on demand as it transfers data back.

I would guess the real use case is: you considered becoming a Snowmobile customer but that's too heavy-duty and you need something for smaller amounts of data. Or you are a Snowmobile customer but sometimes you don't need to back up the truck for a relatively modest-in-size update.

So AWS made this.

Now they have an offering for you & everyone like you, that they can pitch to CxO execs, especially those who came from Snowmobile-using companies.

AWS already had Snowball available as an intermediate level:

https://aws.amazon.com/snowball/

For upload it doesn't make much sense. For download it's less than half the cost compared to AWS outbound bandwidth to the internet.

Holy hell. I'm looking at the pricing example on this page [0], and at $460 to transfer out 15 TB, you're telling me this is less than half the price of normal internet outbound traffic?

I knew AWS bandwidth costs were absurd, but it wasn't until seeing those numbers that it really hit me just how absurd they are.

0: https://aws.amazon.com/snowcone/pricing/

Yeah, come for the cheap compute, stay for the expensive network costs... forever.

A residential cable connection realistically offers something like 25Mbps down/3Mbps up. That's 246 days to upload 8TB!

According to Speedtest.net [0], the average US internet speed is 138Mbps down/51Mbps up. That's certainly not amazing, but on average that's only ~2 weeks to upload 8 TB.

I can certainly see where this makes more sense if you're below that average, though.

0: https://www.speedtest.net/global-index/united-states#fixed

I would add the standard caution when looking at averages, that symmetric 1/1 fiber may be pulling the upload value up significantly. Or that Speedtest users have higher internet speeds.

But I really want that stat to be true. The world of American ISPs is sad and depressing.

Average is useless here, because upload is so very restricted on some connections, but not others.

The first google result for median says it was 60/5 two and a half years ago. If you can sustain 90% saturation that's 164 days. Over half a year at 80%.

Also, what type of customers run speed test normally? Those on the more technical side, thus those running the test are likely to have higher level packages from their providers, further skewing the data

Wow! That's very surprising. It's been a while since I lived without fibre; maybe DOCSIS got a lot smarter in the last few years?

It hasn't in the US. Comcast's 1G down is still 10Mbit up

Mine is 35. Still terrible.

> It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services, and it would open up some niche use cases like having a tiny server cluster in places where I otherwise wouldn't have infrastructure, like a remote research camp or rural community.

You’re looking for Greengrass, which is hidden away in the IoT tooling AWS provide. It allows you to run Lambda functions and Docker containers on Raspberry Pi (and smaller) sized devices, controlled entirely through the AWS console.

Snowcone supports Greengrass, in fact

My employer has locations all over the world and provide services to our customers. Some of these are AWS customers as well and depends on a small edge compute that they control and knows is compliant with this and that.

For them Snowcone is perfect as it's much cheaper than Snowball and does the job of trickling data back home.

> Hardware failures on amazon.

“Someone else’s problem”

That’s what you’re paying for on any cloud provider. Any hardware problem isn’t your problem and it (should be) abstracted away.

That's true for hardware that's actually "in the cloud" (aka at an AWS data center) where a hardware failure simply results in your workload automatically shifting to other hardware. But if a Snowcone breaks on you, it still certainly is your problem, because you're the one that now has a physical brick in your office and you need to find a replacement hardware for whatever you were previously using Snowcone for.

My bad. I didn’t actually read the article!

> So is this simply a sneakernet external harddrive because physically shipping an 8TB hard drive has better bandwidth than using the internet?

Yeah. When you have to migrate 100TB data from your own data store to AWS it is much easier and practical to just send the drives. They even help you with that, for a price of course.

I'm thinking defense / electronic warfare. All the data we'll be collecting in the field needs to travel back to a data center somehow. Not sure what the rules of the game for on vs. off-premise storage in that scenario, but I still think this is one of the main demand drivers

When talking about petabytes of data is definitely faster to physically offload the data, ship it, and physically load it on an edge location.

Regarding the size of the data. Well, yes. There are companies with petabytes of data. Banking is one example that comes to my mind when thinking about petabytes scale data.

When you realize that your data is going to take 24 months to copy elsewhere (e.g., the cloud or another DC) over the fastest link you have, you have a real need for something like this.

I had up back up several years of training data off-site and we used snowball to ship a few tbs of images. Not sure it was completely necessary but it was fun

its physical edge storage, wasn't the use case clear? you can have the cloud right next to you! an entire node of a content distribution network, by itself in your rugged remote environment, but with you!

you can cloud compute separated from the cloud, but together, with your snowcone!

Reply on news.ycombinator.com