Hacker News Reader

Comment by onychomys | Hacker News Reader

So I must be missing something here. They drive to your workplace with a truck and drop off a bunch of servers and wires and stuff and get them all set up for you. Then you log into them and use the same APIs that you'd use if you connected to a normal AWS bucket? Why not just do the cloud server instead? Is this for people who have to move/store very large amounts of data such that transfer time is an issue? Or is it just a thing to let you not worry about sysadmin-type stuff because the Amazon dude does all of it for you?

View on HN

This is for businesses which have regulatory requirements that they store data in-state. State governments don't like it when certain data is stored out of state. They like the idea that they can just dispatch their own police to seize the required records from the server rather than needing to ask for the cooperation of another government and its own police force.

I know specifically its popular in the regulated gambling and insurance industries.

Yes - take for example gene sequencing - that requires a huge amount of data shuffling and going cloud for that is infeasible simply for the data transfer costs.

With Outpost this becomes feasible. This "cloud in a box" idea has been around for a long time, but I guess Amazon has enough brainshare for the tooling that it may make sense for companies to have this for data-intensive compute and keep the same skillset requirements as the rest of their fully-clouded operations.

I do genome sequencing and this is very true. You are either all-in on cloud or do the processing completely in house. The data egress fees are more than enough to offset any benefit to a hybrid approach.

I’m more curious if there is an equivalent to S3 for on-prem. I bet not due to reliability constraints, but something like this would be nice. I can’t say there would be much benefit to using AWS Outposts for genome sequencing without an S3 local cache.

If you are asking if Outputs includes S3, yes, https://aws.amazon.com/s3/outposts/

If you are asking if there are stand-alone S3 API offerings, you might want to look at MinIO. Reliability mostly requires extra hardware and configuration - https://docs.min.io/docs/minio-erasure-code-quickstart-guide...

I didn't know they also had S3 for outposts... that's interesting. That also requires an Outposts rack as opposed to the servers the original article was talking about.

> 380 TB S3 tier is priced at $0.1000/GB-mo ($38,912.00/month)

But it's also extremely expensive. $38K per month for 380 TB of storage. Even if we assume 3X redundancy, you can get 500TB of raw capacity (12-16TB HDDs) for ~ $25k. After 3 months of usage, you'd have already covered that initial expense. I know there are other benefits to having managed services, but again, cost is definitely prohibitive for many use-cases.

I use minio locally to support S3-workloads on our local cluster. We can then migrate workloads to AWS/S3 proper with minimal changes, if needed.

How do you even get the petabytes of data into S3 without incurring massive charges? Previous estimations of getting the initial dataset into cloud was.. prohibitive (note this was 3-4 years ago).

What charges in particular are you referring to? The cloud services only charge data egress fees. Data ingress is free.

The ENCODE Project data were a hundred terabytes or so at the point I migrated it into S3 (it's now over a petabyte.) We were on a fast academic network with a 100GB interconnect so could just `aws s3 sync` rather than worry about using AWS Snowballs or anything.

Uploading data into AWS is free.

Regulated industries of any kind (whether legally or pragmatically), latency, security (not necessarily for what runs on Outposts, but for what Outposts connect to), etc. etc. while treating it essentially as another amazon region.

In the early days of the "cloud" it was common to talk about "shifting loads to cloud" as temporary scaling measure, where cloud services were not default and instead an extension of your on-premise setup - this is the opposite way around, extending cloud to on-premise.

> Why not just do the cloud server instead?

Having explored this previously in the context of this (which was mostly vapourware when I looked at it a couple of years back) and the Azure equivalent:

1. Regulatory requirements. 2. There are no network - and hence egress - charges to the rest of your infra. 3. Limited ability to operate disconnected.

On the last point, the Azure offering is vastly, vastly superior, since the AWS one used to phone home and stop after a few hours, which undercuts the value on the third point. There are quite a few places with no local cloud presence, and unreliable connectivity to the nearest one; for example, last time I was in New Caledonia, there were multi-day outages and degradation to their sole fibre to the world. They do significant research workloads (thanks to the SPC) and something that could endure that while allowing you to keep using cloud APIs would be very convenient.

The attraction is "works like AWS but with high bandwidth and no ingress/egress charges between the Outpost and your on-prem equipment". Which doesn't help with every use case, but helps with many.

Reply on news.ycombinator.com