Hacker News Reader

Comment by jumploops | Hacker News Reader

Funny story time.

I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud.

Our calculations said it would cost 3x as much to run our infra on the cloud.

We pushed back, and were greenlit on creating a hybrid architecture that allowed us to launch machines both on-prem and in the cloud (via a direct link to the cloud datacenter). This gave us the benefit of autoscaling our volatile services, while maintaining our predictable services on the cheap.

After I left, apparently my former team was strong-armed into migrating everything to the cloud.

A few years go by, and guess who reaches out on LinkedIn?

The parent org was curious how we built the hybrid infra, and wanted us to come back to do it again.

I didn't go back.

View on HN

My funny story is built on the idea that AWS is Hotel California for your data.

A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region.

The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that."

The sticker shock was enough that the customer simply inherited the old account, rather than making things tidy.

Is R2 a sensible option for hosting data? I understand egress is chesp.

R2 is great. Our GCS bill (almost all egress) jumped from a few hundred dollars a month to a couple thousand dollars a month last year due to a usage spike. We rush-migrated to R2 and now that part of the bill is $0.

I've heard some people here on HN say that it's slow, but I haven't noticed a difference. We're mainly dealing with multi-megabyte image files, so YMMV if you have a different workload.

awesome. I remember reading about this a while ago, but never tried. Since it has the same API i can imagine its not daunting as a multi-cloud infrastructure.

I guess permissions might be more complex, as in EC2 instance profiles wouldnt grant access, etc.

Just to make sure nobody is confused by this - R2 has the same API as S3, not GCS. We had to build a simple abstraction around GCS/S3 to perform the migration. But if you're migrating from S3, it's pretty much drop-in. We even use the AWS-provided S3 Java library (especially convenient for making signed URLs).

Eh? I've never had a problem moving data out of AWS.

Have people lost the ability to write export and backup scripts?

My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly.

It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).

AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/

They do not!

> Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more.

https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang

Why would they make it convenient to leave?

Oh, TIL! Thanks for correcting me.

That's only for data into AWS though, not data out

Just in network costs, there's a huge asymmetry. Uploading data to AWS is free. Downloading data from them, you have to pay.

When you have enough data, that cost is quite significant.

The ingress/egress cost is ridiculously high. Some companies don't care, but it is there and I've seen it catch people off guard multiple times.

Oh come on from the description both accounts could be sitting on the same datacenter LAN.

There's a cost for data egress (but not ingress)

It’s the cost of data egress, which isn’t free.

But there is no paid egress when we are moving data between account within one region, rigth?

There is. You pay a price for any cross-VPC traffic.

This isn't true, at least not anymore.

You can peer two vpc's and as long as you are transferring within the same (real) AZ, it's free: https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vp...

Even peered VPC's only pay "normal" prices: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer

"Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."

Yes, I do believe autoscaling is actually a good use case for public cloud. If you have bursty load that requires a lot of resources at peak which would sit idle most of the time, probably doesn't make sense to own what you need for those peaks.

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck!

I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere.

I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable.

For a lot of techs though crap documentation is a deal breaker.

Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.

I agree that bad documentation is a serious problem in many cases. So much so that your suggestion to write the documentation after the fact can become quite impossible.

If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code.

(It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.)

(Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)

> Quite frequently though its outsourced development agencies that refuse to write it

It's also completely against their interest to write docs as it makes their replacement easier.

That's why you need someone competent on the buying side to insist on the docs.

A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.

Unfortunately it’s also possible that e.g the company switched from share point to confluence and lost half the entire knowledge base because it wasn’t labeled the way they thought it was. Or that the docs were all purged because they were part of an abandoned project.

> the first thing I do is update / actually write the docs to make them usable.

OK so the docs are in sync for a single point of time when you finish. Plus you get to have the context in your head (bus factor of 1, job security for you, bad for the org.)

How about if we just write clean infra configs/code, stick to well known systems like docker, ansible, k8s, etc.

Then we can make this infra code available to an on prem LLM and ask it questions as needed without it drifting out of sync overtime as your docs surely will.

Wrong documentation is worse than no documentation.

Just to be clear, after I (and a few others left), they moved everything entirely to the cloud.

Even with documentation on the hybrid setup, they'd need to get a new on-prem environment up and running (find a colo, buy machines, set up the network, blah blah).

Documentation? What for? It's self-documenting (to me, because I wrote it)!

"Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written."

I can always guarantee a stream of consciousness one note that should have most of the important data, and a few docs about the most important parts. It's up to management if they want me to spend time turning that one note into actual robust documentation that is easily read.

Reply on news.ycombinator.com