CTO Headaches: Top cloud-to-cloud migration woes (and how to solve them) | Hacker News Reader

CTO Headaches: Top cloud-to-cloud migration woes (and how to solve them)

cncf.io

66 pointsllarsson30 comments

To me, if you're using AWS, GCP, or Azure, you should be using their "hard to migrate away" services. That's the value of the brand name cloud platforms. Plumbing that's done for you.

If you're going to the trouble of being agnostic, then you might as well deploy to a way cheaper bare metal datacenter and enjoy market rate storage, compute and egress.

Running cloud agnostic on AWS, GCP, etc, is the worst of both worlds.

Thank you. That brought a new perspective to me.

Seconded, I thought it was very insightful.

I’ve always said this to our CTO. If we were going to use AWS, we should go all in. Once we stopped trying too hard to avoid lock in, we stopped halfassing our architecture.

And if AWS hikes rates (or reduces discounts) on you once their algorithmic AI determines your switching cost exceeds the rate hike?

"Oh they would NEVER do that"

And these gee-whiz heavy-binding services would NEVER steal your data.

The strange thing at work is people bend over backwards to use AWS stuff:

"I made this virtual network card that I can detach and attach to another VM when we want to keep the same address for a box!"

How about we use DNS.

AWS services are great for things like their Postgres as a service and other things that fundamentally have standards based access.

But the others like big data streams will be a bit more convenient in the beginning and then you're beholden to their supported features and the fact you are handing them visibility to your data.

> And if AWS hikes rates (or reduces discounts) on you once their algorithmic AI determines your switching cost exceeds the rate hike?

> "Oh they would NEVER do that"

Interestingly, what you insinuate is exactly the opposite of actual reality when you look at what has happened over the years: things have gotten CHEAPER.

Off the top of my head:

- Lambdas are now billed per 1ms instead of per 100ms which will be a significant cost reduction for everyone using Lambda

- EKS started at $200 / month for the cluster and is now at ~$72 / month

- Each bump in EC2 instance generation is always cheaper for the same or more performance

- DynamoDB added on-demand capacity which is at face value more expensive, but allows you to scale to 0 and will usually result in an overall cost reduction

- This December they announced GP3 which can scale storage and IOPS independently allowing you to much cheaper scale up your needs for more IOPS without paying for TBs of unused storage

and probably many many more.

That's AWS just keeping itself from being irrelevant. Every cloud provider is pushing that, and in order to acquire customers and maintain customers, they have to do that.

Right now, times are good. Money is good. Top spot. Discounts if you can negotiate it.

But eventually this will commodify more tightly (k8s is just a warning shot).

When you buy "binding" AWS services, it is the "nobody got fired for picking IBM" with a BIT more market pressure. AWS non-commodity services are the new IBM mainframe.

Nice straw-man. Keep HN being HN, two thumbs up.

OP here.

I agree with you: if you know your deployment is, and always shall be, on e.g. AWS, you essentially lose out on great integration if you try to be cloud agnostic.

However, as in the introduction:

"Whether motivated by the need for a multi-cloud strategy, expenditure minimization, legislative or regulatory demands, or simply to get closer to end users, many organizations find themselves migrating from one cloud to another."

I work with many customers that migrate away from the big US providers because GDPR and the "Schrems II" ruling makes it legally questionable to continue to use these services for certain data processing.

Or ones that have been bitten hard by cloud outages, causing them to fail to deliver service to their customers. Getting a tiny rebate on your cloud bill that month does not begin to cover the lost reputation due to downtime.

>If you're going to the trouble of being agnostic, then you might as well deploy to a way cheaper bare metal datacenter and enjoy market rate storage, compute and egress.

Interesting. I wonder a which scale this becomes true. It's been a long time, so I'm not current on rates, but I've been involved with a couple migrations from datacenter to AWS and AWS was substantially cheaper for the same level of infra in both cases. These were smaller use cases, though.

What about the case where you're not migrating away, but do need to support multiple clouds for regulatory, failover, customer requirements, etc. reasons? Is there a way to still leverage the first-class features of, say, AWS and GCP without resorting to one of these issues:

- death by abstraction of cloud differences

- teams incompletely reimplementing features on each service

- giving up and going to bare metal

Not really a way to leverage the different services by cloud vendors, because the services really are different. So it is either the abstractions you mention, or going with self-hosted. And the latter cases, you really start to appreciate cloud native tech such as what you find at the CNCF.

S3, SQS, RDS, etc. are some damn nice primitives and also have analogues across all the clouds. The headcount to run their open source equivalents to the same SLAs is not free either.

Disclosure: I work at AWS, and might surprise you that I disagree. Opinions my own.

I see this line get touted a lot but it is almost never accompanied by any depth. It sounds good, but what does it actually mean in practice? Do you have concrete examples of what using the best parts of AWS, GCP, and Azure are? From where I'm sitting, the best parts of the cloud were created 10-15 years ago: On-demand compute (without dealing with the joys of subnetting, DHCP, and configuring Linux interfaces), object and block storage, managed databases/DNS/load balancers. The newer features tend to lack the things you really need in a production ready service: monitoring, debugging, configuration, etc.

The right answer to this multi-cloud question is to stop having so many different components in your stack. Be painstakingly deliberate about what you adopt and have someone who can articulate (ideally in writing for posterity) why you're adopting it. Pick a few foundational primitives and master them operationally. One relational database, one web server, one load balancer, one cache, zero message queues ;).

(Spoiler alert: this is what all of the cloud vendors themselves do internally.)

(also work at aws)

i think that spoiler alert is a little overgeneralized.

first, there’s plenty of advances in those categories in the past 5 or so years (lambda, fargate, qldb, a bunch more) that move the needle significantly on the versions of things that existed 10-15 years ago. we didn’t collectively get everything right by 2012, and i heartily endorse customers using things that take away operational complexity but are less “complete” compared to the lower level counterparts.

second, you missed mentioning that circular dependencies is a very real reason for why cloud vendor service teams can’t (not won’t) use services that would otherwise benefit them. this is a problem that is much less apparent to customers of clouds because, to them, the suite of services is provided as a single flat layer of options.

i do agree that you want to make sure any service you use is “complete” for you, but that completeness can exist on a spectrum and sometimes getting onboard early and driving their roadmap can benefit you greatly. i also agree that you want to be conservative with how many services you do use, because each one without exception will become a learning experience.

"It sounds good, but what does it actually mean in practice?"

I think what I was saying is pretty straightforward...

"If you're going to the trouble of being agnostic"

Meaning that if you're trying to build some sort of facade that makes AWS and GCP look the same, you might as well just make a facade that points at something cheaper than AWS/GCP. AWS/GCP markup for things like internet egress, storage, and compute are high, partially because of all that plumbing. Any facade puts you in a lowest common denominator situation where you're paying for lots of stuff then not using it.

Or, in short, "cloud agnostic" running on cloud is a weird idea that throws out lots of the value add. Perhaps there are some exceptions for very simple architectures, as you hint.

> I think what I was saying is pretty straightforward...

I was asking you to give specific examples.

> Perhaps there are some exceptions for very simple architectures, as you hint.

Simplicity should be the goal and the starting point.

S3 and DynamoDB would be the two I’d mention first (since know it or not, folks are relying on them anyway), then things like Fargate, QLDB, and PrivateLink.

Agree on the simplicity point, though the key is not just simplicity for coding, or for communicating about the architecture, but emphasizing simplicity of operation and honesty at the skills, abilities, and criticality of operating and resolving things under stress.

Several years as AWS CAB member show this topic is hotly debated. I think there’s this “third way”, to purposefully exploit the different DNA of the top CSPs.

I choose that word “exploit” to appeal to the board room, to feel they get more than they have to pay for (which I think is true, thanks to CSPs’ focused spend). A more friendly way of putting it from CSP point of view is “leverage what we’re best at”. Rare case of genuine win-win.

> From where I'm sitting, the best parts of the cloud were created 10-15 years ago

I just moved from a startup that was running things on services that were either created or only really matured within the past three years, to a different startup running on tech from 10-15 years ago.

The differences in Ops overhead and scalability is huuuuuuuge to say the least.

Modern infrastructure on AWS for me would be:

- Lambda

- DynamoDB

- S3

- Fargate

- EKS to a certain extent

- And extremely excited about Serverless Aurora V2

Older infrastructure with massive ops overhead:

- EC2

- ElasticBeanstalk

- RDS MySQL/PostgreSQL

Every year at AWS re:Invent I see the needle moving into more serverless stuff, and for a very very good reason.

To maybe take up some of your points:

> On-demand compute

EC2 and RDS are the absolute bare minimum you can do for on-demand compute and was the initial introduction to the elasticity of the cloud. It's slow and cumbersome to scale up and down, and also very costly to do so (RDS is basically a x2 of you instance size every time).

That's skipping automatic scale - for RDS? Forget about it.

> The newer features tend to lack the things you really need in a production ready service: monitoring, debugging, configuration, etc

State of anything running on EC2: You are 100% responsible for getting your metrics and logs out of the instances, you get close to worthless OS metrics by default, that might tell you info about some scaling events, but no insights into you actual applciation.

State of serverless: Built-in integrations with CloudWatch for monitoring and XRay for profiling. Out of the box you have error reporting on Lambdas along with DLQ and much more.

> without dealing with the joys of subnetting, DHCP, and configuring Linux interfaces

I honestly hold the opinion that if you deal with things at this level of the infrastructure stack, then the "modern cloud" is not relevant to you (AWS, Azure, and GCloud), but you would fit better or something like Hetzner or with dedicated servers.

I always thought that "cloud" was overhyped and never understood why you would want to use stuff like EC2 over a dedicated server somewhere.

Honestly, the idea that the new services have a "higher level of abstraction" and that old services are an insufficient extra level of abstraction is something I could never articulate properly.

Thanks.

Aren’t message queues a basic primitive in a highly available system ?

I respectfully disagree.

There's a sweet spot where companies can use the lowest foundational building blocks, build their own stacks on top, and save resources (comparatively to running your own datacenter).

Things like EC2, EKS, S3, and RDS Aurora can be considered foundational but you don't really have to use Athena, CloudWatch, or RedShift (they are super expensive anyway).

Absolutely, and not said enough — but do this in a risk managed way. Hold yourself accountable to really give that thought. Terretta doctrine on this is “Aggressively exploit CSP differentiation, with a credible migration plan you keep in the drawer.”

To borrow an AWS metaphor on “One way doors versus two way doors” decisions, do pre-think, but don’t pre-code. Just think as you design the house about whether you might someday want to go through that door another way. It’s inexpensive now to make design decisions that make that easier later.

At an enterprise, fully understand the BCP or business continuity plan. You will likely be stunned how long functions are considered OK to be offline for regulator-compliant BCP planning purposes, relative to the SLAs that I.T. gets yelled at about.

Your job before committing to vendor lock in is to write down a plausible plan to get out of the lock in that fits (when effort is doubled twice) inside the BCP plan window, and maintain that plan as part of the platform documentation for the capability in question.

When you study the economics of re-investment by Azure, Google, and AWS, it’s impossible for an enterprise to achieve the same capability-evolution curve without floating on that rising tide. The hard part is to figure out how to not get capsized by change waves or get swamped if the CSP pulls the plug in your boat. These are part engineering, part business/procurement/legal problems, but much less costly than holding yourself to either a lowest common denominator or fully abstracted principle.

Startups tend to get very attractive credits on AWS et.al. This is carefully designed so that by the time the bills start to roll in and become surprisingly huge, it's way too late to try to unroll the dependencies on AWS.

There's definitely value in keeping options open by sticking to standard processes on Linux VMs so there is a painless migration path if and when the CFO freaks out.

“ But what I’ve also seen is that if you are using a managed Kubernetes service, your users and permissions handling is perhaps tied not to Kubernetes role-based access control (RBAC) features, but rather, to cloud-specific offerings. Like AWS Identity and Access Management (IAM). Great service, but ties you to the AWS platform. “

- precisely why Kubernetes as a proxy for “if we build on K8s, we can easily move across clouds” from an infrastructure point of view is not true. If you develop your application for K8S, then yeah, by all means good for you. But, don’t say your reason for taking on the complexity of Kubernetes as a platform is the ability to become multi-cloud easily.

Four days ago, AWS rolled generalized OIDC support for their cluster auth. Presumably, this makes it easier to transition providers since the user store can remain consistent.

https://aws.amazon.com/blogs/containers/introducing-oidc-ide...

I don't know if this will penetrate upward to the CTO level, but 'rclone'[1] is an open source, standards based solution to cloud-to-cloud migration.

You can use rclone - and all its powers - without installing or running it locally:

  ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah

... as there is a cloud storage provider with rclone built into the platform.[2]

[1] https://rclone.org/

[2] https://www.rsync.net/products/universal.html

I would hope a chief technical officer with their salt would realize how powerful this setup is. Unfortunately, the lack of a cute name, branding, and marketing engine behind it may cause some to dismiss it as 'it's too simple, it must be missing something.'

At a certain scale anything purporting to be as simple as a one-liner CLI program is suspect.

Larger organizations get messy, it's (almost) never as simple as "clone from S3 bucket X to drive folder Y".

Yeah, that migrates data files from A to B. And that has tremendous value, as your command line shows!

But, it does nothing for making, e.g., the message bus service that AWS offers compatible with the one Azure offers, though, so your application will surely know something's up when it doesn't work any more.

Reply on news.ycombinator.com