I'm spending a lot of my weekend hours racing against the clock to keep some old app engine apps online. I wish I had started in 2020 :)
I've learnt my lesson and will be sticking with boring technology I can take anywhere in future. Node and Postgres for me from now on.
this isn't academic for me - I've got a service that's on vanilla RDS PostgreSQL but growing fast, and I'll soon need to decide whether to move to something cheaper/faster but proprietary or blow my budget trying to stay generic...
I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your budget, but remaining generic?
Naively, isolating the proprietary interface in a wrapper in your code minimizes changes should you need to move — and blowing your budget isn’t a long term solution.
So does it come down to the proprietary option or fail?
There are so many moving pieces doing any migration at scale, your code is the least of the problems.
I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that.
Yes, this was with AWS Professional Services (where I use to work) doing much of the work.
Perhaps you could justify your point rather than attack other people and appeal to authority?
How is any of that relevant to what I said?
You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable:
Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc.
- - - -
Edit to include a specific example:
Let’s imagine I’m migrating a dataset, A. And my copy is A’.
In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface.
So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests.
In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’.
But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing.
In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.
They are talking to the board about strategic decisions and what will give them a competitive advantage. He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices.
If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”.
A dependency always slips in somewhere unless you are constantly testing and preparing for portability like Uber does.
Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen.
Let alone a dependency on all of the arns always having “aws” in them not thinking they wouldn’t work in China or gov-cloud. I hardcoded the partition.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
Another case is that “we are using MySQL to avoid lock in with AWS”.
Years later.
“Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”
> He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices.
Of course not — he’s paying experts to avoid that situation, aligning with his strategic priorities.
That’s my job.
You also resorted back to personal attacks because you can’t make your point directly.
> If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”.
Like this ridiculous comment to someone whose job is being hired by CTOs looking to migrate.
They literally come to me, for help doing exactly that.
> Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen.
> I hardcoded the partition.
> “Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”
Yes — that’s why companies pay me: I’m used to doing things that can’t depend on a region, service, etc while AWS engineering frequently makes that mistake (eg, all the US-East-1 magic).
That’s precisely my point:
Quality coding avoids and bounds mistakes like that, which slow your migration.
- - - -
So you’ve resorted to personal attacks, but didn’t actually respond to my specific example, and were insistent on my credentials but still haven’t shared the largest system you personally have migrated.
But please keep it up: I make my money from people like you causing CTOs to hire people like me to clean up the mess — “whoops, accidentally wrote regionalized code!” or “we need over a year to migrate 100 VMs due to self-induced complexity!”
Are you going to:
A) write an ETL job that is going to be slower and more complex to avoid the “lock-inz”
Or
B) just call run
“load data from S3…”
As a sql extension?
Are you going to tell them that they can’t use Glue? Athena? Redshift? Quicksite?
When your company needs a call center are you going to tell them not to use Amazon Connect? Now they need to integrate the call center with Salesforce - are you going to tell them they can’t do that because they have to use Lambda?
They want to have speech to text - now they need Amazon Lex.
Then they want a reporting dashboard that comes from their contact trace records. Now they need Kinesis which is going to stream to S3.
The typical company uses over 250 SaaS products that they have some form of integration with. Are you going to tell them not to use any of those either? Are you going to tell them not to use Salesforce? ServiceNow? Microsoft Office?
If you work in the health care industry are you going to tell them not to use Epic?
Which contrary to your repeated personal attacks is the sort of conversation I have with corporate leaders: they determine the strategy; I make it work technically.
But yes — there are clients who insist on avoiding those services (and generally would, if they’re migrating out of S3).
And in this particular context, we were explicitly discussing the decision of how to migrate a product into the cloud where I advocated using the cloud solution but taking a small step which would both ease that transition and any future ones — and you flipped out.
- - - -
To address your dramatic nonsense:
You don’t have to use Lambda to integrate with Salesforce; and you know that non-AWS call centers are used by most AWS customers.
> Are you going to tell them not to use any of those either? Are you going to tell them not to use Salesforce? ServiceNow? Microsoft Office?
This is just ridiculous dramatics completely unrelated to anything I said.
- - - -
Let’s ask a few questions to refocus:
- why were you so insistent on my credentials, but won’t share yours? …are you embarrassed by your largest migration?
- why can’t you address my specific example? …because it shows I have good advice and you spouted nonsense?
- why are you melting down into unrelated dramatics? …or pretending I advocated against using cloud solutions? …or integrations at all?
I think you’re just throwing up arguments at random because you can’t admit that on the original issue I was right.
But you’re so concerned with “cloud lock in” and you completely ignore the other 200+ services that the average enterprise is using. Why are you so concerned with the cloud, but not Salesforce, ServiceNow, Sharepoint, Okta, Azure AD, etc?
It makes no technical sense to spend developer time and add complexity to create an ETL job to do what a simple SQL statement can do.
None of this is “making the beer taste better”.
I’ve shared by credentials up thread - 2 years heading a migration and integration when a private equity owned health care company was acquiring companies to get big and go public, 3 years leading the application modernization effort for another company that was acquired at 10x revenue and three years working at AWS in the Professional Services department and now building out a professional services practice for an MSP for applications modernization.
And before that over 20 years as a professional developer
https://help.salesforce.com/s/articleView?id=sf.integrate_wh...
> But you’re so concerned with “cloud lock in” and you completely ignore the other 200+ services that the average enterprise is using. Why are you so concerned with the cloud, but not Salesforce, ServiceNow, Sharepoint, Okta, Azure AD, etc?
This is a fantasy you invented about me because you cannot respond to my actual point — a strawman.
My original advice was to use the cloud — and I was addressing someone else’s concern about lock-in, while giving advice that would aid their initial migration to using cloud services.
I think that it’s very telling you can’t be honest about what I said.
> It makes no technical sense to spend developer time and add complexity to create an ETL job to do what a simple SQL statement can do.
As is this — another strawman you invented.
- - - -
Why do you have two strawmen, but haven’t in several posts actually addressed my specific example?
Are you unable to?
- - - -
What you’ve said is that you spent two years migrating just a hundred VMs, which sounds like a migration delayed by self-induced complexity.
I appreciate arrogantly incompetent developers like yourself — who can’t address technical points and instead resort to fallacious thinking:
Your work gets me paid.
So I’m an “incompetent developer” who someone managed to get hired at AWS Professional Services not a third party partner - you to consult the largest organizations on the planet?
I’m sure an AbstractFactoryRepositoryFacade is going to prevent cloud lock-in while you advise people to create a complicated workflow when a simple SQL statement using an AWS extension would be faster to develop, more performant and less maintenance?
It’s not a straw man it goes to the core of the efficiency gains you get by taking advantage of your vendors functionality and integrating into it tightly instead of trying to maintain a leaky abstraction.
You're always "locked in" to some degree and it's an almost worthless thing to invest into if you never actually need to actually migrate off.
The difference being whether you are internally maintaining your special snowflake, or else paying for a solution maintained for you by a 3rd party.