I was told under no uncertain terms not to even think of touching this VM because “the budget has been approved”.
I was shocked at the flagrant waste of money and assumed it was a one-off aberration.
Nope, for months afterwards I kept hearing the same refrain from manager after manager, from product owners and dev team leads.
“Don’t touch! We fought hard for this budget! You’ll take it from our cold dead hands!”
Eventually I soured on the whole idea of cloud cost optimisation a service for unmotivated third parties and gave up on the whole notion.
"I'm going to reuse this VM, to help our ... fleet scale better."
That way your management continues to use their allocated budget, and your real prod systems work slightly better (also will eventually require less additional $ to scale up - helping the company i.e. shareholders).
The thing to remember:
You would assume all middle management really manages are a top line and a bottom line. Numbers related to their KPIs/OKRs are roughly a top line, and numbers related to their resources (humans and cloud infra budget) are roughly their bottom line.
The reality: Middle management's resources (humans and cloud infra budget) are not their bottom line. Middle management gets rewarded (promoted) when they have "enough scope", scope has roughly always been defined by number of people (it now also includes things like cloud cost budget). As such middle management has to say "we need to do more with less", but they are promoted based on these numbers going up!
Is this reward structure in the best interest of companies (i.e. customers and/or shareholders)? No, neither. Is there a better system? Not yet. Is the reward structure created by middle management for middle management? Likely.
So in the meanwhile, if you don't want to become unmotivated, might as well work within the current reward structures.
Heh, I tried that too! I found a lot of setups using old HDD disks that were at 100% of their IOPS limits and CPUs that were idle. When they were built, Azure didn't have Premium SSD, so that's forgivable. I offered to rearrange where the expenditure goes, such that they have newer and faster CPUs, Premium SSD, but fewer cores which means a reduction in licensing costs. Cost neutral while improving their performance and capacity.
That got a very firm "Nope! Nope! Nope!" from most (but not all) teams as well.
> As such middle management has to say "we need to do more with less", but they are promoted based on these numbers going up!
I came to the same conclusion. Many project managers or product owners introduced themselves in the first meeting by proudly proclaiming the huge size of their operation. I.e.: "The system I'm responsible for is a multi-million dollar project with a small army of developers!"
I've also noticed that there's a tendency to exponentially blow out complexity of what ought to be a trivial system for the same reason. Something that could be static HTML on an S3 bucket or Azure Storage Account turns into a microservices monstrosity draped across three clouds and four external SaaS services.
Those resumes don't pad themselves.
One of my managers was a champion at that, taking scraps from everywhere for undercover projects. One of the few who managed to get things done in a highly bureaucratic company. Also helped by the fact he is good at picking competent people.
Note to apple: please start concentrating on the enterprise sector. We're dying over here. My Dell weighs 3x my personal M1 MBP, has a shitty keyboard with keys designed for Borrowers, the battery lasts 8 minutes and it reduces my sperm count if I put it on my lap. It feels like I have a ball and chain around my ankle 24/7. My only escape is WSL2 which is broken as fuck as well (can't run services, cron jobs, X problems etc) and we can't install a simple non-WSL VM on the node because Device Guard requires hyper-v to be enabled excluding sensible and pure VM options like VirtualBox. Docker for windows is a comedy of errors too.
6 years ago our department of 200 people "went rogue" and started provisioning Macs, because it was the only way we could hire developers and provide a good developer experience for the work we were doing.
This took some convincing, but it was possible. We agreed to be unsupported by the in-house help desk, but we had 2 people in IT that supported us for provisioning and fixing machines, and sorting out a small amount of required enterprise software like a Cisco VPN client and some fleet management background agents.
Otherwise, we self-organized support over Slack and in-office and also made use of Apple's business support directly.
As of earlier this year, our department is now over 600 people, and we've given our internal IT enough incentive to officially support Macs, which they now do, alongside Windows.
They use some kind of MDM software to manage and update and monitor our Macs the same as they do Windows.
There are also now additional much larger teams in the organization exploring Mac adoption where it makes sense for their developers too, and we could soon have thousands of Macs in use.
So it's definitely possible, even if you have to start small.
I sympathize with you though, it sounds like there's no will to do it, which sucks :(
Where I work, every conference room in every corporate office has a TV and a dongle that you can plug in via either HDMI or USB C.
2. I have found that face-to-face meetings where other people bring their laptops usually means that they get distracted and don't participate.
It’s also not like I wrote my first line of code in 1986 (Hint guess what the 74 in my name signifies).
Now that hopefully we can dispense with the idea that I don’t know how this computer stuff works. Please answer the question. How do I get everything I need from my computer to this thin client? How do I keep my environment in sync with this thin client?
You should just walk away.
Just send me a fucking workstation. Nope too hard.
That said, we also supply external monitors, keyboards and mice for anyone who wants one, and every worker has a discretionary budget to spend on home office accommodations (chair, standing desk, etc).
I also suspect if a developer specifically asked for a desktop computer and could (lightly) justify the need for it, we would get them a desktop, and a laptop.
Granted, macOS changes a lot between each release but our company is paying for Docker Desktop licenses and the experience has really been disappointing.
Enterprise sales are where the customer is not the user. Apple does best when the user is the purchaser.
Also, I know for a fact that Macs are well supported at scale by many large tech companies including my own.
We all use Macs at work so we knew it was a matter of time before we were on ARM. I’m glad we made the transition. M1 airs are a delight to work with and Graviton machines are great bang for buck.
EG: GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64... likewise aws-cdk literally could not run an arm64 fargate ecs deployment for months after support was added (They simply did not support the required attribute in the container definition).
I would love to see this change as I've had nothing but great experiences with graviton for virtually anything arm supported.
I've found arm64 builds on amd64 take longer when using one build context/arch (but doing multiple platforms), but that's as it's being emulated.
It's the oppostite on my M1, the buildx amd64 takes longer.
I've got to say though, I've not experience that delta in build times, even emulated on my machine.
What type of container, and on what runner? That has not been my experience at all, a cross-compiling buildx build with Python and a bunch of libraries takes only slightly longer for arm64 than it did for x86.
Second our laptops and our CI are amd64 machines, and being able to run the same docker images in prod and locally is nice, and not having to build the image with qemu on the CI is also good.
I don't mind cloud-ARM, but there definitely are good reasons not to use it (which of course don't apply to everyone)
The whole proposition relies on the idea of a sunk cost being accepted.
So yes, back to servers please. IaaS should be the maximal offering that is accepted by a business from a risk perspective unless the tool or technology is disposable in a 6 month window. There is space there for gains. PaaS hell no.
Edit: worth mentioning that AWS support is somewhere near dire. We've had issues with multiple services and despite being a VERY high roller with enterprise support we can't get anything fixed in any reasonable time. It's just someone else's crap you're using and they aren't any better at it than you are, just adding lead time to any issues. In some cases I've had to actually call out complete bad implementations that break function guarantees provided by open source projects (I can't logically warn people away from services as it's pretty obvious who I am if I do). One rule I've developed is that if it's not a core project: S3, EC2, EBS, ALB etc then it's probably a commercial liability in some way. There are no people working or with any knowledge on some major bits of AWS infra.
8x?? that's crazy, what where you doing wrong then?
SMEs can't reliably manage that transition with any skillset and still deliver a product at the same time.
Did you use reservations to reduce costs?
Was it a lift and shift with VM configs staying as-is? (I’ve seen a lot of empty 1TB “app” drives burning money in the cloud!)
You complain about PaaS services, but I can’t imagine 8 data centres worth of stuff being converted to PaaS in hurry!
So it wasn't purely cloud/renting racks issue
How is that possible?
E.g.: With Kubernetes and AWS you ought to be able to use clusters with a base of "reserved" capacity plus spot pricing for peak hours on top, right?
From what I've seen (in my limited experience), that should reduce costs for most orgs, not increase them!
Please someone hire me so I don't have to live through this nightmare any longer.
TLDR: AWS is really expensive compared the equivalent compute elsewhere, to the point of overwhelming its advantages.
---
Long answer that I wrote before realizing how long it was:
I'm not the person you're replying to, I don't know the exact details, I don't know their company. However:) I can tell you that AWS is a factor more expensive than renting or owning your own metal. They (claim to) offset this by 1. taking care of management for you so you need fewer ops people, and 2. letting you scale up and down as needed. The first is plausibly legitimate, but really depends on the size of your compute and the size of your human teams (and how much benefit you get from AWS-managed offerings). The second can help, but only if you've got really spiky workloads, that are only really running a tiny fraction of the time, with absolutely tiny base load. Scaling up and down helps reduce your AWS bill compared to not scaling up and down on AWS, of course, but it doesn't help that much when their elastic offerings are that much more expensive. Say, for the sake of argument that you can run most of your EC2 instances just 6 hours a day, during peak hours. That lets you win... if, and only if, EC2 instances are less than 4x the cost of just owning your own bare metal machines. I've gone so far as trying to price out using spot instances for CI work - just selectively, when they were the cheapest! - to augment instances on Hetzner Cloud. Guess what? They're so expensive that even at spot prices EC2 is a factor more expensive.
I just priced up a 128-core (dual EYPC) Dell server with specs comparable to a matching Azure cloud server (HB120rs_v3), and the "1 year reserved" price for Azure came out to about the same as the purchase price over 3 years. The 3-year reserved price is about the same as the purchase price over 4 years. There's a new 5-year reservation option, which is the equivalent of owning the hardware for 7 years. Spot pricing is the equivalent of amortizing the purchase price over 12 years!!
Meanwhile, using the cloud, you can upgrade to the 9004 series in just a few months, not 12 years from now. And then whatever comes after the 9004 series in like 2 years, not 5, 7, or 12.
So it looks like that at the larger scales, the pricing is very directly competitive with on-premises hardware.
Consider that the cloud hosts include most basic operations costs, such as cooling, electricity, data centre floor space rental, the SFP ports and cabling, etc, etc...
Speaking of which, a quick back-of-the-envelope calculation is that a 128 core server will cost about $20K in cooling and electricity over its lifetime.
Note that the Azure HB120rs_v3 sizes come with 200 Gbps InfiniBand "just thrown in" for laughs. Try pricing that up some time, in case you want to build your own hyperconverged infrastructure!
Admittedly, at the smaller sizes, Azure and AWS are less competitive, but you do get flexibility, automation, and a bunch of other stuff that's difficult and expensive at scale on-prem.
Like there's just no point in coloing when you're small because either all non-server bits will cost you for no reason or you're using something managed which is just cloud but more annoying.
As for avoiding ARM, we do only x86-64 because corporate security policy demands that we have Windows laptops so that some box ticking overlord can fill out a security policy compliance form. That means we're stuck limping along with docker and WSL2. Every single engineer in the org has an arm64 machine at home already and wants a proper computer at work, which can ironically work in the same policy framework if anyone gave enough of a shit to deal with it.
So that's why we don't use Graviton; corporate security policies. Our customers will just have to eat the price hikes.
But it's important that all builds are 100% reproducible on all build targets and that includes non docker ones. That is much more difficult if you have to cross compile stuff. We can barely manage one architecture.
No cloud provider will give you the option to create as many instances as there are available ones, they all have limits from the get-go. Usually you have to write them/fill out some form if you want to go above the standard limit, Hetzner Cloud as well.
> Your account is too new to request a limit increase. Please note that we generally do not answer questions regarding limit increase on the telephone.
Not no reason; it adds work and risks incompatibility. Now, that work might be relatively small, and most software these days is compatible with aarch64, but compared to amd64 (which is the de-facto standard, already supported by everything, the default without needing to set anything up) it's still something, and businesses are risk-averse.