It's tough being an Azure fan
alexhudson.com
alexhudson.com
The offending process costing me money didn't appear on the Azure console and I had no idea it was running, how to access it to stop it or even to know what was going on. When it turned up on my bill I nearly had a heart attack.
Thankfully they let me out of it once I pointed out I was getting billed for something I couldn't see. I appreciated that greatly but I've never gone back to Azure and the experience scarred me so much I don't think I ever will. This was about 4 years ago so more than likely they have sorted it out.
They keep reminding me I have $50 of test/dev credit on Azure through my Visual Studio subscription but it flat out frightens me to even try to use it.
AWS isn't perfect but at least you have half a chance of working out what is costing you money through the billing console.
This page should show you a full breakdown of exactly what’s costing you money across all regions.
> Dear Amazon Web Services Customer,
> Your AWS Account is about to be suspended. There is still an outstanding payment problem with your account, as our previous communications did not lead to successful payment of your past due AWS charges.
> We were unable to charge your credit card for the amount of $0.26 for your use of AWS services during the month of Aug-2021. We will attempt to collect this amount again. Unless we are successful in collecting the balance of $0.26 in full by 09/30/2021, your Amazon Web Services account may be suspended or terminated.
The balance never increases, even though it always says the charges are from the previous month. It's as though they forget the balance each month because it's not worth collecting, then my account runs up another small charge, which they promptly forget.
And of course they have yet to terminate the account. I would gladly close it myself, but I'm locked out, and it's not really worth my time to figure it out how to get back in (plus they keep telling me they're going to terminate it, which is exactly what I want them to do!).
What a dysfunctional service.
AWS have a setting in their billing that can fire off if you have exceeded a certain threshold which would be pretty much invaluable if you're running something that auto-scales.
The first few times I used AWS for tutorials, something similar happened to me. I thought I shut everything down, but kept getting billed and wasn't able to find it without contacting them. It was just a few dollars, but I've been wary about any services where you can't cap the billing.
Cloud platforms generally don't let users cap the billing, because those overages are good income for them. I prefer using services like DigitalOcean or Linode where you can be sure that your new site crashes for 15 minutes instead of bankrupting you.
It can be really frustrating and annoying when you can't hunt down where costs come from.
AWS and Azure both have pretty straightforward free tiers, but people end up accidentally racking up large bills anyway. A UI to see a list of running services, sorted by cost either doesn't exist or is not prominent enough.
It's slow, you can't open links in new tabs, one wrong click and you have navigate to the right page and wait for everything to refresh again, but you can get the list of billed items. Plus a "Other subscription charges" that contains some mystery costs not linked to anything.
Hard to believe that really is the case for a product targeting serious customers in 2021. Doesn’t that make a surprise bill still possible?
And it makes sense from a growth perspective: new products grow revenue, bad billing systems only annoy customers (but mostly invisibly).
The UI to see a list of running services, sorted by cost is precisely what Vantage offers (and more)
By definition, I don't understand AWS, so figuring out how to turn it off was nearly impossible. AWS "support" didn't exist. Stackoverflow AWS geeks were in high dudgeon that I'd ask the question, "how do i disable this?" and would kill my question. Finally some kind soul did give me the trick to finding the last service to disable.
BTW the tutorial is absolutely useless. Just a thousand different incantations to repeat. No real understanding communicated. Felt more like an ad for myriad services.
0/10 would not recommend.
The reason those consultancy firms exist is because billing scales terribly. Once you’re a business using AWS you’d likely have a multitude of projects running across a multitude of departments which need to be billed to a multitude of different customers and internet cost centres. This all needs to be processed by an internal 3rd party financial system managed by non-technical people who wont even know what AWS stands for let alone what it does and how it works. In those situations the problem of billing becomes exponentially more difficult than a one person hobby project.
And yet, we have horror stories of students and even experts being hit by surprise AWS bills.
Although I agree about usage - I never go anywhere near AWS unless someone else is paying for it.
> And yet, we have horror stories of students and even experts being hit by surprise AWS bills.
They obviously didn’t bother to manage it. But that doesn’t mean it’s not manageable. It takes all of 5 minutes to set up a budget alarm on CloudWatch. It was one of the first things I looked into doing when I set up my own AWS account years ago specifically because I didn’t know what I was doing back then and thus didn’t want any surprises. If I managed it then, then I find it hard to believe others cannot too.
Billing alerts aren't good enough.
Consider a scenario where you're consulting on someone else's small- or medium-sized project and your bug costs the client a huge amount of money in the middle of the night. Now who pays? Say goodbye to your paycheck or reputation, even though it should have been preventable.
Another scenario: you launch a startup, and a bug empties the bank account and kills the company. If the solution is to just not use things like AWS and GCP (including Firebase, which has no billing cap) when you're getting started, why are they advertised that way?
I’ve built solutions for customers who would prefer an unplanned outage over an unplanned $250k expenditure. Many customers think that I’m a dinosaur for saying it, but if there’s a 2-5 year expected lifetime for something, it’s almost always more cost efficient to use traditional colo or VPS.
Also, operationally it’s possible to have something more than all or nothing. Big companies usually have “Tier-0” services that must be up all costs. AWS is no stranger to complexity - this type of function doesn’t exist because it would cost them money. They probably make 9-figure money from obviously idle services.
AWS is like Lego. You’re supposed to build on it to create the behaviour you want
If you are out of money, all of your services needs to be destroyed immediately: including all of your database hard disk drives, because every single piece of infrastructure yields some regular cost.
It means that it is NOT going to be “a simple unplanned outage”, it’s going to be more akin to formatting your hard drive with all of your family photos on it.
Pretty certain AWS just finds it easier to sometimes write off costs rather than implement something so radical that customers may be even less happy afterwards.
You can also set alarms that warn you of projected usage.
> Now who pays? Say goodbye to your paycheck or reputation, even though it should have been preventable.
If it’s legitimate usage then I’m not really sure what you’re advocating; are you implying a service being suspended in the middle of the night because a hard spend limit has been hit is somehow better for your reputation?
Or maybe you’re suggesting that it’s not legitimate costs, in which case you’ve set AWS wrong to begin with and thus your reputation probably deserves to be queried.
> Another scenario: you launch a startup, and a bug empties the bank account and kills the company. If the solution is to just not use things like AWS and GCP (including Firebase, which has no billing cap) when you're getting started, why are they advertised that way?
No cloud service operates that way. In that situation you’ll almost always get charges refunded. Even in instances of gross negligence (which would be the case here since for the bank account to be emptied it means you’ve not been watching your spend for more than a month and no business should operate that way)
I do get the points you’re trying to make but I’ve been working with the cloud for some time and have seen plenty of horror stories, all of which were due to gross negligence and most of which were still refunded by AWS as a gesture of good will. They’d much rather have your repeat business than burn their users with bills that cannot be paid.
That's $9/month until I disable that credit card, which I might do one of these days.
I thought they'd notice and disable my account after I successfully disputed the charge one time, but the bills just keep rolling in each month.
This might be a bit overreactive, but... I'm building an MVP for a SaaS app and I sure as heck am not going to host it on AWS.
Happy to help you if you need any assistance
Disclaimer: I don’t work for Amazon but I do work heavily in AWS.
It's been 5 years since I was responsible for anything AWS, but back then at least support was surprisingly good even when spending less than $100 a month. Emails would get answered and I could often get someone on the phone within 24 hours if I needed.
I share this sentiment after going through a lot of tech tutorials and onboardings. And it's not limited just to AWS. I find myself forgetting most of the information I was supposed to learn. I'm trying to be extra mindful and offer additional explanations when I'm writing procedural guides myself, but I still have a lot to improve.
No. They don't have caps because it's really hard to implement and because the negative press of an app going down because the cloud didn't scale is far worse than any received by surprise bills. Scaling is a large chunk of what you're paying for, after all.
This is quite obvious when you look at how lenient AWS is with retracting surprise bills. And the money perspective doesn't make sense, either: AWS is living on customers that have bills in the 5 figure range and up. The occasional 10$ from someone playing around aren't even a drop in the bucket.
So you might say, simply project the cost (with some magic) and prevent that from going over the limit. So, imagine, your app suddenly experiences a load peak and you need to scale up. However, adding a VM would increase that projection too much. Do you not scale, despite this possibily being a small load peak, and let the app go down? Or do you risk the situation described above? Doesn't matter, you'll get bad press either way.
And beyond that, you'd still need to project costs like traffic volume, which can vary extremely. Not to say anything about the technical difficulties of coordinating that billing information across hundreds of services in real time.
And even if you do all that, you still get bad press of the likes of "we forgot to remove our payment limit and it killed our app while being on the front page (and our alert did not trigger because we couldn't afford another mail)".
There's no way AWS (or any other cloud) is eating all these drawbacks just to have a limit. I bet it's orders of magnitude cheaper to just eat the occasional surprise bill.
Azure have the ability to stop everything at a specific limit - they choose not to make it available,
I’m willing to bet it’s the latter. If in a normal account, you’d then incurred $152.78 or $166.39, did the limit work? Would customers agree?
My cloud bills continue to change for several days past the end of the month (for legitimate calculations that come in for usage incurred during the month).
We couldn't technically make it stop exactly at $150.00, but only at $167.89 or whatever, so we are letting it run to $15k.
For catastrophic cases it doesn't matter. If it saves a person from an unexpected $15k bill then it works. Even for many businesses it would be ok to drop everything - I know some which can withstand being offline for a day, but not a $250k bill.
Two approaches:
A) Hard limits: freeze the services immediately if your cap is reached, ideally by giving a heads up some time beforehand with predictions, if possible; this is what many VPS providers out there do for unpaid bills and such, which makes sense
B) Courtesy: allow the services to keep working, but at a degraded performance level - that's what some of the other VPS providers out there do; for example, decrease disk performance, cap the CPU performance, limit the network speeds etc.; probably eventually also block writes, but don't delete data outright; any of the aforementioned should trigger monitoring alerts on the developers' side and Zabbix or another solution would alert them in minutes, as well as the vendor should also send e-mails about these measures either currently being put into place, or about to be put into place, so that the necessary actions can be taken
There's a difference between having the current capacity with a degraded performance during the spike and killing the entire app. You don't always need to scale up, depending on your failure modes. Having consistent service response times is overrated, as is needing to serve every single request without ever telling a small portion of your users that your service is experiencing high load - there should be solutions in place to deal with the backpressure and prevent data loss even under these circumstances anyways.
Unless you work in a Governmental organization or another critical piece of software for society, degraded performance is probably okay and no one feasibly cares or remembers even small outages - regardless of whether it's large sites, or small non profits or even side projects. Whereas if you do, then you probably have enough money to throw around for billing caps to not be relevant.
If you subscribe to those beliefs about always needing to be up and serve requests, however, then there's another option:
C) Billing alerts: something that most of the providers out there already provide in some capacity, however in fairly bad ways; if AWS can bill you for Lambda functions on a 1ms basis, then there's no excuse for not receiving billing alerts the very instant when this spike first happens: https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-changes-duration-billing-granularity-from-100ms-to-1ms/
> Doesn't matter, you'll get bad press either way.
Bad press? As opposed to what, going broke and not being able to pay your rent because of unpredictably large bills with no way to limit them, just because your side project got popular on Reddit or Hacker News?
There's a world of difference between what's needed by corporations and what's feasible for private individuals, so for as long as there's a chance of such bills, i will not use Azure, AWS, GCP or any other platform like that.
Remember: these surprise bills will only be "eaten" by the larger providers based on their own goodwill. There's not much preventing them from banning you outright.
There are other providers that are far more reasonable in that regard for my needs: https://news.ycombinator.com/item?id=28639196
The reality is kortilla is the same person that if AWS deletes all their data when they hit their "cap" to stop the billing will be on here complaining.
Payment method not go through? Hit your cap? To stop charges AWS needs to delete forever almost everything in your account. All your S3 data gone. All your backups / databases and archives gone.
Oh - you actually DON'T want them to blow up your platform? Maybe they could provide instead a billing console
https://console.aws.amazon.com/billing/home
or maybe alerts and alarms?
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
Cost or usage budgets (and many more)?
AWS Budgets (includes alarming options).
Some folks seem to have almost no clue about what AWS customers who pay the billions want. Is there ANY chance that AWS listens to its paying customers? Maybe has become successful by doing so (at the cost of total feature sprawl in my view?).
My quick trick is to login a day or two after I think things are shut down and look at projected bill and current month billing. I've left some very large instances running a time or two, easy to turn off.
And with billions of requests (PER SECOND) on the aws network, there is NO WAY they are doing real time billing. That is not happening. Look for daily aggregation and similar. Just the scale of permissioning on API calls must be insane per day. These are going to need to be doing local counters that aggregate periodically.
I do wish they'd maybe aggregate 4x per day (6 hours).
Realize that most customers are more focused on will their data be preserved.
Blowing out your entire EC2 / RDS / S3 / Glacier backup stack over a billing issue (or someone setting a cap up in the accounting department) makes no sense.
Are major customers really asking for this? Why risk it, why even build a tool that can blow out a customers setup so completely.
This is why I don't understand HN sometime saying AWS is "BS" etc. Does HN not thing AWS talks to their big customers to find out what they want?
I’m not saying it would be simple, but it wouldn’t be a bad idea to have a global monthly billing maximum where it’ll nuke the account at that number. There’s lots of new developers that are probably too scared of the free tier to use it without something like that (I have AWS experience, and I don’t use it for personal stuff specifically because that doesn’t exist. Which means I probably won’t tell my employer to use it either).
Surely it's the customer's role to decide whether or not it is a bad idea?
How 'bout you run a non-profit and have an allocation to run the services, would you prefer your nonprofit to lose the website for a couple days, or for it to go bankrupt?
What if you run a company for which online presence is a means of advertising and not the revenue-generating platform, would you prefer to run your advertising campaigns on a budget or without limits?
And so on.
I'm serious, what large businesses wants to lose EVERYTHING (all static IP's, all glacier and S3 data, all database and compute) over a billing issue.
is it?
if you gave user an ability to: scale infinitely or scale up to the $$ point, then how would anyone could be reasonably mad?
Certainly it is the customer's decision to make, and not AWS' ?
I have one that goes off monthly at about how much I expect to spend in a month.
It's like asking why the knife you're using keeps cutting you when you put your finger on the sharp end. Learn to use your tools, and they won't surprise you. Learn to track your costs, and you won't have unexplainable recurring bills.
You can even enable scans for unused respurces that generate costs..
Most cloud providers don't have this billing problem and so the analogy breaks. Its more like, why does this knife keep shooting me in the foot?
I spent hour or two trying to find anything running and didn't find a damn thing. Yet Amazon decided to charge me a buck or two every month for "storage" so I just canceled whole account. I mean, if I can't find what I am paying for when not using it, how am I supposed to understand the bill when I actually have a dozen of instances?
In both cases I learnt my lesson. I'm careful not to put my fingers where a knife can cut it, and to not put my bank details where AWS can charge them.
100 of them have provided an opinion and 99 of the opinions are that it is. Everyone in the room is bleeding.
I think it's fair to say that there's an issue with the tool, Stockholm syndrome not withstanding.
I know I’m inviting replies airing grievances with GCP, and there certainly are many, but I’ve come to really like what it offers. Especially, GKE is really cool and at least to me feels very whole-assed as far as Kubernetes offerings go. Obviously that would make sense, but still, it really is nice to use.
Same thing happened to me on AWS. I was trying to get a Windows VM in the cloud to run some CAD software. Ended up blowing through $500 credit in a week and shut it down before it went through the credit.
Unexpected bills like that makes for a terrible experience to be honest...
I then tried to close down another account based on AWS organisations, and it was super complicated to get it all removed to stop billing to the extent I would have needed to hire an experienced consultant to do it rather than just click “close account.”
Why not just use something like DigitalOcean (https://www.digitalocean.com/products/droplets/), Vultr (https://www.vultr.com/products/cloud-compute/), Hetzner (https://www.hetzner.com/cloud), Contabo (https://contabo.com/en/) or even smaller regional ones: personally i use Time4VPS in Lithuania because it's close to me (https://www.time4vps.com/?affid=5294, affiliate link to make my hosting cheaper if someone else registers)?
Providers that just give you VPSes that you can run whatever containers on (or just host things the old fashioned way) and basically use whatever open source software that your project needs. More importantly, providers that give you predictable billing at a flat figure per month (or less, if your VPS isn't active all of the month) and simply slow down your network connection if the set limits are exceeded.
If you're just doing something to practice and haven't sold out to SaaSS (https://www.gnu.org/philosophy/who-does-that-server-really-s...), you probably really don't need to worry about scaling just yet - you can migrate over to AWS, GCP, Azure, or any other of the large providers at any time, by just running your containers on their scalable infrastructure, if there's even any point in doing that, since the smaller providers also can scale similarly in most cases.
Lastly, it just feels dangerous to give AWS, GCP, Azure or any other entity that's known to give people insane bills your personal details and personal credit/debit card details - what if they decide to block you because you can't pay and your complaint doesn't become popular on Reddit or HackerNews? It almost feels like setting up a shell company and using limited virtual credit cards would be more reasonable, same as people always say that you should have a separate Google account for your personal needs and anything in any professional capacity, so nothing gets blanket banned.
Sadly, there aren't many tutorials that start with: "Here's how you set up a company that's detached from your personal details, and here's how to easily make one credit card per vendor." Honestly, even the internal workings of companies like https://privacy.com/ are unclear to me.
I thought I'd deleted the resources but months later I was getting emails about past due charges
But even with a simple webservice you quickly get to 50 - 100$ alone from testing and deployment.
I heavily recommend to rent a server for private use. They are always the cheaper option at fixed costs. Of course you don't have those fancy services... I only use a virtual server right now and pay $45 quarterly with domain. It still does have quite a lot of power though.
Personally I'm in love with Hetzner cloud. It has less "features" (=proprietary complexity) than AWS/Azure, but I can understand everything perfectly and, most importantly, I have a guarantee I won't be charged over a limit.
The VS credits are hard capped. At least they were on my account (didn't even have a credit card loaded)
I think it's because we aren't a large corporation in terms of headcount.
In any case, that meeting ensured that no further consideration of Azure would take place, and it's very unlikely that it would be considered in the future.
"Dear Google Cloud: Your Deprecation Policy is Killing You": https://steve-yegge.medium.com/dear-google-cloud-your-deprec...
That said, the k8s version deprecation policies, mixed with significant changes in cluster setup from times to times you have to keep up with is peculiar. I kind of think this is something we accept when going into k8s, but I'd understand people not at ease with that philosophy.
This thread is nightmarish for someone looking to deploy a SaaS with three 9s
As in the article, if you already run Windows servers at scale, going to AWS is possible but won't be your first choice. Microsoft sales people being condescending will basically reflect that situation.
In most other situation I can think of you'll go to Azure only if you can't use the alternatives, so again you're basically at their mercy.
To digress, if you run linux servers with an open source stack, going to Azure will bring you virtually nothing, and will probably be an expensive PITA at every step. In a previous ruby shop we had more empathic dev evangelists walk us through Azure, but we felt like wasting their time as it clearly wasn't a priority to them. Then looking at the price we'd need to drop them as soon the discount prices expire, so it was just a losing proposition for everyone involved. I totally understand how more sales focused employees would weed out shops like us and go look for those who have to stick with them anyway.
My company does that. We have started moving new things to the cloud, but we took one look at the Windows pricing in the cloud and ran away. And my company is DEEP in Windows.
So I’m not sure Azure really has much of a leg up there. I think the only real benefit they have is their hybrid cloud for people who do use Windows in the cloud as well.
So what incentive do you image they have to make those services efficient?
My favourite example is Log Analytics. It can easily cost up to 20% of the virtual machines it is monitoring! If you have a very heavily loaded website and you're logging every HTTP request, it can exceed the cost of the service it is monitoring.
They charge you a ludicrous $2,380 per terabyte of ingested data. This is 20x the cost of the underlying storage, even if it's Premium SSD! For comparison, AWS charges just $500, which is still overpriced.
Now consider: If you're an Azure software developer and you find a way to reduce the bloat in the log data stream format, what do you think the chances are of getting that approved with management?
They have a firehose spewing money in their cloud. I can't imagine them ever saying: "I think it's a good idea to turn that down to a mere trickle!"
As others have pointed out, all of their other services have similar moral hazards: Bastion, NAT Gateways, Private Endpoints, Backup, Snapshots, etc...
(there are good reasons for this, but still declared consumption is different from internal consumption)
However, the wire format is super verbose JSON.
They bill per GB of the latter, not the former.
To put things in perspective: How many $ of CPU time do you imagine it takes to column-compress 1 TB of data? I would estimate that a single modern CPU core could do this in a minute or so. Factor in various inefficiencies and make it a super generous 1 hour. At spot pricing, that's about $0.01! One cent!!!
The larger cost would be bandwidth. Azure charges a huge markup for traffic (just like AWS), so for example zone-to-zone data costs $10 per terabyte at retail pricing (not internal costing).
They store that data for 30 days "for free" (lol). Assume a worst-case compression ratio of 10:1 and then that means that they have to retain 100 GB for 30 days. That's $9.43 for a Premium SSD at retail pricing.
So their hosting costs for Log Analytics is something like $20 per TB ingested, but they charge well over $2000 for it.
That 100:1 markup is pretty sweet if your KPIs are based on recurring revenue.
There is no way in hell they will ever "optimise" this. Any accidental improvement will be rolled back or "adjusted" to ensure the revenue stream doesn't fall off a cliff.
Have you not wondered why it's taken them so long -- over ten years -- to enable any feature to filter logs at the source?
We really should revolt against this. I should be able to have a view of all of my billing without having to pay extra. It also shouldn't be hidden behind a bigquery export, it should be easy view what is being spent and what is causing it.
So I tried setting up the same ingestion database with Kafka Connect and Mongo on a $200/month VM. It worked flawlessly, and Azure helpfully suggested I downsize that VM because it was underutilized based on CPU statistics.
What incentive do the Cosmos engineers have make it more efficient, or to make the RU pricing model more reflective of actual usage? Zero. It's a money hose. Why would you turn that off?
I started by spinning up a small one in my lab but when I saw the pricing I back-pedalled very, very fast. Deleted the whole Resource Group and never looked into it again.
But also, every time I start stringing together cloud services, I experience two things: first, exploding complexity because now I'm adding points of failure, integrations, transformations, to keep it all running; and second, this sense of "the whole point of the cloud is to simplify things, to offer canned services and features that save me the trouble of doing this in code for myself." Once I'm using cloud features to work around cloud limitations, I bail out because if I'm going to spend that time (and money), I'm going to get the benefits of something much more direct.
> They charge you a ludicrous $2,380 per terabyte of ingested data. This is 20x the cost of the underlying storage, even if it's Premium SSD! For comparison, AWS charges just $500, which is still overpriced.
just makes me happy about our monitoring cluster on the german hoster Hetzner. The old systems are at 40 Euros / month for 900GB storage and the upgraded ones are 40 Euros for 1.3TB / month. There's some manpower per month in there, and some egress costs, but it's still very cheap.
As a .NET full stack web dev for like 14 years, I finally decided to put all my new learnings into just doing static sites on React and trying to figure out some AWS stuff because I wasn't going to try and use .NET on AWS when I had it so fairly figured out on Azure.
For a serial entrepreneur and maker, it just couldn't cut it anymore. Now I do NextJS on Vercel with minimal extra services out of what they provide and I get way faster stuff for free pretty much and I guess I'm no longer the only .NET guy struggling to hold the fort in a big tech community that also thinks .NET is too old or boring or non-sexy.
I still do like Azure better than AWS. The stupid, weird UX is still nicer than AWS. The docs by MS are 1000x better than everything on AWS. I miss MSSQL and the SQL Server Management Studio, but I don't miss the cost for scaling it enough to actually use it for scraping or data processing.
I tried to sell friends on Azure and even got a part time gig from MS themselves to try and help local startups use it, but no one cared or was interested. It just doesn't have that same "standard" or "sexyness" or "built into every new tech" feel to it, so I doubt it'll ever really change or pull ahead in comparison.
This is my pet peeve with cloud providers. Each one of them seem to have a gotcha somewhere hidden.
It's very hard to compare what your final costs will be early in the project. You try to compare GCloud, AWS and Azure on VM, ingress, egress. After that comparisons become harder as services don't necessarily map 1:1. You end up choosing one and always find something else that adds to the cost you forgot to include, or maybe you just underestimated some metric.
Egress feels abusive pretty much across the board. Without really a good reason. Feels like they all sat at a table and decided to fix the price there.
While you are developing your business you find you want to use some feature (like managed VMs on Azure's case) that is priced way out of a reasonable amount. You feel robbed, maybe you can still pay for it with your budget, but even then it leaves a bad taste, like you are getting a bad deal.
Indeed. You can significantly lower your price for hosting static files by using an external CDN in front of S3 or GCS.
Little known fact: egress from GCS to Cloudflare is half the price than their usual egress fees. So combining the Cloudflare CDN for caching static files with this egress discount can lead to a 3-4x saving over just serving files out of GCS directly.
And this is the actual problem. Azure doesn't compete on quality. Microsoft rarely does. They have a few products and a market position that make them the default choice for countless customers. Nobody punishes them for their failings as long as the feature boxes keep being checked, and so the cycle continues.
Teams and Azure DevOps are some of the worst software I've ever used in my life. I've used worse software before, but it was hobbyist stuff written by single developers, and therefore don't really compare fairly.
Checkout stage of our code from Azure Repos easily count for half a minute. Npm install goes for a 1.5 minute with npm cache hit. Total build times are around 20 minutes...
It took me an escalation from our account manager, to get a good support guy, who informed me of this functionality.
Here is the summary of my support ticket:
WEBSITE_RUN_FROM_PACKAGE” and value as 1.
We performed deployment and it took around 30 secs. We also verified that the app is working fine.
Now you will do some changes and deploy once more to verify end to end pipeline.
Zip deployment is a feature of Azure App Service that lets you deploy your function app project to the wwwroot directory. The project is packaged as a .zip deployment file. The same APIs can be used to deploy your package to the d:\home\data\SitePackages folder. With the WEBSITE_RUN_FROM_PACKAGE app setting value of 1, the zip deployment APIs copy your package to the d:\home\data\SitePackages folder instead of extracting the files to d:\home\site\wwwroot. It also creates the packagename.txt file. After a restart, the package is mounted to wwwroot as a read-only filesystem.
Article for reference: https://docs.microsoft.com/en-us/azure/azure-functions/run-f...
The problem with Teams is that companies use it to replace Slack, which is much more pleasant to use.
The reason is that it’s impossible with the desktop app to browse channels that you’re not a member of, whereas on mobile you can! When I mentioned it to my colleagues they were shocked. This whole time they could have been communicating in other channels and they had no idea they existed. I have no idea if this is a bug, or some kind of admin setting.
There are plenty of other issues with DevOps too. You have to buy in to the whole Microsoft package apparently, and none of the parts are best in class.
Simple things like being able to sort tables un the web UI have had open issues for years.
In our case, all we wanted was a static IP in front of an Azure Container Instance. Easy right? Let's put the container in a vNET, place a NAT Gateway in front of it and we are done. However, for some reason NAT Gateway is not supported for Container instances, instead, the official documentation suggests setting up an Azure managed firewall in front of your container that starts at a whopping 600EUR/month. That is a steep price increase from your ~30ish EUR/month for a basic container instance and it doesn't seem to be any other official alternative.
I have opened an issue with the docs team [1] about it and I hope there is another way of doing this that doesn't incur a doubling of our Azure monthly spending.
[1]: https://github.com/MicrosoftDocs/azure-docs/issues/81274
However, I believe you could have set up "public IP prefix" using azure cli. I do not think you needs a azure managed firewall.
Adding managed firewall just to have edge IP is like saying I want to add a outside patio to my house, sure let's add a security check point for the neighborhood first.
What place would you suggest? We had bad experience with Azure support we could never fight through on the first support line.
> However, I believe you could have set up "public IP prefix" using azure cli. I do not think you needs a azure managed firewall.
I don't have deep experience in networking stuff on Azure so my understanding can be wrong, but I think "public ip prefix" is just a group of continuous IP addresses what you can reserve. You still need to assign those to something eg a NAT Gateway. As far as I know you cannot assign them directly to an Container Instance.
>> I don't think azure-docs repo is the right place to ask for help/suggestions This is correct -- The azure docs repo feedback mechanism (using GitHub issues) is primarily for providing feedback on the documentation itself. We try to make this clear via the buttons on the bottom of the page; one is for 'Product Feedback' and the other is for 'Feedback about this page'. I would agree that the distinctions can be blurry, but I see the three categories as: - Product Support: I need help with a product - Product Feedback: Product A is missing feature B, and I want you to add it - Documentation Feedback: The documentation is unclear, has a typo, or the example provided no longer works
For Product support, your best bet is to go through the standard support channel. I'm sorry that you didn't get a better response when you tried contacting support. Do you have paid support? If you're a large customer, you may get a dedicated customer support account manager. Additionally, there are community forums including https://docs.microsoft.com/en-us/answers/topics/azure-contai... and https://techcommunity.microsoft.com/t5/azure-compute/bd-p/Co... , which can also be used to submit product feedback.
Other teams do get burnt out on docs though, especially when customers use them for abuse or free architecture help. My favorite was someone asking me how to use an Oracle product. I know our branding is confusing but it's not that bad... Is it?
All those features and no decent integration between them, unless you're a multi billion 100k employee company you'll have no luck with their customer support either.
Now, this employee didn't really help, but they were obviously professional and had database experience and didn't act condescending / like they were doing us a favor at all.
They just worked through the issue with me, which was a very pleasant surprise.
Oracle Cloud Infrastructure provides NAT gateways for free. You pay (low) transit costs, but unlike AWS+Azure (idk about Google) the NAT gateway itself costs nothing, so you don't pay twice for NAT traffic.
All the traffic at these cloud operations gets handled by cloud scale SDN systems. I suspect the actual cost of the few tens of bytes necessary to track a NAT connection is lost in the noise of such platforms. So to my mind the high cost of some of these cloud operator's NAT gateways seems abusive.
Fortunately there is indeed competition that accommodates my view.
Edit: oh, no, you just need a public ip prefix/address, right?
- x86 just has better support for the stuff Azure's "enterprise" customers want
- ARM servers are often more expensive to spin up than an equivalently specced x86 option
- PRISM compliance is easier on x86 (half-joking, half not)
I like ARM, and I owned a Rev1 Raspberry Pi when those were cool. But even now, ARM still has yet to make a strong case for existing on the server. And that's before we even discuss architectures like RISC-V that are out on the horizon, much better suited for servers than ARM. I'm not planning on an "ARM revolution" taking place in the next decade unless x86 is critically compromised in some way.
This is several years out of date: AWS Graviton instances are usually a fairly substantial savings over similar Intel, with AMD in between, and Cloudflare has been reporting rather good numbers as well:
https://blog.cloudflare.com/designing-edge-servers-with-arm-...
The main reason I suspect Azure doesn't have it is both Windows' legacy x86 hyper-focus (the days where NT ran on half a dozen platforms never really panned out) and a smaller number of managed services. AWS has very popular services like RDS, ElastiCache, ElasticSearch/OpenSearch, etc. where you can simply check a box and wait a couple minutes to see savings, not to mention things like Lambda being only slightly more work for many users, and that's a great way to get volume usage even if the average enterprise IT department is scared to go near it for VMs.
There have been multiple consumer ARM Windows products on the market for the last 9 years, I'd say Windows (client, at least) on ARM64 is as solid as its x64 counterpart
Have you really had the experience that a large organization has the ability to recompile everything they run? Most will have a lot of code which is provided as binaries by a vendor, and their in-house code almost certainly has dependencies and optimizations which will need to be dealt with & revalidated. No, none of that is unsolvable but it means adoption is much harder than, say, changing an RDS instance type and you'd be taking on all of the support rather than the cloud provider's much larger team.
That's what I referred to in my original comment — in my experience, the average Azure user works at a Windows-heavy enterprise IT shop where those issues would be common. That doesn't mean that I don't expect ARM servers to happen there — Microsoft announced it was coming years ago, after all — but that it's going to be slow since the upfront investment will likely have slower adoption.
About the ability to recompile, if you can't build ARM software what's the point of using ARM instances? You don't need cross-architecture compatibility like you need it on the desktop version, which is what consumer complain about when talking about Windows on ARM
I think AWS has successfully been pushing this because they know they’ll see that initial volume from people seeking savings on their own managed services and things like Lambda which are easy to switch, and that will fuel interest in switching other services which require more work.
Every comparison I've seen was cost saving for the datacenter.
B2C on the other hand is a different story. Every few months we have to roll out a new tenant in our system. Tenants are identified by B2C "applications". Every single time, the new application doesn't work. Every single time the fix involves editing the JSON spec, changing something random (like a "true" to a "false"), saving, and changing it right back again.
Doesn't exactly inspire confidence... we're planning a migration to AWS.
Also, try Googling for documentation related to "Microsoft Azure AD B2C". Almost every shred of internet wisdom is related to AD and not B2C. Even with Microsoft's own documentation you sometimes follow a link from a B2C API reference and find yourself in AD-only land and it isn't obvious. This makes the task of researching features and debugging infuriating.
Go read through some issues, look at some closed ones, try and skim through the source code. Realize there's two enormous python scripts in the repo, one with "2.0" tacked on the end.
If Azure is somehow not just rebooting/killing VMs that lack the magic handshake, I'd highly recommend dropping the agent.
After all of this news, and what is on display in walinuxagent, do you really want some network-connected agent listening, who's often-most-touted feature is being a persistent backdoor?
"It's tough being an Azure fan." Eh, it's a nice cyclic problem. They have no depth of caring about engineering (hence why Azure is littered with services that are impossible to fully utilize because their own engineers don't understand the how/why of what they're building half the time). Which in turn, along with crap career advancement and constant un-appreciated, unmitigated live-service burnout, is why they can't retain actual Linux talent to save their fucking lives.
Most of the time Azure feels like just another 'we have virtual machines and a crappy API' service. Kinda like a pretend-cloud where a lot of products come almost together but never finish. Almost similar to the way Windows and backwards compatibility means you end up with 100 libraries, frameworks, languages and versions all sitting side-by-side and not really working together, just differently in parallel. Not useful for automation at all, which makes it not useful at scale. (Except when scaling means: we want to run windows VMs with AD and go from 10 to 100 and change no parameters at all)
It went something like this:
Code used to be running on a processor, then it ran on an OS on a processor. Then it became on a runtime on an OS on a processor. Then it an abstraction layer was sandwiched in between. Then a filesystem. Then a compatibility layer. Then a database. Then a browser. Then we went and created a runtime in the browser to run code again. We have come full circle.
This probably applies to SharePoint but also the real-time OS on your GPU, the MSSQL database which has its own OS facilities you can run stuff on. Add too many features and the application becomes the very thing we wanted to get away from...
It’s like the business equivalent of the “once you understand a monad you can no longer explain it” meme.
You sell to the managers and equip them with hollow buzzwords because you know they're gonna override the engineers on every decision anyway. By then, MS has your firm's money and all you can do is deal with it.
Used to be a function doc would tell you:
1. All the parameters and their types 2. What they did, explained 3. All possible exceptions raised by the function 4. All possible return values 5. Supplementary documentation on the object or structs passed in or passed back 6. Some examples of it in use
Now, the current docs sometimes have examples for Azure CLI/SDK stuff, but there is one convention that drives me bonkers that is as bad now as it ever was.
For examples, often times you'll get the most important part replaced with a [insert your thing here]. The format of the thing you fill in there is often left as an exercise to the reader to intuit or guess.
I find the Azure docs to be decent. They are consistent across services for the most part, so you learn how to work with them after a bit of experience on the platform.
- Customer asks for something trivial to be deployed. I say, "no problem" and start beavering away on a Bicep template or whatever to deploy their stuff.
- I hit some small but showstopper issue with a service that I expected to work, but it doesn't. I open a support ticket.
- Inevitably, it turns out to be some stupid, stupid limitation caused by unfathomable laziness of the Azure developers. No workaround, no mitigation, but we're "working on it" with no ETA offered.
- Literally years later a trivial fix for the glaring issue goes into "PREVIEW" for 9 months, long after the original project was closed. I no longer care...
Networking is especially bad, with endless limitations that make no sense, like:
IPv6 is incompatible with everything. Turning it on for anything anywhere in a vNET will permanently block unrelated features like IPv4 NAT.
But of course, you can't "protocol translate" from IPv6 on the outside to IPv4 on the inside, so you end up painted into a corner.
No bring-your-own-subnet, which means many lift & shift scenarios are impossible (we have customers using a class B pubic range internally).
Azure forces NAT on IPv6, which makes no sense at all.
All Azure PaaS services have firewalls that are IPv4 only.
The built-in firewalls (e.g.: Azure SQL Database) do not support service tags, only CIDRs.
I could go on and on...
You can't move a VM to a different subscription without also moving the entire vNet along with it. (I bet this worked great in the developer's lab with one test VM.)
You can't move a VM to a different backup resource without deleting all backups, going back years.
VMs can't be rolled back to VM snapshots.
ExpressRoute bandwidth can be increased non-disruptively, but the only way to decrease it is to recreate it -- incurring a ~30 minute outage.
The Activity Log (and most other audit logs) don't log the identity of the administrator that triggered the action about 10-50% of the time, depending on what kind of activities are going on. This makes it 100% useless as an audit log. There is no other way to obtain audits.
Most network resources are Zone Redundant, except for NAT Gateway, which is now required in some scenarios. It's Zonal only, which means you need 3 subnets, one for each zone.
Upgrading an internal load balancer from Basic to Standard cuts off Internet access, for "reasons". I'm told these are "security reasons". Uh-huh...
App Gateway doesn't use a "user-agent" header for its monitors, which makes it incompatible with a surprisingly wide range of CotS software. This includes a bunch of Microsoft Software.
I could go on and on...
This is all microsoft products though. There have been .NET Identity bugs that have been open and acknowledged for multiple years.
AKS suffers, AFAICT, constant API server outages. We tried to escalate into a ticket, but we just get motte & bailey'd between "you're putting too much load on the API server" — okay, what load? how can I see that, control it? — "here is the top consumers" — they're all AKS itself? — "well, there's too much load on the API server" gah! (Yes, we pay for the "SLA".)
You can't add IPv6 anywhere in a vnet, it will break unrelated things. We tried to add a managed PSQL server on IPv4 (b/c IPv6 is not supported): it "failed" (the API call to create it timed out with an internal server error … after 2 hours or so!) because something unrelated in the vnet used IPv6.
ACR has a 20 TiB limit, no way to prune containers, we've had to work around IDK how many 500s, the API is slow as dirt (response bandwidths of ~50 kbps — bits — and their team does not think that's a problem. It can take 10 minutes to enumerate a few megabytes of metadata…) Undelete-able manifests that I guess we will just pay for indefinitely? I feel like I could build ACR on top of Azure Blobstorage and it would be more reliable with better performance…
VMs shipping with buggy kernels. (Support wanted to know what weird thing we were running to hit kernel bugs. "Docker"?) Global outages. Known outages often don't get mentioned on the status page intentionally. I still don't know what the difference between an availability set and a VMSS is.
Everything is in preview. Everything.
Audit logs fail to load some times. Beyond complicated interactions/differences between "service principals", "applications", "enterprise applications". 2FA app now requires authenticating twice, per login, because each tenant acts as a separate yet not separate login. So. much. auth. Role assignments that don't know what principal is being granted permission, b/c ARM & AAD don't do referential integrity.
Docs that are outdated. Requests for updated docs closed without update, because "we don't have the data <from some other internal team, I think, but so?>"?. Docs that describe API calls badly. "foo: the foo query parameter" No docs, at least, that I'm aware of, about what permissions does the API call require. The docs conflate "permission" and "role" (different in Azure) all. the. time. Azure doesn't know what permissions some calls require, and simply says "give it Contributor" (close to all permissions)… and yeah that works but I want to show auditors we're doing PoLP?
Support… The SLA often isn't met, we're assigned reps in China (n.b., this isn't a language barrier problem, it's, how is someone who is literally sleeping during my business hours because that's timezones for you supposed to even meet a support SLA that wants a 8/4/2 hour response? And AFAICT from response times, they're not a night shift…?), the first response is worthless (doesn't answer the inquiry, requests information present in the original request, often isn't technically proficient, etc.), the writing is broken and sloppy. Half the time a simple "reread what you're about to send. Does it solve their problem?"… we literally got a blank email back. We've had tickets where the first response we get is "we haven't heard back from you" because sometimes their responses fail to get linked into the ticket in the portal. They lack any formal bug reporting mechanism, and support is not equipped to handle bugs.
My God, it's full of bugs.
From a business perspective, this is brilliant. From a technical perspective it's not awesome, obviously.
Our current project uses what I feel like are pretty standard features for a SaaS app. They include C#/.NET Core, Linux App Services, SQL Server, FrontDoor, SignalR, Functions, etc. Functions is the only feature that has been bumpy for us deployment-wise.
The journey of the various portals has been fun. The current one isn't perfect but it's far better than my experience using AWS. It's got to be a challenge to organize such a massive portal developed by so many teams.
The effort MS has put into documentation has been really great as well.
That said, there's a lot that could be better. A lot of the PMs are on Twitter I tend to be a squeaky wheel there about various problems so hopefully they're listening.
Haven't had issues with Azure functions, though DI is wonky in them if you try to build anything complex (imo you shouldn't). Other than that, I guess I just don't use them often to really understand what all the boilerplate does. Finally, when we switched to Python from .NET, it didn't feel like any of the function knowledge carried over somehow. Felt like developing on a different platfom.
I also HATED the Azure certification exam compared to GCP. GCP tried to actually teach you something practical, Azure tested a bunch of memorization that you can google.
Nothing in the documentation to mention this, you just need to deploy and learn from your mistakes!
The whole story regarding organizing the code, breaking changes between versions, confusing plans, different styles of configuration between that and aspnet web apis, trying to configure individual functions in the same app without affecting others, etc. is not good!
One thing I do find is that although their documentation is now open-source, it can still take a long time to get changes reviewed and merged (if at all). I waited about 3 months once for a simple change to documentation that was clearly wrong and by the time they looked at it, the docs had been re-factored.
They need to learn to embrace the Amazon Turk and have the right people review documentation edits. Should be able to quickly check a proposed change and then just merge it.
I recently moved off Amazon Cloud storage to OneDrive because Amazon didn't support rclone. Microsoft's OneDrive is quite a bit less expensive than either Google or Dropbox or Amazon.
The storage is there but managing it - what a mess it is!
First, there are like a gazillion URLs to access:
azure.com onedrive.live.com admin.microsoft.com microsoftonline.com office.com office365.com onedrive.com onmicrosoft.com sharepoint.com windowsazure.com
Each one of those portals take you to SOME view of your account with a gazillion settings. Many of them are repeated, and changing it on one portal doesn't necessarily reflect in others. For example, I enabled 2FA (I seriously don't know how or where - but I was able to login using the 2FA), but going to admin.microsoft.com showed 2FA disabled for the user - go figure!
Something as simple as figuring out how much space is currently used on your OneDrive is a challenge. There's a set ritualistic series of incantations and clicks that will get you there, but you really need to be persistent. Googling for it gives you an answer, but most answers will lead you to live.com which is only for personal accounts and not business accounts - it won't allow you to login.
office.com/launch/forms takes you to, and allows login to your business account, but going to forms.office.com (which is the top search result in google) redirects you to live.com and doesn't allow login to your business account!
One major difference perhaps is on the expenses, but are you sure you are hiring the right number of engineers and paying the right amount of $$ for those software and machines?
To provide some context: I figured eventually the configuration as code will reach the point that essentially people start to implement their own DSLs. When new people join they are easentially config boys/girls who needs to spend a lot of time to learn and unlearn a few DSLs created by the smartest guy/gal in the company who moved on to greemer pasteries...
"fan" is just in the title.
That was 4 years ago.
Last place we did pretty similar stuff with 3 data centers, all self hosted, self managed, mostly OSS stuff with about 6 sysadmins and way less hassle.
Agreed. The apps and services I manage aren't very big compared to a lot of posters here, and my company is certainly not going to ever pay the big bucks for real talent.
So instead, I have to design everything knowing that my ops team is going to be mostly $100K / year "devops" guys with a few "cloud" certs but no real CS, dev, or even Linux knowledge (yes it's that hard to hire good people now (at the low rates my employer wants...)).
I've gotten to the point where I absolutely mandate that they don't try to use Terraform or CloudFormation scripts, because in the end, there are so many edge cases or glitches that it's easier to just write an install guide that shows them which buttons to click in the AWS or Azure console. <sigh>
And when I look at the costs we spend per month - including all the unanticipated charges like NAT Gateways and $20 / day "managed" Postgres instances, I assume we'd be better off dumping the cloud and reverting to our 2008 setup: Spending $10K on some Dell servers in a managed data center and hiring an old-school Linux admin to manually install and manage it all.
I would see that as a "Major" bug because it is so unecessary and been there so long. MS's attitude? Bit too hard to fix, just use dotnet core instead, like we can all just do that with our legacy production apps!
VS is also pretty cool function-wise but still too many lockups, cache corruption causing strange compiler errors and files left locked after exiting but instead of doubling-down and refactoring the core code to work properly (I think that is a "thing"), instead they kind of start using VSCode instead even though it doesn't have half the functionality of VS but don't make VS become the thing of beauty it is supposed to be.
It took quite literally months to get the issue escalated to someone who could do anything about it, and as far as I know it’s still broken, although it’s scheduled to be fixed this year. We had to pay for this “support” too. We worked around the problem in the end, but still.
We once had an error with Azure MySQL and AKS, where it would just stop dropping packets with basic instance types. Support could never fix it, we ended up just upgrading to standard because that worked.
I will say this - the manager of the team at Azure did comp our upgrade, because they couldn’t figure out why it was happening. I tried very hard to get our project off Azure, but because it was negotiated as part the Enterprise Agreement with Microsoft, it was “free” so the CTO wouldn’t budge. I assume this is how many people end up needing to deal with Azure.
I never understand it when I see people say this about Azure. Did you ever try creating a support ticket through the Azure portal? My experience has been nothing but stellar. They respond very fast, they provide very in-depth expert knowledge, and they seem to be quite thorough, escalating any issues if needed. This is my experience with creating tickets for various organizations, both big and small.
It's quite likely they had insider knowledge and therefore had a better idea of where to look. But I'm not sure that makes it "suspicious", as opposed to... well, what else would you expect in those circumstances? Or are you saying that the unfair advantage of insider knowledge is what makes it suspicious?
Anything I need to do, that I haven’t done before (or recently) requires roughly the same spin up time to get familiar with how each provider functions anyway.
Do you mean like cloning VM A destroys it and creates identical VMs B and C?
Not that that is an excuse for azure of course (or conversely the fact azure does it isn't an excuse for aws).
Everything else could have been written about AWS as well. They all feel duct taped when you know what's going on behind the curtain. Enterprise developers spend a crapload of time messing with infra on AWS. I think it all sucks. Unfortunately its better than nagging the BOFH for a few VMs (which you are only going to get this month if he knows you AND likes you, AND you can keep info-sec off everyone's back).
I wish the people in charge of these docs cared a bit more about their quality...
Unnecessarily complex, arcane and frustrating is how I’d describe basically everything we had to do.
The fact that they ship a whole Python runtime is basically the least shocking thing I’ve learnt about Azure.
GCP's console is just stupid though. It takes so long to load, and instead of just making it faster, they made it load in incrementally, so you just have to sit there waiting for whatever widget you actually want to use, load in correctly.
And even when it has loaded, the entire thing is extremely laggy.
They practically just copied some Azure CI stuff over (which was obviously written from a completely different perspective/mindset) using a tech stack that is old yet immature when compared to even relatively young products like GitLab CI.
If a company wants to come up with a CI product but can't even get on par with basic nominal GitLab CI features and usability, what's the point...
Why does it require so many steps? Why are they so verbose? It feels like writing glorified bash scripts (in which case why am I using a CI tool?) but worse.
simply mindboggling
The most egregious thing (for now) was an Auto Scale failure. For an entire business day the autoscaler failed to spin up VMs which effectively killed our product for the day. We could not scale manually either. No logs, no errors, absolutely no idea what happened. Support tried to convice us for an entire month that it was our own fault for misconfiguring the autoscaler by quoting the documentation that was saying another entirely different thing. After insisting for a month the issue was finally escalated to someone who could read the logs and immediately see what went wrong: they had run out of VMs and could not scale up.
So we have :
1. Unreliable services, breach of SLA.
2. Zero ability to anticipate, prevent, debug, or ensure the problem will not happen again.
3. Incompetent support lacking basic reading abilities trying to gaslight me.
4. An obvious issue that should have raised alarms long before it arose.
Thanks Azure.
This week I spend half a day trying to understand why Azure would not create an Application Gateway using a configuration that was identical to another one we had already (do _not_ attempt Azure without Terraform). Turns out it was another outage with absolutely no way to know it. Best part is the official fix: "Please, retry in case of failure during the deployment." It takes 30 minutes to create said resource before it enters a failed state. XKCD 303 applies.
When I finally got my data back, I moved to DigitalOcean and haven't looked back.
Also, lots of stuff doesn't worth with GCC High
2020: Move from cloud hosting to save yourself from surprise billing hassles.
[*] In early 2010s, WhatsApp could handle 2M connections on a single 1U with Erlang; MigratoryData could handle 10-12M connections with Java on Linux. Epyc servers are even more powerful now.
This is my feeling. And I think it makes sense - vast swathes of how Google and Amazon work already rely on cloudy engineers (although they were there before cloud was a thing). Microsoft has much more split focus, with Windows, SQL Server, AD, Dynamics, Xbox, and Office all being large areas of engineering specialism that in no way feed into cloudy areas.
Bing and Xbox Live do, I suppose, but I bet they're built in a totally different way to both each other and, say, Teams, which I think is k8s in the back.
So my theory is there's much less of the company that does cloudy things, and even when they do (e.g. Bing, Xbox Live), there's probably a lack of a commonality of technical approach compared to what Azure exposes to customers.
I do appreciate the detailed list of current failings, to add to my own systemic rejection of this system based on their management objectives
you are under an illusion, you are REQUIRED to use it in service of The Company
Can we just have a decent cloud provider offering services on a reliability and scale level as AWS/GCE/Azure, but with decent customer support and no way for (necessary) anti-fraud/abuse mechanisms to kill off entire digital lives?
Azure Support is aware of the issue, it happens to many of their clients, but they have no idea why it is happening.
Hilarious. But still managememt was sold a vision of the #1 Cloud, so here we are writing more and more bandaids on issues Azure has.
Another one from the top of my head is Scheduled Events for AKS nodes. Azure provides you with a managed nodegroups solution, but has no project to support those events and you have to rely on not-supported community projects. Which often die simply because of lack of interest in Azure.
They're the type of mistakes where it looks like there's no process or culture in place. Not a good thing for a cloud provider.
And all of that without a changelog... I just run a change detector on a bunch of Microsoft docs urls.