Hacker News Reader

Comment by amarant | Hacker News Reader

Story time!

A couple of years back, I was working at Mojang (makers of Minecraft).

We got purchased by Microsoft, which of course meant we had to at least try to migrate away from AWS to Azure. On the surface, it made sense: our AWS bill was pretty steep, iirc into the 6 figures monthly, we could have Azure for free*.

Fast forward about a year, and an uncountable amount of hours spent by both my team, and Azure solutions specialists, kindly lent to us by the Azure org itself, we all agreed the six figure bill to one of corporate daddy's largest competitors would have to stay!

I've written off Azure as a viable cloud provider since then. I've always thought I would have to revaluate that stance sooner or later. Wouldn't be the first time I was wrong!

View on HN

When I worked at Jet, a shopping website trying to compete with Amazon, we obviously did not want to give money to Amazon, so we used Azure.

For the most part it was just fine, until we started using CosmosDB (then called DocumentDB).

DocumentDB, in its first incarnation, was utterly terrible. The pricing was extremely hard to predict, so we would end up with ridiculous bills at the end of the week, the provided .NET SDK for it was buggy and horrible, but the very worst part was the WebUI appeared to be directly tied to your particular instance of CosmosDB.

Why is this bad? Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! We got into situations where we had to turn off the entire application, just to bump up the resources for Cosmos. It felt like it was a complete amateur hour from Microsoft.

My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth. If I end up getting some free credits or something, maybe I'll give Azure another go but I would definitely not recommend it right now.

> Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources!

What the ever loving heck... seriously?! Why wouldn't this be a control plane API that reconfigures a data plane?!

I think it was querying the database to get usage information, and it was blocking the loading of the page as a result. It's been a few years, but if I recall correctly, the .NET API for changing the database provisioning did work, so some people would hack up a quick script to change resources in an emergency.

I think they did fix it eventually, because I know that multiple people on my team complained to Microsoft about it. Very short-sighted decision on their end, but to be fair it was a brand new product.

This is how hosted Elasticsearch works today, too. Really mess up your ES? Good luck with the web UI!

> What the ever loving heck... seriously?!

What's so surprising about that? If your CRUD operations take longer to do, and you're doing those to drive a GUI, of course the GUI will lumber along.

It's good practice to separate out your control plane and data plane, so in this kind of scenario you can use the control plane freely to manage and scale up the data plane without worrying about the data plane underresourcing affecting your control plane operations.

The reverse also applies; by separating them you can have issues with your control plane but not have the database go down.

Control plane and data planes should be running on different hardware.

It shouldn't lag the web interface. There are common ways to handle this, such as making all resource mutations asynchronous, or if you want to keep it synchronous, ensuring that control processes get guaranteed CPU resources. It is not a particularly challenging thing to do.

Because that would be the smart and sensible way to design a service like that, and those words are unequivocally not associated with Azure in my experience.

> My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth.

A couple of years ago I stumbled upon a Azure project which started off using the old timey Cosmos DB. Looking at the repository history from those days, I saw a bunch of Entity Framework configurations and navigations and arcane wizardry that would take an engineer months to really figure out.

Then there was an update to CosmosSDK, and all that EF noise was replaced by single CRUD operations that took the unserialized object, id and partition key as input. That's it.

Worlds of difference, and a simple key-value query takes ~10ms to do.

Yes, it's worlds of difference.

> Worlds of difference, and a simple key-value query takes ~10ms to do.

Unless that query goes over the Internet to another continent, that's a really long time isn't it?

Yes it’s ages but not multi continent long. Us east 1 to us east 2 is about 10-15ms in my experience.

You're right, I exaggerated unnecessarily there. Sorry

No reason to apologise - I assumed and tried to give a better reference.

It is an absolute eternity though. A KV lookup is fractions of a microsecond in a managed language like c#. A http request is in the microseconds range on localhost and a smidge more on a performant local network. A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network.

It’s an absolute eternity, and when thinking about this stuff and scaling, think “is it worth slowing down the normal case by 1000x to introduce an external service”

Some numbers no one asked for :)

ConcurrentDictionary<K, V> read latency is going to be around 7-15ns for the data hot in the cache, scaling with key length if it is string-based, and anywhere between 75ns and 150ns for reading that out of RAM. Alternate implementations like NonBlocking.ConcurrentDictionary can reach down to 3.5-5ns for integer-based keys given all data is in L1 and the branch predictor is fully primed, on modern x86_64 cores.

I've gotten in arguments over overuse of Redis for this very reason.

Redis is cool but people use it too willy nilly. It's expensive to run, and you have to pay a cost of (de)serialization, and network latency. Sometimes that's necessary, especially if you need to share stuff across multiple nodes and you're afraid of hammering the database too much, but a lot of the time, I'd say even most of the time, you can get away with just a big ol' global ConcurrentDictionary (or whatever the equivalent is in your language of choice).

Basically any thread-safe dictionary is fine, because no matter what kind of locking strategy they're using, it will almost certainly be faster than anything that has to hit the network, by several orders of magnitude, and you have one less thing to manage in your application. You can figure out the best way to invalidate old entries, or find a library to do it (e.g. something like a Guava cache), and you'll likely much better performance than arbitrarily farming to Redis.

> (...) but a lot of the time, I'd say even most of the time, you can get away with just a big ol' global ConcurrentDictionary (or whatever the equivalent is in your language of choice).

That's fine if all you're doing is caching limited data independently on each node and you have no requirement for consistency or even durability.

Most of the usecases you stumble upon do not fit that scenario. That's why "a lot of the time" suggesting a dictionary is plain wrong.

> That's fine if all you're doing is caching limited data independently on each node and you have no requirement for consistency or even durability.

You've assumed you have multiple nodes here. Our point is that you can scale _significantly_ more with a ConcurrentDictionary (or equivalent) than you think - far more than a single node backed by redis will go. You have a point about durability, but in my experience, most apps that lean on Redis don't actually solve that problem. If the app goes down mid request, it will leave broken state behind. The data itself in redis might be valid, but the app can send partial data. The solution to this is the same as handling durability in a non-redis case too.

Also, runnign Redis on the same machine for persistance is an option for durability.

> Most of the usecases you stumble upon do not fit that scenario. That's why "a lot of the time" suggesting a dictionary is plain wrong.

I disagree. Most of the usecases _do_ fit this scenario, and Redis is (over) used as a tool to allow for horizontal scaling caused by an unwillingness to scale vertically. At the level of Meta/X yes you absolutely need it. But for a web app with about 10k concurrent users, you can handle it on a single node.

> A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network.

I think you need to revise your assumptions. Even optimized DynamoDB setups shows average response times in the 15ms range and P99s at the 20ms range.

One of the mistakes you seem to be making is assuming that to do a query on a globally distributed database with multiple partitions, you'd only need to hit one single box and no data exchange would take place. That bears no resemblance with reality.

> Unless that query goes over the Internet to another continent, that's a really long time isn't it?

If you're hosting a service in a cloud provider and you implement your services so that your call to the cloud provider's database goes over the internet to another continent, you have serious problems but none of them are caused by the cloud provider.

Also, CosmosDB is globally distributed.

They meant 10ms is slow.

> They meant 10ms is slow.

I don't think anyone making that sort of claim knows anything about cloud services. A single roundtrip of a no-op request within the same data center takes 0.5ms. Add querying across multiple partitions and data seeking, and you don't find cloud providers doing better.

To frame how oblivious that claim is, DynamoDB is lauded for it's response times being sub-20ms.

https://stackoverflow.com/questions/34552625/how-to-get-sub-...

No that's exactly what I mean: I expect the round-trip within a datacenter to take roughly 0.5ms, I expect the key lookup to take about that time or less, so 10ms is roughly an order of magnitude more than I would expect a "simple key-value query" to take

For comparison if you run your own hardware and do a memcached KV lookup with a different server on the same rack, p99 times are slightly under 1ms. Given the guarantees of cosmosdb ~10ms isn't that bad for a p100

A team in my org worked with Jet for 2+ years to help y’all scale.

It was interesting seeing the biweekly status updates, they basically all started with “This is how Jet.com broke Azure core services this week”.

As much as it sucks, this was a deliberate strategy all the way from Satya - every employee knew Azure was a joke, but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.

> but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.

I don't get it. There's lots of distributed systems theory that could provide a more robust, analytical approach to a scalable architecture. If a system is regularly breaking like this, it sounds like it should be a "back to the drawing board" moment.

Back to the drawing board risks delaying your products with several years. Their strategy was probably the right one.

It sounds like their product has been considered crap for years, so they've had the time.

“We’ve had years to fix it, all the experience and knowledge we’d need, but we’ve tried basically none of it and we’re at a loss as to why our product is still garbage! It’s truly a mystery” - Azure leadership I assume.

My experience working on Google Cloud Platform is similar. Even when the solution is relatively technically simple, there are many organizational impediments and politics because the solution likely involves multiple fiefdoms.

The only thing that seems to get the fiefdoms to work on an issue is if a big enough customer calls enough of an attention to the issue so that no leader/fiefdom wants to be blamed in such a "high profile" issue that everyone got their work in gear for. This leads to organizational dysfunction since now everyone understands that there's no point in trying to fix anything until there is enough attention on it, and that is mainly achieved when a big customer complains.

I've even had projects that took months of engineering work that would resolve many user complaints about a lack of really basic functionality, and different elements of leadership would block the launch. I can only guess that some fiefdom did not like change and would not state their objection publicly, so I only got stonewalled, and I never got an explanation to throw away months of work. I am as certain as I can be that if a big customer complained loudly enough about the problem that such basic functionality was missing, the organization would demand that the fix be launched.

You would assume that the UI process would be running on Very High priority.

> My understanding is that Cosmos has gotten a lot better

For some values of "better", I guess. Performance is still terrible, their data visualization/inspection tools are shameful, their SQL dialect is finicky and has no error reporting beyond "something is wrong with the input value", and their official Python SDK has a race condition bug that can silently clear out your documents when under heavy load.

I used to work at a Cosmos-heavy house and I would utter "fucking Cosmos" around 15 times a day.

Your story reminds me of when Microsoft acquired Hotmail in the '90s and they tried migrating from FreeBSD & Solaris onto Windows NT/IIS. Having the world's largest email service running on the Windows stack would have been a huge endorsement. It took years until they were successful.

https://www.zdnet.com/article/ms-moving-hotmail-to-win2000-s...

https://jimbojones.livejournal.com/23143.html

Ha, I worked on that project. That drove a lot of good requirements into Windows that set us up for web based services (eventually)

> Windows that set us up for web based services (eventually)

...and then .NET and SQL Server started shipping for Linux.

There are 2 decades between those 2 points, .NET was -4 y.o. at the first one.

SQL Server is really Sybase tho, which was always capable of running on UNIX.

Can't say much more, but I worked on a huge (internal) Sybase ASE on Linux based app (you've _all_ bought products administered on this app ;) ) way back (yes, pre-SSD, multi path fiber I/O to get things fast, failover etc.) and T-SQL is really nice, as is/was ASE and the replication server. Been about 20 years tho, so who knows.

I used to work at a Sybase shop in the late 90's. It was way nicer to work with than Oracle!

SQL Server uses NT and Win32 APIs, so the SQL team built a platform independent layer. Meaning NT and Win32 is still used by SQL on Linux. It’s pretty cool tech.

https://www.microsoft.com/en-us/sql-server/blog/2016/12/16/s...

I worked with SQL Server a bit, writing a Rust client for it back in the days. The manual is really good, explaining the protocol clearly. That made it really easy to write a client for it.

Can't say the same for Oracle...

Only if you are speaking about SQL Server 7 and earlier, meaning around 2002.

Microsoft SQL Server has long stop being Sybase SQL Server, and works on Linux by making use of Drawbridge.

https://www.microsoft.com/en-us/sql-server/blog/2016/12/16/s...

Of course, that's fair.

I guess wording wise for my comment, the "Hah, they didn't actually write that themselves, they just bought the Sybase rights to everything license" got the better of me :)

To be fair again, from what I hear, SQL server at Microsoft did some nice things on top of Sybase but the base, T-SQL, is just nice overall and by itself. I really want to like Postgres (and I do) but some of the awesome things I had with actual Sybase ASE 20+ years ago, Postgres still does not have. And that was a piece of software that had those features I loved for 10+ years prior to when I started working with it. The app we're talking of here was 15 years old when I worked on it 20+ years ago and it's probably still around and very probably still uses Sybase ASE (tho the actual app was converted from Smalltalk to Java ;)

I also later on had to use Oracle and had the same "WTF? You can't do that?" experience :shrug:

Are you free to expand upon your role and perhaps some of the actual tech/fixes that made it back into Windows?

Like IIS running some part of the code in the kernel? ( http.sys ) :x

It has its advantages… but wasn’t done until Svr 2003.

https://learn.microsoft.com/en-us/iis/get-started/introducti...

> It has its advantages

Yeah, the advantages (RCE) were copied by modern web browsers. /s

Seriously, Windows 2000 was one of the most stable OS back in the day, rock solid. I used 2000 server as a desktop OS, instead of 98.

unlike shit show that was windows 95/98/ME

Windows 2000 server was peak windows. All the subsequent versions just got harder to maintain as they gradually ruined the user interface. Nobody cares about the UI on consumer windows but if you’re spending a lot of time in RDP the vista based server products are terrible.

I don’t hate windows 2019 but Linux is better, easier, faster and a relief after any futile attempts to use IIS or sql server in 2025.

> vista based server products are terrible.

The first generation of tabletised 8/Metro interfaces made me audibly groan every time I had to RDP into machines running 2012.

Oh yes. I still have a client that has 2012 and it physically hurts to use

The stuttering over RDP when the start menu animation tried to slide in the tiles was amazing.

windows xp x64 edition was pretty slick; and so was NT4. I agree that 2000 was pretty cool, but perhaps a lot of that is design nostalgia. It was very "serious business OS" where XP and Me looked like jellybeans and cartoons. My favorite windows, though, is win 7 ultimate, Steve Ballmer Edition. i was sad when i had to upgrade to winten.

ninja proof: https://i.imgur.com/l29rDVo.jpeg

I get the nostalgia for XP, it was the first windows consumer edition that didn’t suck, but for a server OS 2000 was so lean and easy to manage it makes me wonder how MS lost to Linux. Back then, it was a genuine competition, now you’d have to be crazy to choose windows to deploy anything.

i never used windows XP, i went from 2000 pro to XP x64 edition, which came out 2 years after XP did.

Maybe, but Win 3.1 was good for me.

Windows Server still has it's place. AD DS, file services, and SQL Server being the big ones. Linux doesn't have apps that do these things 'better'.

I wish MSFT could build Active Directory and the associated constellation of services on Linux. You can make a reasonable simulacrum with Samba but it isn't as well-integrated.

(My fever dream wish is for a "distribution" of NT that boots in text mode and has an updated Interix subsystem alongside Win32. Throw in ZFS and it would be awesome.)

An NT that boots into text mode wouldn't be terribly useful for software designed for NT today given the high dependency on UI libraries.

I too wish for an NT that was CLI-only, striped of services as much as possible.

     Starting Windows NT...
     
     C:\>

It's too bad Microsoft has no interest as a business in on-prem software.

Server Core is close to what you're talking about re: CLI only.

I agree re: MSFT having no interest in on-prem software. It saddens me.

Core isn't close. It's still a graphical interface.

I guess I misread you, then. I thought you were arguing that a text mode wouldn't be useful. That's why I suggested Server Core. It's CLI, but uselessly framed in a GUI framework.

Like I said in my earlier post, text mode NT is my fever dream fantasy. Maybe you were saying the same thing.

Yes, I was saying the same thing. There are very few applications we could run on a text-mode DOS/Linux-CLI type NT due to the upstream dependencies that require (or implement) a graphical interface.

Text-only mode would be wonderful even if all you could do is look at a blinking cursor.

After SP2 the worst wrinkles are taken care of. Oh, and skip ever second OS release, of course.

I'm not as much against windows as I uses to be but I'm not budging off Ubuntu LTS even though they too try really hard to rock the boat.

Powershell was 2006, so I suppose the real "peak windows server UX" was 2016 when PS was relatively mature and came out-of-the-box with the latest version.

If MSFT had back ported servicing stack updates to 2016 it would still be usable. As it stands it bogs down unreasonably when applying updates and needs lengthy DISM /CleanupImage processes to be run periodically to reclaim disk space.

> Seriously, Windows 2000 was one of the most stable OS back in the day, rock solid. I used 2000 server as a desktop OS, instead of 98.

Really? Oh, compared to other Windows versions...

Because it never came close to the stability of OS/400, Netware 3, AIX, Solaris or even OS/2 v2.

I dunno how to compare stable to stable but I ran Win2k for so long that I got bored with it (something like 5-7 years) and never experienced a single crash. This is coming from a Linux guy btw… so I’m no Microsoft fanboy, just saying, it was as stable as any other stable OS.

Didn't mean to bash you, sorry.

I saw years of uptime on those systems whereas Win2000 iirc needed a reboot for every single update of the OS, and even for applications like IIS or Exchange.

Compared to NT4 it was probably very stable, since I remember telling most clients to just shut it down Friday evening and boot it Monday morning cause the pre-SP4 NT4 could not stay up more than three weeks.

Compare that to AS/400, where we pushed updates all over the country, without warning clients, to system running in hospitals, and there never was even the slightest problem. It sounds irresponsible to do that today, but those updates just worked, all the time and all applications continued to work.

> I saw years of uptime on those systems

This just means security updates were never installed.

(Or you claim that all those operating systems never had kernel-level security issues which seems doubtful...)

Since these systems were from the 90ies they indeed did not get security updates.

Most were only locally connected (for example OS/2 had a Token Ring in one building). The WAN connection (for AS/400) was trusted.

You are comparing supermarket apples (Windows) with localy grown plums (AS400). Even today, Windows is not able to update Office without closing it.

I will fully agree on OS/400, of the operating systems and platforms I have worked with, it is by far the most stable.

That is easier to achieve when your operating system only runs on your own proprietary hardware. (No mess of millions of drivers to write for one).

It worked well for years without any sysadmin touching it.

Well my mom was trained to be the "sys admin", which meant rotating backup tapes.

Part of my 1994's Summer job role. :)

While I don't disagree with that, in my experience all Windows instability on WinNT family (and I tightly worked with all end user versions of Win from 16 bit W3.11 to the recent Win11 with a very few exceptions) are caused by faulty hardware and/or bad drivers that can't handle it. I don't think I could remember any issue that I can't attribute to bad HW/3rd party driver.

Wrt Win95 & it's kind - all processes in that family essentially run in a single address space, and data "isolation" were "achieved" only through obscurity. If you knew some magic constants that were easily obtainable from disassembly, you could do anything there. So no wonder it was as bad as the worst program you've installed..

Almost all instability I’ve had with modern Windows or Macs has been caused by corporate installed malware - MDM software and virus software.

Haha, yeah, that crap adds indeed

I went from 98 to 2000 (rather than ME) and it was an amazing experience. It showed me what an operating system could be like. Of course, what I really wanted was Linux, but I didn't know better at the time.

So it is one of the most successful examples of dogfooding in history?

Pre Microsoft hotmail is one of the things I miss about the 'old' internet, logging in with Navigator 3.something in the library at uni.

Businesses are theoretically all about money but end up being driven by pride half the time.

Makes sense to me. After all, businesses are run by humans, who have egos to satisfy.

Engineers might take pride in their work, but on this level in a big organization, I rather suspect turf wars as motivation.

Of course. Why would you expect anything but? Pride is actually a very good driver of change if you ask me because people often do their best work when they are proud of what they are building.

Perhaps ego is a better term. Concretely, why migrate hotmail from unix to Windows except due to ego? The NPV has to be negative here.

Links in the original are dead but I think this is the Microsoft doc on “what could Windows do better” - https://web.itu.edu.tr/~dalyanda/mssecrets/hotmail.html

Thanks for linking to this.

The tone and content of this document is shockingly candid and frank. I think it did a ton to make Windows Server a better product. I have a lot of respect for the people at MSFT who reviewed the company's own product in such a critical light.

> It took years until they were successful.

The 90s were the dark ages of cloud computing. It was the age of system administrator, desktop apps, Usenet, and the start of the internet as a public service. At the time concepts such as infrastructure as code, cloud, and continuous deployment, were unheard of.

AWS, which today we take for granted, was launched on 2002, and back then it started as a way to monetize Amazon's existing shared IT platform.

Of course migrating anything back then was a world of pain, specially when it's servers running on different OSes. It's like the rewrite from hell, that can even cover the OS layer. Of course it takes years.

> At the time concepts such as infrastructure as code, cloud, and continuous deployment, were unheard of.

There existed different names and solutions for things like cloud. I worked with Grid Engine in 2000 after Sun acquired Gridware, but that project started in 1993. By 2000 we were experimenting with running Star Office on the grid and serving UI to thin clients (kind of what Google Docs or Office 365 do now, but on completely different stack).

IIS was wide open in Win NT/2k days. It took Microsoft some good years to patch the holes.

You've got me curious: what was the single biggest barrier to migration, if you're able to disclose it? I'm guessing it was something proprietary to AWS, like how they handle serverless or something that couldn't translate over directly, but I'm always eager to learn why a migration from X to Y didn't work.

Curious about details too. The parent's conclusion is to write off Azure, but I wonder if it's actually AWS or the way they use AWS that makes it hard to migrate.

Or put it in another way, if Mojong were to start with Azure but couldn't manage to migrate to AWS, which provider is the parent going to write off?

My experience has certainly been that AWS is both a) more stable, and b) has more migration resources and guides than the reverse.

It was easier to go to AWS than to Azure; and I've done both in the past ~4 years. Migrating to AWS was just technical work. Migrating to azure was 'fight unexpected bugs and the fact it doesn't actually work in surprising situations'.

The only reason to go to azure was that Microsoft waved a big fat $$$ discount to use their services.

Migrating to AWS was a breeze.

Yup, the hardest part about migrating to Azure was jumping between all the managed services that work everywhere else but are insanely buggy on Azure. We ended up with the most basic architecture you can imagine (other than AKS, which works great as long as you don't use any plugins) and we're still running into issues.

We have a very long list of Azure features and services that we've banned people from using.

Just got off a call with someone at Azure today who told us to setup our own NAT gateway instead of using Azure's because of an outage where we made too many requests and then got our NAT Gateway quota taken away for the next 2 hours.

To be fair AWS also has quotas on NAT Gateways.

Maximum 55k concurrent connections. After that, they make you deploy NAT gateways in other availability zones. And a max throughput of 10 Gbps.

I imagine AWS would also tell you to deploy your own gateway if you were running into the 55k concurrent connection limit of managed NAT.

AWS tends to be flexible with their quota enforcement in my experience, though.

Yea, I don't have a problem with the quota, more that the "out of quota" throttling lasts 2+ hours even after the traffic spike dies down.

> Yup, the hardest part about migrating to Azure was jumping between all the managed services that work everywhere else but are insanely buggy on Azure.

Care to point out a concrete example? I've worked with Azure a few years ago and I wouldn't describe it as buggy. At most, I accuse them of not "getting" cloud as well as AWS. For example, the whole concept of a function app is ass-backwards, vs just deploying a Lambda with a specific capacity. That is mainly a reflection of having more years working with AWS, though.

My experience with AWS is the documentation can be bad/wrong, it can be difficult to find stuff, but the actual services are very solid. It does what you tell it to do. Stuff works.

My experience with Azure is that it simply breaks in ridiculous and frankly unacceptable ways, all the time. It's like someone is unplugging network and power cables every couple hours just for fun.

> My experience has certainly been that AWS is both a) more stable, and b) has more migration resources and guides than the reverse.

I'm not sure how well founded the stability argument is. I still remember the infamous series of AWS outages that took place a couple of years ago.

The fact that AWS invests heavily in vendor lock-in is a problem created by AWS, not their competitors.

… I was scrolling back through the “400 reasons not to use azure” in the OP and off the top of my head I’ve seen a dozen of them personally.

You decide if that’s more or less stable than AWS.

I’d say the evidence is pretty empirical, but hey, all I can say is my experience was utterly unambiguous.

You can argue a lot of things, but hundreds of azure fails in a big giant list is probably one of the tougher ones to go “no, this is fine compared to AWS!” about, imo.

The networking is so, so bad in azure. We ran into all kinds of craziness with simple things that kept kicking us in the nuts like port exhaustion between different subscriptions. I quit my last job partly because they were wedded to azure.

This is a couple of years ago, so I fully expect most of the issues we had back then to be fixed by now, but it was definitely Azure that was the problem.

We wanted to use their hosted kubernetes solution(I forget the name) and pods would just randomly lose connection to eachother, everything network related was just ridiculously unstable. Host machines would report their status as healthy, but then be unable to start any pods, making scaling out the cluster unreliable. I also remember a colleague I regarded as a bit of a wizard being very frustrated with cosmosdb, but I cannot for the life of me remember what the specific issue was.

Our solution was actually quite well written, if I do say so myself, we had designed it to be cloud agnostic, just on the off chance that something like this would happen (there may have been rumours this acquisition would happen ahead of time).

But Azure was just utterly unable to deliver on anything they promised, thus the write-off on my part.

Pods loosing connection to each other is still very very common in azure.

To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway.

What’s frustrating is that it happens rarely enough to give the illusion of being stable, making it easy for PM to postpone hardening work, yet often enough to put a noticeable dent in your uptime if you don’t address it. Perfect degree of gaslighting.

> To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway.

Keep in mind that in Azure it's a must-have, whereas everywhere else it's either a nice-to-have or a sign your system is broken.

At that point, aren't you better off actually making your own services regularly disconnect clients so you iron out all of the reconnection bugs?

Having worked on the infra side of azure, I'm not surprised. Network is centrally managed and that team was a nightmare to deal with. Their ticket queue was so bad they only worked on sev 1 and the occasional 0. Nothing else got touched without talking to a VP and even then it often didn't change things.

Ohh, AKS. I had the 'pleasure' of using it quite early on, 5/6 years ago. We kept killing it by actually using it with more than 50 pods. You know you see one of few serious users when you get through the 3 layers of support and gets to talk to the actual developers in the US.

But my impression is that it's better now. In general my experience with azure is that the base services, those who see millions of hours in use, are stable. Think VMs, storage, queue etc. But the higher up you go in the stack, the fewer hours of use do they see, and the lower quality it gets.

I appreciate the context! Honestly, every time I start thinking that K8s might be the "universal cloud" orchestrator I've been hoping for/working towards, stories like this remind me just how...tenuous it can be relative to traditional VMs and standard containers-as-appliances. Still working to get my Admin cert for the spec, but it's definitely not something that sparks joy, if you catch my drift.

Thank you for the insights!

Current Mojang employee here, we moved fully onto Azure as of a few years ago AFAIK.

Some more game-oriented technologies of course have helped in the years since though.

Edit: AWS -> Azure :)

> we moved fully onto AWS as of a few years ago

Did you mean Azure?

Yes, I do. Whoops!

> Current Mojang employee here

Can I have my Mojang account back?

I literally cannot log in to it after it was forcefully migrated to Microsoft. Microsoft doesn't recognize my computer as not-a-bot. Something to do with being Linux, I imagine.

Or, can I get a refund?

I bought my kid Minecraft and spent two hours trying to get it set up to run on a Windows 10 machine.

In the end I gave up and he plays it on an old Ubuntu laptop instead.

How did it take two hours to install on windows 10?

it probably didn't work right away and two hours was spent on troubleshooting

This is a story from some parallel universe right here. You bought a microsoft game and wasn't able to run it on windows, but it works on ubuntu?!? I almost spilled coffee reading this.

Sadly, I don’t find it even a little bit surprising.

The problem was you probably ended up falling for the UWP version instead of the Java version. The Java version remains the community accepted “proper” version to this day.

Their bot detection systems are _extremely_ overzealous and don't even _tell you_ they're denying you nor offer ANY recourse to correct the problems.

I think I finally managed to make it work in a new/clean Chromium browser profile.

Every time I've made a Microsoft account it's let me create it, then two days later locked me out if I don't give up my phone number.

The first time it was a throwaway account only needed for one day so I abandoned it. The second time I had to give them my phone number. Very scummy.

Oof, this won't help you, but I recall sending a very angry email to me new corporate overlord in the Xbox org for blocking me from using their web services if my user agent said Linux.

This was during the "Microsoft <3 Linux" campaign, and I think I cited that and then told em Minecraft would not be able to move forward with Xbox account migration until they stopped such idiocy.

Since I was the dev tasked with migrating Mojang accounts to Xbox accounts, I felt I had at least SOME credibility to my claim that it was blocking me.

But honestly, modifying my user agent was easy, it just pissed me off.

They did fix that the same day tho, so I guess the believed me!

Moved my Minecraft license to a new Microsoft account as required. Microsoft account was flagged and blocked when I checked a few days after. And that's how I got scammed out of my Minecraft license.

It wouldn't have helped if you hadn't, since they've now deleted all non-Microsoft accounts.

Reminder to whomever is reading: if you bought the game during alpha, you have the right to all future Minecraft games and a premium account forever. Microsoft barely tried to uphold this by giving a free Bedrock license to alpha buyers for a limited time several years ago. I suppose you'd have to sue them now if they break it, and the judge will wonder why you bothered to bring a $20 dispute to court.

Oh you guys finally managed!? Cool to hear! I guess Azure must've gotten better then, back when I was there, the conclusion was that Azure simply wasn't mature enough to host Minecraft yet.

Again, this is a while ago: I remember when I started we were just starting to replace the old yggdrasil servers with the new micronaut based system which I think is still in use today?

I still remember that application fondly as the best architectured piece of software I've ever worked on. I hope all is well!

The vast majority of our services were overhauled when we started Minecraft: Earth (defunct AR game) as it was _very_ service heavy. It was actually really nice stretching our legs with services and growing a team on an incubator project, then moving onto the "real deal".

Those services and any written since then have all been .net based. Still lots of older services hanging around though :)

It has since migrated to Azure. I suspect there was a gap in the technology that was since closed, as AWS certainly had a head start in general.

IMHO, you're gonna struggle if you move anywhere else from AWS. We're migrating to GCP and there are gaps all over the place.

is it because there are features that AWS provides to you that are not available in GCP, or just the fact that setting up exact replicas of processes is hard for migrations like these?

Out of curiosity, what are the biggest gaps you've hit in GCP?

This is the reason I try to avoid proprietary bullshit services. Use EC2, Postgres, and S3, and you’ll be fine in any cloud or even on bare metal.

> proprietary bullshit services

> S3

Hasn't this recently been an issue where Amazon arbitrarily changes the S3 contract and all software following it as a spec has to play catch-up?

I think this happened a few times, but given there are many S3 clients, they can’t keep doing thiat or there will be a backlash.

Although perhaps it’s about time we made a proper standard based on S3, yeah.

It has, but that's not exactly on S3 is it?

The S3 system is proprietary to Amazon, and it's your fault if you're not using Amazon but you're relying on Amazon to not change it anyway, because they have no obligation to you.

The concept of object storage is not proprietary. You should be able to change your code to use a different object storage provider.

I didn't mean to throw any shade at S3. It certainly isn't their fault that people use their private protocol on other projects.

I was trying to point out that S3's protocol is "proprietary" so if you're using it you're still (somewhat) using "proprietary bullshit" in your stack!

I mean yeah, it isn’t really open – but it’s the informal “standard” that everyone is using nowadays. You can’t really expect every cloud provider and every app to just switch to WebDAV or something (though we can push for a gradual change here somehow!)

That sounds to my self-hoster ears as an expensive way to do self hosting. Isn't at least half the point of AWS to use their SaaS thingies that all integrate with each other, I think people now call it "cloud native software"?

Not that most of our customers whose AWS environment we audit do much of this, at least not beyond some basics like creating a vlan ("virtual private cloud"), three layers of proxies to "load balance" a traffic volume that my old laptop could handle without moving the load average above 0.2, and some super complex authentication/IAM rules for a handful of roles and service accounts iirc

(The other half of the point is infinite scale, so that you can get infinite customers signing up and using the system at once (which hopefully pay before your infinite AWS bill is due), but you can still do that with VPSes and managed databases/storage.)

The point of moving to AWS is often to benefit from their data centers and reliability promises. So VPC, EC2, IAM, maybe S3 have a clear point.

And one small note: apart from S3, virtually all AWS services are tied to a VPC, any kind of deployment starts with "ok, in what VPC do you want this resource?".

You can get 95% of the reliability for 10% of the price at any dedicated hoster. AWS just figured out the magic word "cloud" means they can charge you 10 times the price.

At Azure or GCP you pay a similar price but you don't even get the reliability so literally why would you use them? The only reason I see is that "cloud" means you can start instances at any time without a setup fee or contract duration. But with the amount of cost difference, you could have three times your baseline "cloud" load running all the time at a non-cloud hoster, and still save money!

It is an expensive way to do self hosting, yeah! I guess one reason is, sometimes it’s easier to just use one of big N clouds – e.g. if you’re in a regulated industry and your auditors will raise brows if they see a random VPS provider instead. (Or maybe not? If you’re doing that kind of audits I’d love to hear your thoughts!)

> Isn't at least half the point of AWS to use their SaaS thingies

It is. (That’s how they lock you in!) I think it’s okay to use some AWS stuff once in a while, but I’d be wary of building your whole app architecture around AWS.

I’m in the self-hosters camp myself :-) I’m building a Docker dashboard to make it easier to build, ship, and monitor applications on any server: https://lunni.dev/

The is sort of petty, but… your web page does the horrible scroll-and-watch-the-content-fade-in thing. This is annoying and makes me want to stop reading. It also makes me wonder if your product does the same thing, which would be a complete nonstarter. Seriously, if I’m debugging some container issue, I do not want some completely unnecessary animation to prevent me from seeing the dashboard.

Thanks for the feedback! No, the product’s UI is definitely on the pragmatic side: I think the only blocking animation we have right now is dialog windows sliding in (and we try to avoid these altogether!) Both the landing page and the app disable animations when prefers-reduced-motion is enabled.

I’ll rethink the landing page animations a bit later! (I was thinking about redoing it from scratch again, anyway :^)

I’ve just dropped you a note in the chat. We’re also on Swarm, and dealing with most of the stuff you address, and some more. Would love to contribute to Lunni, if you’re open to that :)

Got the note, thanks! I’ll get back to you later today – would love to chat sometime :-)

> I’d be wary of building your whole app architecture around AWS.

Yep yep, we're on the same page here, just that it sounded a bit like you recommend people to use EC2, managed Postgres, and S3, when that seems a bit like that defeats the purpose of using a big cloud vendor in the first place since those products can be gotten much cheaper elsewhere

Just clicked the link to your website: your hero statement I sorta have on a t-shirt (there is no cloud, just someone else's computer). The footer statement "To improve your browsing experience we don’t use cookies" I have nearly literally on one of my websites. We are definitely on the same page xD. The website seems to think I'm from Talinn btw, maybe a reverse proxy issue (that it looks up the IP address' location of a proxy instead of that of my German IP address)? I do second the remark posted in a sibling comment about fading in content, but (maybe you adjusted it already) it fades fast enough that it doesn't bother me as much as on some other websites. Could still be faster though imo

Good luck with the product!

> [...] and S3, and you’ll be fine in any cloud

Except Azure - AFAIK it's pretty much the only cloud provider that doesn't support S3 API, see e.g. https://learn.microsoft.com/en-us/answers/questions/1183760/...

Looks like one more reason for the list!

I’m personally thoroughly unimpressed by the bare metal S3 implementations I’ve tried.

And there’s also an issue with client libraries and their compatibility with various implementations. I recently discovered this issue:

https://github.com/boto/botocore/issues/3394

This is GCS implementing the S3 API incorrectly in a way that really ought not to break clients, but it’s still odd because the particular bug on GCS’s end seems like it took active effort to get wrong. But it’s also boto (the main library used in Python to access S3 and compatible services) doing something silly, tripping over GCS’s bug, and failing. And it’s AWS, who owns boto, freely admitting that they don’t intend to fix it, because boto isn’t actually intended to work with services that are merely compatible with S3.

As icing on the cake, to report this bug to Google, I apparently need to pay $29 plus a 3% surcharge on my GCS usage. Thanks.

Time to check out OpenDAL, I suppose.

> As icing on the cake, to report this bug to Google, I apparently need to pay $29 plus a 3% surcharge on my GCS usage. Thanks.

That's the price of a support contract, not a "bug report". And it's not "plus", it's "or": support costs $29/month or 3% of your monthly billing, whichever is greater. It comes with SLA agreements for fixing or working around your reported problems. Though obviously in this case they'll probably just tell you to use their own python library and not boto.

Oh, thanks Google, if my cloud spend is more than $967/mo, then I don’t get dinged by the $29 minimum. But this is the price of a bug report, because I can’t file a bug report without paying it.

And this situation is bad business. Google advertises that GCS has S3 interoperability support. And they have customers who use it in its interoperable mode. Presumably those customers could use GCS’s biggest competitor, too. Shouldn’t Google try to make the S3 interop work correctly?

GCS and AWS are commercial products for which you pay, not open source projects you can expect to support you for free. I don't know what to say here, if you have something "serious" to do on these platforms then a 3% overhead for support seems like an obvious choice.

Honestly it seems to me like you're excited to have found a bug and want to report it for glory; we've all been there. But No One Cares about that stuff in the world of commercial software. They fix bugs for real customers, not internet rock stars. If you aren't losing even $29 (one mid-tier meal!) from this bug, well... does it even rise to the level of "yell about it on HN?".

> Though obviously in this case they'll probably just tell you to use their own python library and not boto.

What’s the point of S3 compatibility then?

S3 isn't a spec. You'll never be 100% compatible with a third party product with ad hoc behavior. Companies do what they can as a migration aid, but at the edges there will always be bugs like this (which I'm gathering is really an issue with boto being inflexible about unexpected details and not so much "incompatible" behavior by GCS).

Again, this isn't an open source product. It's not like there's a universal agreement that coordination and compatibility are ideals to be respected, nor a public place for discussion and development. Google is trying to make GCS as S3 compatible as possible, but obviously they prioritize their paying customers (who seem not to have reported this issue). Amazon does not want GCS to be boto-compatible at all, really.

It is also the reason to use proprietary bullshit services. If there was not utility gap, then a reasonable evaluation would return that a migration to them is not worthwhile.

Now of course, that's a very sizable if...

I think the main reason isn’t the utility gap – more like “nobody got fired for buying IBM^W AWS” kind of thing.

That highly depends on what services your're using.

We migrated from AWS to GCP in 2016/2017 (mostly VMs and related stuff, CloudFront, etc - no lambdas) and it was pretty painless and everything worked smoothly until the end of that company.

Yep, sounds like my experience. Years ago, we migrated of Rackspace to Azure, but the database latency was diabolical. In the end, we got better performance by pointing the Azure web servers to the old database that was still in Rackspace than we did trying to use the database that was supposedly in the same data centre.

I kicked up a stink and we migrated everything to AWS in under a week.

When my day job used Azure we had the misfortune of using their managed Postgres instance, the latency of which was _appalling_.

We raised support tickets, which were mostly closed to the tune of “seems fine to us”. They seemed to think 20-30ms for a basic Postgres query, which for the same schema, data and hardware was <7ms on RDS.

Interesting, because until now I have the same opinion in reverse.

Each case is a special case how everything gets configured, but between Azure, GCP, AWS and IBM clouds, the ones with smoother experience on my case, have been on Azure, based on Java and .NET technologies.

And we also have our share of support tickets across all of them.

Now Azure back in its early days, 2010 - 2016 was kind of ruff, maybe this is the timeframe you're referring to?

> Fast forward about a year, and an uncountable amount of hours spent by both my team, and Azure solutions specialists, kindly lent to us by the Azure org itself, we all agreed the six figure bill to one of corporate daddy's largest competitors would have to stay!

Being tied to AWS and being unable to shake off a huge bill is not a trait of its competitors. It's a trait of AWS, and stresses the importance of not picking them as your cloud provider.

Also, I think it's unbelievable that a monthly 6-figure invoice charged to a company already with cloud engineers in their payroll is not justification enough to shift their workload elsewhere.

Low 6 figures is just a dev. If a team of 5 devs has to work on the problem for 5 years, then it will pay for itself in 25 years. Likely beyond any planning horizon of a company with yearly performance evaluation cycles.

> Low 6 figures is just a dev.

...in the US.

In Europe, even the likes of Amazon pays it's SDEs 70k/year. In Sweden, for example, Microsoft pays it's SDEs south of 800k SEK, which is about 70k dollars/year.

Low 6 figures is an entire team of Microsoft SDEs working full time for a year.

A great story of MS incompetence and Amazon’s vendor lock in.

Do you think it would be any different/better if you had to migrate to GCP (for example)?

Do you think it was the migration itself or the services on Azure?

Having worked with all three, there's certainly things that suck about all of them but I've found aws "most reliable" but also seems to have a large amount of disparate services needed to do things that were simpler on Azure.

GCP was pretty meh, but depends on what services you used.

Azure is a good choice for .net and sql server (azure sql or whatever it is now) but in but sure a service built for aws is going to "just work" on Azure (or vice versa).

after using AWS and Azure extensively AWS seems to be quite well engineered by some very smart people. The isolation between regions is extremely good and the Availability Zone model is quite effective in making very reliable systems if you are willing to pay the cost of inter-AZ data transfer. My company has an Active Directory controller in 3 different AZs

Azure is a mess designed by smart people with no time and little budget. Azure flat out lies about the AZs they have by claiming two halved of one data center is two AZs

I remember that shit show.

>Microsoft acquired Hotmail in the '90s

it was around 2007 (I'm not that old!)

https://en.m.wikipedia.org/wiki/Hotmail

> Founded in 1996 by Sabeer Bhatia and Jack Smith as Hotmail, it was acquired by Microsoft in 1997 for an estimated $400 million

Wrong decade, they really did acquire Hotmail in the 90's.

Reply on news.ycombinator.com