Hacker News Reader

A team in my org worked with Jet for 2+ years to help y’all scale.

It was interesting seeing the biweekly status updates, they basically all started with “This is how Jet.com broke Azure core services this week”.

As much as it sucks, this was a deliberate strategy all the way from Satya - every employee knew Azure was a joke, but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.

> but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.

I don't get it. There's lots of distributed systems theory that could provide a more robust, analytical approach to a scalable architecture. If a system is regularly breaking like this, it sounds like it should be a "back to the drawing board" moment.

Back to the drawing board risks delaying your products with several years. Their strategy was probably the right one.

It sounds like their product has been considered crap for years, so they've had the time.

“We’ve had years to fix it, all the experience and knowledge we’d need, but we’ve tried basically none of it and we’re at a loss as to why our product is still garbage! It’s truly a mystery” - Azure leadership I assume.

My experience working on Google Cloud Platform is similar. Even when the solution is relatively technically simple, there are many organizational impediments and politics because the solution likely involves multiple fiefdoms.

The only thing that seems to get the fiefdoms to work on an issue is if a big enough customer calls enough of an attention to the issue so that no leader/fiefdom wants to be blamed in such a "high profile" issue that everyone got their work in gear for. This leads to organizational dysfunction since now everyone understands that there's no point in trying to fix anything until there is enough attention on it, and that is mainly achieved when a big customer complains.

I've even had projects that took months of engineering work that would resolve many user complaints about a lack of really basic functionality, and different elements of leadership would block the launch. I can only guess that some fiefdom did not like change and would not state their objection publicly, so I only got stonewalled, and I never got an explanation to throw away months of work. I am as certain as I can be that if a big customer complained loudly enough about the problem that such basic functionality was missing, the organization would demand that the fix be launched.

> My understanding is that Cosmos has gotten a lot better

For some values of "better", I guess. Performance is still terrible, their data visualization/inspection tools are shameful, their SQL dialect is finicky and has no error reporting beyond "something is wrong with the input value", and their official Python SDK has a race condition bug that can silently clear out your documents when under heavy load.

I used to work at a Cosmos-heavy house and I would utter "fucking Cosmos" around 15 times a day.

> My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth.

A couple of years ago I stumbled upon a Azure project which started off using the old timey Cosmos DB. Looking at the repository history from those days, I saw a bunch of Entity Framework configurations and navigations and arcane wizardry that would take an engineer months to really figure out.

Then there was an update to CosmosSDK, and all that EF noise was replaced by single CRUD operations that took the unserialized object, id and partition key as input. That's it.

Worlds of difference, and a simple key-value query takes ~10ms to do.

Yes, it's worlds of difference.

> Worlds of difference, and a simple key-value query takes ~10ms to do.

Unless that query goes over the Internet to another continent, that's a really long time isn't it?

Yes it’s ages but not multi continent long. Us east 1 to us east 2 is about 10-15ms in my experience.

You're right, I exaggerated unnecessarily there. Sorry

No reason to apologise - I assumed and tried to give a better reference.

It is an absolute eternity though. A KV lookup is fractions of a microsecond in a managed language like c#. A http request is in the microseconds range on localhost and a smidge more on a performant local network. A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network.

It’s an absolute eternity, and when thinking about this stuff and scaling, think “is it worth slowing down the normal case by 1000x to introduce an external service”

Some numbers no one asked for :)

ConcurrentDictionary<K, V> read latency is going to be around 7-15ns for the data hot in the cache, scaling with key length if it is string-based, and anywhere between 75ns and 150ns for reading that out of RAM. Alternate implementations like NonBlocking.ConcurrentDictionary can reach down to 3.5-5ns for integer-based keys given all data is in L1 and the branch predictor is fully primed, on modern x86_64 cores.

I've gotten in arguments over overuse of Redis for this very reason.

Redis is cool but people use it too willy nilly. It's expensive to run, and you have to pay a cost of (de)serialization, and network latency. Sometimes that's necessary, especially if you need to share stuff across multiple nodes and you're afraid of hammering the database too much, but a lot of the time, I'd say even most of the time, you can get away with just a big ol' global ConcurrentDictionary (or whatever the equivalent is in your language of choice).

Basically any thread-safe dictionary is fine, because no matter what kind of locking strategy they're using, it will almost certainly be faster than anything that has to hit the network, by several orders of magnitude, and you have one less thing to manage in your application. You can figure out the best way to invalidate old entries, or find a library to do it (e.g. something like a Guava cache), and you'll likely much better performance than arbitrarily farming to Redis.

> (...) but a lot of the time, I'd say even most of the time, you can get away with just a big ol' global ConcurrentDictionary (or whatever the equivalent is in your language of choice).

That's fine if all you're doing is caching limited data independently on each node and you have no requirement for consistency or even durability.

Most of the usecases you stumble upon do not fit that scenario. That's why "a lot of the time" suggesting a dictionary is plain wrong.

> That's fine if all you're doing is caching limited data independently on each node and you have no requirement for consistency or even durability.

You've assumed you have multiple nodes here. Our point is that you can scale _significantly_ more with a ConcurrentDictionary (or equivalent) than you think - far more than a single node backed by redis will go. You have a point about durability, but in my experience, most apps that lean on Redis don't actually solve that problem. If the app goes down mid request, it will leave broken state behind. The data itself in redis might be valid, but the app can send partial data. The solution to this is the same as handling durability in a non-redis case too.

Also, runnign Redis on the same machine for persistance is an option for durability.

> Most of the usecases you stumble upon do not fit that scenario. That's why "a lot of the time" suggesting a dictionary is plain wrong.

I disagree. Most of the usecases _do_ fit this scenario, and Redis is (over) used as a tool to allow for horizontal scaling caused by an unwillingness to scale vertically. At the level of Meta/X yes you absolutely need it. But for a web app with about 10k concurrent users, you can handle it on a single node.

> A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network.

I think you need to revise your assumptions. Even optimized DynamoDB setups shows average response times in the 15ms range and P99s at the 20ms range.

One of the mistakes you seem to be making is assuming that to do a query on a globally distributed database with multiple partitions, you'd only need to hit one single box and no data exchange would take place. That bears no resemblance with reality.

> Unless that query goes over the Internet to another continent, that's a really long time isn't it?

If you're hosting a service in a cloud provider and you implement your services so that your call to the cloud provider's database goes over the internet to another continent, you have serious problems but none of them are caused by the cloud provider.

Also, CosmosDB is globally distributed.

They meant 10ms is slow.

> They meant 10ms is slow.

I don't think anyone making that sort of claim knows anything about cloud services. A single roundtrip of a no-op request within the same data center takes 0.5ms. Add querying across multiple partitions and data seeking, and you don't find cloud providers doing better.

To frame how oblivious that claim is, DynamoDB is lauded for it's response times being sub-20ms.

https://stackoverflow.com/questions/34552625/how-to-get-sub-...

No that's exactly what I mean: I expect the round-trip within a datacenter to take roughly 0.5ms, I expect the key lookup to take about that time or less, so 10ms is roughly an order of magnitude more than I would expect a "simple key-value query" to take

For comparison if you run your own hardware and do a memcached KV lookup with a different server on the same rack, p99 times are slightly under 1ms. Given the guarantees of cosmosdb ~10ms isn't that bad for a p100

> Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources!

What the ever loving heck... seriously?! Why wouldn't this be a control plane API that reconfigures a data plane?!

I think it was querying the database to get usage information, and it was blocking the loading of the page as a result. It's been a few years, but if I recall correctly, the .NET API for changing the database provisioning did work, so some people would hack up a quick script to change resources in an emergency.

I think they did fix it eventually, because I know that multiple people on my team complained to Microsoft about it. Very short-sighted decision on their end, but to be fair it was a brand new product.

This is how hosted Elasticsearch works today, too. Really mess up your ES? Good luck with the web UI!

> What the ever loving heck... seriously?!

What's so surprising about that? If your CRUD operations take longer to do, and you're doing those to drive a GUI, of course the GUI will lumber along.

It's good practice to separate out your control plane and data plane, so in this kind of scenario you can use the control plane freely to manage and scale up the data plane without worrying about the data plane underresourcing affecting your control plane operations.

The reverse also applies; by separating them you can have issues with your control plane but not have the database go down.

Control plane and data planes should be running on different hardware.

It shouldn't lag the web interface. There are common ways to handle this, such as making all resource mutations asynchronous, or if you want to keep it synchronous, ensuring that control processes get guaranteed CPU resources. It is not a particularly challenging thing to do.

Because that would be the smart and sensible way to design a service like that, and those words are unequivocally not associated with Azure in my experience.

You would assume that the UI process would be running on Very High priority.