Hacker News Reader

This matches my (fairly extensive) experience with CosmosDb. It’s truly an awfully engineered product.

I cannot comprehend what organizational process lead not only to its creation, but also to its continued existence.

Is existence is a constant reminder that the road to hell is paved with good intentions, and that you better do your due diligence before adopting any piece of tech that’s not easy to replace. Even if it comes with a pedigree.

I wrote up some of the horrors stories a while back on HN: https://news.ycombinator.com/item?id=29295871

After suffering through the AWS SimpleDB disaster 10 years ago I will never use any of the cloud providers' hairbrained databases ever again. I'll use bog-standard Postgres or MySQL if they host it but nothing else.

What if I told you there’s now CosmosDB for PostgreSQL.

https://devblogs.microsoft.com/cosmosdb/distributed-postgres...

Which is just managed PG Citus with the word Cosmos in the name it seems.

This was both shocking and not surprising at all, all at the same time.

It’s worth noting how cloud vendors have fallen back on tried and true databases such as Postgres (mind you, often replacing the guts of those databases with new implementations.. but still).

> It’s worth noting how cloud vendors have fallen back on tried and true databases such as Postgres (mind you, often replacing the guts of those databases with new implementations.. but still).

Have they? Amazon created Aurora from scratch, with compatibility layers for different database engines (MySQL, Postgres), and GCP did the same with Spanner, which i wouldn't call "fallen back".

Yes, that's true - those products are definitely more of trying to displace the open source tech by borrowing the interface. I still think it is a plain admission to the fact of the popularity of the original tech (e.g. MySQL and Postgres). For sure, credit due where it is due for implementing entirely new implementations - but it's such a common pattern to "borrow" the open source interfaces and reimplement the core engine.

That said, you didn't mention things like RDS or Google Cloud SQL, which are cloud-based versions of standard open source dbs.

Yes. CosmosDB is a brand first, product second. It launched purely as a rebadged DocumentDB.

Which disaster was that?

I liked simpleDB. They kept on supporting it beyond my expectations even after lots of other options were available and it was long depreciated. Curious what the disaster was.

I have not suffered a disaster, but I have suffered its use, I really don't see a value in them versus a classical RDMS.

Is this analogous to NTFS?

For you young uns, back in the 1990s Microsoft was so convinced that NTFS made file fragmentation impossible that they didn’t provide a way to defrag for a very long time.

In fairness NTFS does usually make fragmentation a non issue away from pathological cases¹, much like ext2/3/4² which didn't have defrag tools³ for a long time either.

FAT12/FAT16/FAT32's earliest-free-block-no-matter-what allocation method⁴ trained people to believe that fragmentation is a universally rampant problem, but with a better designed allocation strategy it really isn't.

----

[1] for instance simultaneously growing files on near-full volumes

[2] more so ext4 with delayed allocation turned on

[3] at least not ones without attendant "are you really sure you want to use this on data you care about?" warnings

[4] I wonder if anyone retrofitted a brighter heuristic into an implementation of these or exFAT. Could be a (not massively useful but) interesting learning exercise for a budding OS developer.

I used it a few months ago, it still is a half-baked piece of shit. Code quality of client libs even worse than AWS.

It could have been easy. We could have used Postgres.

> It could have been easy. We could have used Postgres.

And then all that would have been left is just to make that Postgres in a globally distributed DB and manage it (with something like Citus). Postgres' native scaling out (not even talking about globally distributed) capabilities are basically nonexistent so you need third party tooling.

We didn't need a "globally distributed DB". This was an internal tool for a BigCorp.

> Microsoft thinks the exact workings of the partitioning is something that should work so well you don't need to know it in detail. But, if your usecase is slightly off you end up really needing to know. I know at least one team who routinely copy all their data from one Cosmos instance to another and switch over traffic to the copy just to get a partitioning reset; it is one thing to have to do it; another to discover in production yourself it has to be done with no prior warning..

AWS DynamoDB used to do that. When I was working for a team in AWS we ended up discovering an unexpected bad hash choice, had a hot partition and ended up with a crazy number of partitions and a need to have really high r/w allocation on it. The only option at the time was to roll over to a new table (which we could do and fix our hash choice, thankfully, without too much hassle).

They fixed that stuff a few years ago now so most (all?) of my "here be dragons" concerns about DynamoDB have been addressed.

Spoke with a customer last week who said their cost for cosmos is on the order of 10x what it would be for SQL.

I’ve learned to stay away from services that are high abstractions. Huge lock in, expensive and much more risk of it being eol’d.

In our platform we’ve actually dropped SQL in favor of Az blob and table storage. It’s going to reduce our database costs by 5x.

>(No, not a competitor, just have spent way way way too much engineering time moving first on and then off Cosmos and yes I am bitter)

Can you share what you migrated onto and the results?

We migrated to Azure SQL Hyperscale and are very happy about that.

Ah. That explains a lot of comments in this thread.

Cosmos is used because some tables we use are larger than 64TB. It’s decently useful for large chunks of data.

Well, even if WE do not have 64TB tables (where one need low latency for the entire table), I believe there are plenty of other mature NoSQL offerings out there? Whether you go to the other cloud vendors or self host.

Cosmos is at best unfinished. I used Google Data Store for years and it was finished in the sense that it always did what it says on the tin and didn't cause lots of problems for you (although I think higher latency than Cosmos, so may not be for every usecase -- there is also BigTable and Spanner).

And I don't believe Google would ever have done something like deploying a shared multitenant Jupyter system with full access to all customer's DB. Microsoft actually did that.

As I said the main problem is mainly how Cosmos is marketed, as a general purpose NoSQL. If it was marketed with all the caveats you learn about after you go live, and people only used it if they really had to, it would be a bit different.

Though realistically it would drive most people to other cloud providers.