Jepsen Disputes MongoDB's Data Consistency Claims
infoq.com
infoq.com
I had the misfortune to use MongoDB at a previous job. The replication protocol wasn't atomic. You would find partial records that were never fixed in replicas. They claimed they fixed that in several releases, but never did. The right answer turned out to be to abandon MongoDB.
I imagine things are better now.
on edit: between the lines.
Hyperbole aside, the best option often is somewhere in between. I find that a relational database with columns for primary keys/relations/anything used in WHERE statements in the normal application path, and a json blob for everything else that's just attributes for the current row/object/document, makes for a very flexible and successful DB schema. You get all the strong relational consistency benefits of traditional schemas, plus the flexibility to add and modify features that don't require relation changes without touching the schema, and the ability to represent complex structures while still allowing ad-hoc and admin path queries to peek into them where necessary.
In fact, most "fully normalized" databases end up reimplementing a key/value store or twelve in there anyway (e.g. config settings, user attributes, and the like). Might as well just use JSON at that point.
What's the point of putting, say, every single user management field into columns in a "users" table? The regular application is never going to have to relate users by their CSRF token, or what their UI language is, or any of the other dozens of incidental details associated to a user, some visible, some implementation details.
What matters are things like the username, email, name - things the app needs to actually run relational operations on.
If you look at any real application, pretty much everyone has given up on trying to keep everything relational. That would be a massive pain in the ass and you'd end up with hundreds of columns in your users table. You'll find some form of key/value store attached to the user instead. And if you're going to do that, you might as well use a json field.
And if you do use a json field with a database engine that supports it well, like PostgreSQL, you'll find that it can be indexed if you need it anyway, and querying it is easier than joining a pile of tables implementing a KVS or two. Because yes, I might want to make a report on what users' UI language is some day, but that doesn't mean it has to be a column when Postgres is perfectly happy peeking into jsonb. And I don't need an index that will just cause unnecessary overhead during common update operations that don't use it, when I can just run reports on a DB secondary and not care about performance.
I designed an application in this manner and it has turned out exceedingly well for me. We have only had about a dozen schema changes total across the lifetime of the app. One of them involved some JSON querying to refactor a field out of JSON and into its own table (because requirements changed) and that was no problem to run just like any other database update. If we need to move things to columns we will, but starting off with an educated guess of what will need to be a column and dumping everything else into JSON has undoubtedly saved us a lot of complexity and pain.
One of our tables is just a single primary key and then a json blob. It stores event configuration. Why? Because it's always loaded entirely and there is never any reason to run relational ops on it. It's a huge json blob with lots of little details, only edited administratively, with nested fields and sub-fields (which is way more readable than columns, which are a flat namespace), including a sub-document that is actually a structure that drives generation of a pile of HTML forms. If I'd tried to normalize this I would have ended up with a dozen tables, probably more complexity than the entire rest of the DB, and a huge pile of code to drive it all, and every single application change that added an admin knob would've had to involve a dabatase schema change... All for what? Zero benefit.
So I have to agree with GP wondering why JSON is so important to people, and is even portrayed as a relief or saviour to developers. In my experience, JSON in a relational DB is always a sign of organizational failure, developers using the wrong tool for the job, or not knowing what they want or do.
Fully normalized databases are a nice academic idea, but the supposed benefits of going all the way don't materialize in the real world. That kind of approach is just old school cargo cult - just like full NoSQL is new school nonsense. Good developers know that the answer isn't to use whatever was the fad was when they were in school, be it fully relational databases or NoSQL or anything else, but rather to look at the available technologies and take the most appropriate bits and pieces in order to make a project successful.
After all, if JSON were nonsense, why would a serious relational database like PostgreSQL be adding full support for it? They know it has good use cases.
I know it has full use cases, so I use it, along with proper primary and foreign keys and a properly relational base data model. Yes, all my primary objects are tables with proper relations and foreign keys (and a few constraints for critical parts; not for everything because 100% database side consistency is also an impossible pipe dream, as not every business rule can be encoded sanely in SQL). Just don't expect me to add a user_attribute table to build a poor man's KVS just because people in the 90s thought that was the way to go and databases didn't support anything better. I'll have an attributes JSONB column instead.
And yes, JSON is just a trivial data serialization format that happens to have become de facto. It has an intuitive model and the concept isn't new. It just happens to have been what became popular and there is no reason not to use it. Keep in mind that PostgreSQL internally stores it in a more compact binary form anyway, and if clients for some programming languages don't yet support skipping the ASCII representation step that is an obvious feature that could be added later. At that point it ceases to be JSON and just becomes a generic data structure format following the JSON rules.
I could store it in a table with key and value columns, but since I always use it together as one thing, I don’t see the benefit and it just means more rows would need to be accessed.
Maybe its not a good design, but it works well for me and makes my life easier.
I know that you, dear developer, would never produce inconsistent data. But lots of other developers do.
It is often the case that you will need to query that inconsistent json data, but either the pain is too great, or the value too low, to normalize that data. Thus, you dump it as is into a json field or database.
I don't need JSON per se, I want to store data with a predefined, recursive format which most of the time will be just serialized / deserialized but occasionally also queried without having any idea ahead of the time what the query will be. For all I care, it could be Python pickle, PHP serialize, XML or whatever ungodly format you want as long as the above stands. (XML actually works in MySQL thanks to ExtractValue and UpdateXML but alas, I break out in rashes when I touch XML :P)
Let's say you want to display a page of text and images. (Doh.) But you want to give your authors great flexibility and yet nice styling so you give them components: a rolodex, a two-by-two grid, tab groups and so forth (we have 44 such components). This stuff is recursive, obviously. Are you going to normalize this? It's doable but a world of pain is an accurate description for the results. The data you get back from deserializing JSON should map pretty well to the templating engine.
Rarely you want to query and update things, some analytics, some restructuring etc. If it were just a key-value store with page id and serialized data the only way to do maintenance would be to deserialize each page and manually dig in. Sure, it's doable but having the database do that is simply convenient. That's the reason we use database engines, right? At the end of the day, you don't need SQL, you can just iterate a K-V store and manually filter in the application in whatever ways you want -- it's just more convenient to have the engine do so. Same here. The nicest thing here is that if someone wants ongoing analytics you actually can add an index on a particular piece in the blob and go to town.
You’d be astounded how common it is at so-called “enterprise” startups. It blew my mind.
A lot of people simply never went through the LAMP stack days and have little/no experience with real databases like Postgres (or even MySQL). It’s disheartening.
For us, as a result, it means anytime we add a new model or change a table, we write the table definition in more verbose api, and sometimes resort to sql commands for adding things like defaults in the migrations. (sequelize for some reason can not specify uuid default value for postgres, for example, so we set a default ourselves, even though we don't need one as we have time dependent uuid generator on the client to help with indices).
We kind of learned working around sequelize shortcomings :)
I am still looking for a tool that'd make incremental backups on postgres easier than it is, but for now things are ok.
The biggest annoyance I had with migrations was anything that needed a model to be defined inside the migration. I eventually gave up on that and just wrote my own scripts to work around it. But perhaps I just wasn’t persistent enough...
Are you sure?
Thanks for the clarification.
As an example: would you consider the backend software stack that manages physical access to the campus 'mission critical'?
Mission critical - essential for operating Facebook.com
Is is a company wide system? Its all fun and games until the cooling in a dc starts to have issues and the facilities team can't gain access because....mongo... ?
Disclosure: I worked at Facebook, but not in that department, or anywhere near any MongoDB.
In much the same way I wouldn't say that my site is powered by Microsoft Excel; but you can be sure Microsoft Excel is used in my company.
The docs you likely found on the wiki are dated, but MongoDB is definitely being used in Enterprise Engineering.
Source: I'm currently on the EE traffic team.
It's difficult to make accurate blanket assertions about large technical organizations, which Facebook certainly is.
I am amused by the downvotes that my previous comments received.
But I'd think MongoDB the company increasing in revenue isn't totally related to the quality of MongoDB the database. In fact a lot of their products seem to be targeting the "I don't want to learn how to set it up and understand indexes" crowd.
No. I’d consider adding an index. An index is not free, it comes at a cost and that cost may well be higher than the costs of not having that index. For example, if a reporting query that runs once every few hours is lacking an index, the cost of updating that index on every write (and the disk space/memory used) may well exceed its benefits.
Just to note, this is referring to the features for a hosted databased on cloud.mongodb.com, and not something built into MongoDB the database.
I guess to each their own.
But it auto-suggests what index to use and has a button for you to immediately apply it. I'd say it definitely intends for you to avoid learning how indexes in MongoDB work. The index suggestions it makes are often terrible.
> performance [which is laughable even with indexes]
It really depends on your use-case and how you can structure and query your data. For one such service I'm the lead on I haven't found anything faster that doesn't sacrifice the ability to do more generic queries when we occasionally need to. (Yes we've benchmarked against postgres json fields which is always everyone's first suggestion)
Heh, yeah, you're right. So I guess it's the worst of both worlds: they try to not have you learn about indexes, but they give you terrible index suggestions, so when those don't work out you still end up having to learn about indexes :)
But I've always been curious: as a person who like MongoDB, do you have an opinion about Mongo vs. Couch? Keep in mind that you don't have do convince me that there are niches where that style of DB is appropriate ;-) I'm mostly just interested in the comparison since I've never spent any time looking at MongoDB.
It's great for things like a realtime layer of some app that merges data with a slower and more historical layer of data running on a SQL engine or something safer. Or for services that provide realtime or recent-time analytics, while storing your historical data somewhere else (see any patterns here so far? :P ). In my case the main usage is for an advertising bid and content serving engine, which was pretty much the ideal example use-case for MongoDB mentioned in books I read years ago when first learning it.
Just to note, yes the data integrity problems are "fixed", but only if you configure your instances properly and your read and write statements properly. It's not terribly hard to do, but I don't know if I would really go recommending MongoDB for newbies. If you know how to configure it properly for your data-safety needs, and would benefit from being able to have a flexible schema in early development.... I'd still maybe suggest looking at other document DBs unless you need the read/write speed Mongo can give on simple queries (and fresh projects probably do not need that)
For situations where you don't know the schema or for different schemas per record mongo is a great place to dump.
For data when you care about speed and don't care about losing some data. Think sending back a game screen when the client moves and requires a redraw. Depending on how fast the screen is changing dropping a screen isn't the biggest deal.
Reporting was a little bit more difficult but somehow rewarding.
> Did any of you actually read the article? We are passing the Jepsen test suite and it was back in 2017 already. So, no, MongoDB is not losing anything if you know what you are doing.
https://twitter.com/MBeugnet/status/1253622755049734150?s=20
Can you imagine saying the phrase "if you know what you are doing," in public, to your users, as a DevRel person? Unbelievable.
However I can _quite easily_ see how a non-native English speaker could use the phrase “if you know what you are doing” to mean “if you are careful”.
But obviously it is a tactic of the doomed.
He just replaced "cannot consistently synchronize data" with "cannot consistently deploy a system that can consistently synchronize data". But what's the difference between those statements to people that need to solve that problem? None.
This kind of thing is a scourge. I had a Chinese friend respond to something I said once by saying "that's nice". It looks so innocent... but it's really hard to overlook the fact that "that's nice" is a serious insult coming from a native speaker. I had to ask them to please never use that phrase to me again.
You can also be explicit about who's being nice. "That's nice of you."
This really depends on what you want to mean by "nice".
To my ear (American who grew up in the South, lives in the Midwest, and works with people largely from the Mountain West), "that's nice" could definitely be used dismissively or sarcastically, but there's any number of ways to say it that are actually sincere and genuine, and I can't imagine a situation in which it would be a "serious insult".
I mean, I can imagine that in a professional setting, if the coworker was saying it with a sarcastic tone or being showily bored or dismissive when saying it, that their behavior might be insulting. But anything can be insulting if it's delivered in an insulting way. "That's nice" has no particular edge to it to my ear.
So I have to assume you aren't American, or that this is a regional thing that I don't have experience with. In any case, your reaction to "that's nice" reminds me of an American friend of mine who moved to London and when her coworkers would ask her if she had a preference about where to get lunch, she would reply "I don't care", which would be totally normal (to me) in the US. But to her British colleagues, that word choice made it a very rude thing to say (the appropriate reply being "I don't mind" to indicate that she didn't have a preference and was willing to go anywhere).
- The system warns about unsafe usage at either compile time or runtime, and you ignore at your peril.
- The system does not warn, but official documentation is consistently verbose about what is required for safety.
- Official documentation isn’t consistently helpful and can be downright dangerous, but the community picks up the slack.
- The company gaslights the community into believing it is possible for a non-core-team member to “know what they are doing” from one of the above levels when Jepsen provides written evidence that this is not true.
I’m fine with things that are the third level from the top. I like to live dangerously. But I don’t think anyone can look at that last level and say “people are giving informed consent to this.”
I'm much more concretely worried by a software design for which the authors (not hostile critics) consider "if you know what you are doing" an acceptable safety and quality standard for data integrity.
Actual example: a long time ago someone in my company introduced a race condition into their product because they didn't know about transaction isolation levels (or the locking facilities exposed by MSSQL). I can give many more, as I'm sure anybody here can.
All complex tools need large investment in time to understand. I suspect MongoDB's team are using that as an excuse but in general you must jnow what you're doing with any tool.
If it happens because the driver just forgot to signal, it's a mistake and it's hard to blame the car. It's like knowing what transaction isolation levels can be used, but asking for the wrong one or trusting a random default.
On the other hand, if the directional lights remain off despite pulling the lever because the car is in a probably illegal power-saving mode, activated by turning the headlights off, and the driver should have reactivated directional lights with an ambiguously named item in the depths of some options menu, blaming the accident on user error would be very partisan: it's the car that doesn't know what it's doing. It's like figuring out the correct transaction isolation level according to DBMS documentation, asking for it, and expecting it is actually used.
I pretty much refuse to deploy a new instance of it now, I've been burned too often.
As an intern at Shopify, I got an email from MongoDB asking us to switch. Shopify was 10 years old the time. Plus several coworkers would also receive similar emails two years later (and some in between of course).
I have a shirt from MemSQL that says "Friends don't let friends NoSQL" and I wear it proudly.
"""
Curiously, MongoDB omitted any mention of these findings in their MongoDB and Jepsen page. Instead, that page discusses only passing results, makes no mention of read or write concern, buries the actual report in a footnote, and goes on to claim:
> MongoDB offers among the strongest data consistency, correctness, and safety guarantees of any database available today.
We encourage MongoDB to report Jepsen findings in context: while MongoDB did appear to offer per-document linearizability and causal consistency with the strongest settings, it also failed to offer those properties in most configurations.
"""
This is a really professional to tell someone to stop their nonsense.
>>> I have to admit raising an eyebrow when I saw that web page. In that report, MongoDB lost data and violated causal by default. Somehow that became "among the strongest data consistency, correctness, and safety guarantees of any database available today"! <<<
It's not wrong, just misleading. Seems overblown given that most practitioners know how to read this kind of marketing speak.
Based on this, my understanding is: most of the time you want a relational database. If a relational database becomes a bottleneck for certain data, and you don't want to do typical scaling solutions for relational data, then you need to know what you'll trade for the higher performance. Based on what you trade, you then decide what kind of data store you will use.
So basically whatever MongoDB was doing 10 years ago, they are continuing to do there. They did not change at all, yesterday or two days ago there were few people defending mongo that indeed in early years mongo want the greatest, but it is now and people should just stop being hang up in the past.
The reason why people lost their trust with mongo wasn't technical, it was this.
MongoDB explains that pretty well: https://www.mongodb.com/faq and https://docs.mongodb.com/manual/core/causal-consistency-read...
Postgres most certainly does fsync by default.
It's tru, you can disable it, but there is a big warning about "may corrupt your database" in the config file.
My point is people complain about MongoDB are the one not using it most likely, MongoDB is very different from 10 years ago.
I like to remind people that PG did not have an official replication system 10years ago and as of today is still behind MySQL. No DB is perfect, it's about tradeof.
So wal is synced before commit returns, and if you power cycle immediately after, the wal is played back and your transaction is not lost? So it's fine?
It does not need to sync all writes, only the records needed to play back the transaction after restart. This is what all real databases do.
So PG keeps data consistent by default - unlike MongoDB.
> MySQL and PG are not truly consistent per default, they don't fsync every writes. MongoDB explains that pretty well [links]
Where in those MongoDB doc links is there anything about MySQL or PG?
On MySQL: https://dev.mysql.com/doc/refman/8.0/en/innodb-dedicated-ser...
InnoDB uses O_DIRECT during flushing I/O, but skips the fsync() system call after each write operation.
The fsync thing is more complex than it looks like.
And, obviously that's a bug, it's designed to do so.
Also, if you write with O_DIRECT, a fsync is not needed, as it's how you tell the OS to block until written.
Whatever failings MySQL or PostgreSQL may or may not have are not important at all here.
* Mongo: I like things easy, even if easy is dangerous. I probably write Javascript exclusively
* MySQL: I don't like to rock the boat, and MySQL is available everywhere
* PostgreSQL: I'm not afraid of the command line
* H2: My company can't afford a database admin, so I embedded the database in our application (I have actually done this)
* SQLite: I'm either using SQLite as my app's file format, writing a smartphone app, or about to realize the difference between load-in-test and load-in-production
* RabbitMQ: I don't know what a database is
* Redis: I got tired of optimizing SQL queries
* Oracle: I'm being paid to sell you Oracle
Almost none of is remotely accurate e.g. RabbitMQ isn't even a database.
We use it for a very specific use case and its been perfect for us when we need raw speed over everything. Data loss is tolerable.
It may be good idea to take a break from the computer and find something less stressful to do.
[0] https://www.confluent.io/blog/okay-store-data-apache-kafka/
[1] https://dzone.com/articles/is-apache-kafka-a-database-the-20...
The fact that you put Kafka and RabbitMQ in the same category sort of makes me feel like you're out of your element, Donnie.
It used to be that bargain basement shared-hosting providers would only give you a LAMP stack, so it was MySQL or nothing. But if you're on RDS, Postgres every time for my money.
It's really sad because all the contributors to Postgres have made an AMAZING database that's such a joy to work with. And then there's PgAdmin4 where its almost like they just don't care...
I don't feel I'm smart enough to contribute anything to PgAdmin4 to try make it better. So I stick to DataGrip and DBeaver.
For MySQL, I haven't found anything that beats SequelPro. For Postgres, I haven't found anything that comes close to parity, but my favorite is Postico.
I know people that swear by IntelliJ for their db stuff, it just never hit home for me personally though.
It’s the only DB client that doesn’t look like it was built in the 90’s. Slick UX & UI. Nice balance between developer tool & admin tool
I can't compare against SequelPro as I don't have a Mac, but DBeaver's worth a try for anyone looking for a cross platform DB editor
I'd probably choose Postgres over MySQL for a new project just to have the improved JSON support, but there's upsides to MySQL too:
- Per-thread vs per-process connection handling
- Ease of getting replication running
- Ability to use alternate engines such as MyRocks
Though I'm willing to put up with it due to its incredible compression capabilities...
You can’t cut corners like that without inviting questionS about the character of the primary committers. The pecking order in software is about trust.
People don’t let that stuff go easily, which is why you still see people harping on MongoDB. Once someone is labeled a liar And a cheat, everything they say that doesn’t add up is “guilty until proven innocent.”
The utf-8 situation is on top of a bed of half truths. Things like publishing benchmarks with backends that don’t support isolation. A cornerstone of a good DB is handling concurrent access efficiently and correctly. Drawing attention to other benchmarks is a lie by omission. Better than just being incorrect for a decade, certainly, but still sketchy.
I gotta say, as much as I hate it with a passion, and as often as it breaks for seemingly silly reasons (so many deadlocks), it's at least tolerable (even if I feel like Postgres is better by just about every metric).
Did I miss something huge?
But also no, RabbitMQ and Kafka and the like are clearly message buses and though they might also technically qualify as a DB it would be a poor descriptor.
Back when I worked in LA my CTO used to joke that most places use Microsoft Outlook as a database and Excel as BI tool.
Confluent, the company behind Kafka, are 100% serious about Kafka being a database. It is however a far better database than MongoDB.
I had to work on a tool that shows what's wrong with an assembly line: missing parts, delays, etc... So that management can take corrective action. Typical "BI" stuff but in a more industrial setting.
The company went all out on new technologies. Web front-end, responsive design, "big data", distributed computing, etc... My job was to use PySpark to extract indicators from a variety of data sources. Nothing complex, but the development environment was so terrible it turned the most simple task into a challenge.
One day, the project manager (sorry, "scrum master") came in, opened an excel sheet, imported the data sets, and in about 5 minutes, showed me what I had to do. It took me several days to implement...
So basically, my manager with Excel was hundreds of times more efficient than I was with all that shiny new technology.
That experience made me respect Excel and people who know how to use it a lot more, and modern stacks a lot less.
I am fully aware that Excel is not always the right tool for the job, and that modern stacks have a place. For example, Excel does not scale, but there are cases where you don't need scalability. An assembly line isn't going to start processing 100x more parts anytime soon, and one that does will be very different. There are physical limits.
Unfortunately which specific features of Excel are acceptable to remove are unknown until you have already way over invested into the project.
The best I've seen this done is having Excel as a client for your data store. Where read access is straightforward and write can be done via csv upload (and heavy validation and maybe history rollback).
That way the business can self-service every permutation of dashboard/report they need and only when a very specific usecase arises do you need to start putting engineering effort behind it.
I suppose you can also supplement the Excel workflow with a pared down CRUD interface for the inevitable employee allergic to excel.
Here is another option that we use instead of CSV import.
Our applications support custom reports and custom fields.
Users can define new reports and run them on demand.
They can also define custom field types with validation, data entry support, etc.
This combination provides some of the extensibility of Excel while retaining the advantages of an application.
Edited for wording changes.
The devil is in the details, and software is nothing but details. The product owner at the company I work for likens it (somewhat illogically, but it works) with constructing walls. You can either pick whatever stones you have lying around, and then you'll spend a lot of time trying to fit them together and you'll have a hell of a time trying to repair the wall when a section breaks. Or you can build it from perfectly rectangular bricks, and it will be easy to make it taller one layer at a time.
Using whatever rocks you have lying around is like building a prototype in Excel. Carefully crafting layers of abstraction using proper software engineering procedures means taking the time to make those rectangular bricks before building the wall. End result more predictable when life happens to the wall.
You can complain about their solution or see it as an opportunity.
I posted elsewhere[0] in this thread about my employer's practice of replacing shared spreadsheets with web applications.
This approach works quite well for us and I would encourage you to consider it as an option.
[Source: I was friends with the guy who wrote it as well as other EToys employees. God that was a trainwreck.]
a32c/4214/585e/9cb7/a554/74133a5fc986
a32c/
4214/
585e/
9cb7/
a554/
74133/a5fc986
Basically if you're manually scanning the heirarchy for anything but a consistency check or garbage collection you've already lost.
18:35 $ tree .git/objects/
.git/objects/
├── 02
│ └── 9581d0c8ecb87cf1771afc0b4c2f1d9f7bfa82
├── 3b
│ └── 97b950623230bd218cef6aebd983eb826b2078
(...)
├── info
└── pack
├── pack-b1fe2364423805afb6b1c03be0811c93b19dedc9.idx
└── pack-b1fe2364423805afb6b1c03be0811c93b19dedc9.pack
10 directories, 10 filesWould love to talk to anyone on the EToys team or anyone who has done something similar.
I'm @akamaozu on twitter.
Many of my employer's applications started out as a shared spreadsheet or Access database.
Our development team worked with the users and built a web application to solve the same problem.
This approach has a lot of advantages:
* The market exists and has an incumbent. There's a lower risk of a write-off.
* The users are open to process changes. You still have to migrate people off of the spreadsheet, though.
* It's easy to add value with reporting, error checking, concurrent access, and access control.
* You can import the existing data to make the transition easier. This will require a lot of data cleaning.
Edited to add the following text from another post.
You can cover most of the requirements with a set of fixed fields.
The last 10% to 20% of the use cases requires custom reports and custom fields.
Users should be able to define their own reports and run them without your involvement.
They should also be able to define custom field types with validation, data entry support, etc.
If your web application has these two features and other advantages then you should be able to replace Excel.
ksqldb is actually a database on top of this.
The thing is that they have an incrementally updated materialized view that is the table, while the event stream is similar to a WAL ("ahead of write logs?" in this case).
Because eventually you can't just go over your entire history for every query.
Arguably the world's most popular database is Microsoft Excel.
If a customer's API was down, the event would go back on the queue with a header saying to retry it after some time. You can do some sort of incantation to specifically retrieve messages with a suitable header value, to find messages which are ready to retry. We used exponential backoff, capped at one day, because the API might be down for a week.
I didn't think of RabbitMQ as a database when I started that work, but it looked a lot like it by the time I finished.
Oracle is great if and only if you have a use case that fits their strengths you have an Oracle specific DBA, and you do not care about the cost. I have been on teams where we met those criteria, and I genuinely had no complaints within that context.
Every time I need to work with an Oracle DB it costs me weeks of wasted time.
For a specific example, I was migrating a magazine customer to a new platform, and all of the Oracle dumps and reads would silently truncate long textfields... The "Oracle experts" couldn't figure it out, and I had to try 5 different tools before finally finding one that let me read the entire field (it was some flavor of JDBC or something). To me, that's bonkers behavior, and is just one of the reasons I've sworn them off as anything other than con artists.
The big issue is that sqlite does full db locking for any operation, so during any write you can't just easily read at all.
This can be fixed with WAL mode, but WAL mode is broken in uts early versions, and new versions of sqlite aren't in all disteos yet, despite being out for almost a decade. And even WAL mode gets abysmal performance.
Definitely would not use in a read/write situation for a web API for various reasons, though. :-)
sqlite> create table foo (n int);
sqlite> insert into foo (n) values ('dave');
sqlite> select count(*) from foo where n = 'dave';
1SQLite does not use column typing except in integer primary keys.
> As far as we can tell, the SQL language specification allows the use of manifest typing. Nevertheless, most other SQL database engines are statically typed and so some people feel that the use of manifest typing is a bug in SQLite. But the authors of SQLite feel very strongly that this is a feature. The use of manifest typing in SQLite is a deliberate design decision which has proven in practice to make SQLite more reliable and easier to use, especially when used in combination with dynamically typed programming languages such as Tcl and Python.
It's intended behavior that's compatible with the SQL spec.
I think most people have realised weak typing is not a good idea in programming languages. It’s especially not a good idea in databases.
Logical replication or synchronous multimaster replication may meet your needs.
I can tell you this emphatically as I spent 6 months trying to eke out performance with MySQL (5.6). PostgreSQL (9.4) handled the load much better without me having to change memory allocators or do any kind of aggressive tuning to the OS.
MySQL has some kind of mutex lock that stalls all threads, it's not noticeable until you have 48cores, 32 databases and a completely unconstrained I/O.
EDIT: it was PG 9.4 not 9.5
yes, I should run the test again, but it was 6 months of my life, and I don't see how much could have changed.
They were both the latest and greatest at the time
> redo the benchmark today and I’ll be surprised if you come to the same results.
I would, but it was not just a benchmark, it was a deep undertaking including but not limited to: optimisations made in the linux kernel, specialised hardware along with custom memory allocators and analysing/tracing/flamegraphing disk/memory access patterns to find hot paths/locks/contention. (and at different scales: varying the number of connections, transactions per connection, number of databases, size of data, etc)
It was 6 months of my life.
> PGsql even has a wiki page where they discuss implementing MySQL features and changing their architecture so they can scale.
Just because mysql has some good ideas doesn't mean it scales better. I know for a fact that it didn't in 2015. I doubt that they have fixed the things I found, I could be wrong. But it would have to be a large leap forward for MySQL and PostgreSQL has had large performance improvements since then too.
also, I read that page and it talks nothing about scaling, just that some storage drivers have desirable features (memory tables are very fast, and PGSQL doesn't support it; archive tables are useful for writing to slower media, you can do this with partitioning but it's not intuitive)
I'm familiar with the variant, "InfoSec won't let us deploy a DB on the same host".
They have their own layer on top that happens to use SQLite as the storage format on disk[1]. This layer means they aren't using full SQLite at the application level, but rather using their custom database in the application, and SQLite within their custom database.
Further, they've customised the SQLite codebase as far as I can tell to remove much of the functionality that SQLite uses to ensure that multiple instances can safely edit the same file on disk together, then they memory map the file and just have many threads all sharing the same data.
[1]: FoundationDB also does this, and scales to thousands of nodes. The trick is that it's essentially _many_ separate, very simple SQLite databases, each being run independently.
Edit: never mind, I think the other URL - http://jepsen.io/analyses/mongodb-4.2.6 - deserves a more technical thread, so will invite aphyr to repost it instead. It had a thread already (https://news.ycombinator.com/item?id=23191439) but despite getting a lot of upvotes, failed to make the front page (http://hnrankings.info/23191439/). I have no idea why—there were no moderation or other penalties on it. Sometimes HN's software produces weird effects as the firehose of content tries to make it through the tiny aperture of the frontpage.
I'd pay to watch Kyle screaming at people in the MongoDB offices, not that he screams or anything. Just a spectacular mental image: "IT'S NOT ATOMIC! IT COULDN'T SERIALIZE A DOG'S DINNER!"
The stock market wants to see the product as a competitor with Oracle, so demands all the certifications that say so. MongoDB marketing wants to be able to collect money as if the product were competitive. Many of the customers have management that would be embarrassed to spend that kind of money on a database that is not. And, ultimately, many of the applications do have durability requirements for some of the data.
So, MongoDB's engineers are pulled in one direction by actual (paying) users, and the opposite direction by the money people. It's not a good place to be. They have very competent engineers, but they have set themselves a problem that might not be solvable under their constraints, and that they might not be able to prove they have solved, if they did. Time spent on it does not address what most customers want to see progress on.
Also this isn't 2011. MongoDB is not a competitor to Oracle and never really has been by people that knew that a DocumentDB was not usable as a SQL one. It's other SQL databases that are the real competitors e.g. Snowflake, Redshift are.
It is possible there are still potential users not buying until they get that story. MDB wants those users.
The syntax is very nice, I honestly think a lot of it's early success came from ease of use.
People have told me that they have since changed, but the evidence is overwhelmingly and repeatedly against them.
They seem to have been successful on marketing alone. Or people care more about speed and ease of use than durability, and my assumptions about what people want in a database are just wrong.
I think it depends. One could say the same about Redis, but it's wildly successful and people love it.
The difference is now they are advertised. Redis makes no claims to be anything other than what it is - a fast in-memory database that has some persistence capability but isn't meant to be a long-term data store. MongoDB, on the other hand, made (and continues to make) claims about being comparable in atomicity and durability to traditional SQL databases (but magically much faster!) that haven't withstood scrutiny.
Keep in mind, too, that most data ain't worth much. It's one thing to entrust data of low value in MongoDB; another to store mission-critical data in it. I would look askew at leadership who didn't ask hard questions about storing data worth millions or billions of dollars in MongoDB without frequent snapshots -- and even then, the value mustn't be contingent on the 100% accuracy of said data.
It's easier to reason about systems if there's fewer things that require durability guarantees, ideally you want to be able to draw data flows that look like a tree instead of a graph.
I find that Redis fits great because it's perfect for a whole bunch of different temporal shared state needs, everything from sessions to partial results. I've also deployed things like Ehcache, MongoDB, and Memcached to fit these needs and found other tools such as Kafka or RabbitMQ to be great "glue".
Having the root of your important data be something "boring" like Postgres or MySQL (or even Oracle!) is just good risk management to me. I wouldn't want to trust Redis or MongoDB for important data because it adds to the things I have to worry about. It's "keeping your eggs in one basket" while making sure that basket is really well looked after.
If the service had lasted longer, scaled bigger, and the business it supported had been more successful, we might have ended up with a now-classic MongoDB to pg migration. That was always an acceptable outcome, and it would have not invalidated going with Mongo at the start.
I assume that you mean write once data. If you mean write only you might as well use /dev/null.
[Edit] in case I wasn't clear: Imagine data that is written and rewritten and overwritten before it gets read; a user that never returns to your product - his state data was written but will never be read. That sort of thing.
I think 90% of the Mongo installs I've been exposed to were set up by people that were tired of fighting with Hibernate configurations and schema migrations.
It's also popular among people whose definition of "legacy software" is "that app I stopped working on after three months because I have something shiny and new."
- they don’t know why, it was just the one they learned/heard about first
- there is a lot of tooling for it
We concluded that other databases need to start prioritising usability; something few developer tools usually care about.
[1] https//supabase.io
When are you launching?
> When are you launching?
We're in alpha now - https://app.supabase.io (desktop only)
But, if you need a traditional ACID database, the flexibility comes with punch in the groin technical debt.
I absolutely agree it's been used by people who just don't want to write SQL queries, or being used as a text-search-engine in place of something like more appropriate like ElasticSearch, but to mock successful projects who were based on it seems silly. It reminds me of interviewing candidates at a startup who primarily used PHP/MySQL. Most of them openly laughed and called it all horrible. I voted "no" on them, and sometimes injected a somewhat toxic "ah, you're right - we should close up shop. Someone call Facebook - tell them their tech stack is horrible - shut it all down!".
You can learn a lot about a developer by asking "What do you think about Mongo, JavaScript, or PHP", and if their response isn't a shrug, they're probably more concerned with what editor is correct than if the product they're building is useful. It's an exceptional filter to reject zealots and find pragmatists.
All that said, MariaDB with MyRocks is _awesome_, but certainly not with the default settings :)
Sure, if they’re being rude about it. A developer saying that it will not fit the use case or talking about spending a month of their time fixing a production issue caused by MongoDB will definitely not get a “no” from me. I’m not hiring subservient people I’m hiring people who can think for themselves and choose the right tool for the job, which Mongo rarely is.
Why on earth would that be a problem? Why are you sensitive about tools? Do the tools have feelings?
Tools are means to an end. I much prefer working with people who care about the product and the business than what tools they used to build said product/business. I'm looking for "hey, you could save a lot of time by using tool Y", rather than "You people are idiots for even considering tool Z".
So yes, writing a website with assembler raises eyebrows and it is quite OK to ridicule bad languages such as PHP.
It's a shame that Rethink did so many things right and failed as a company while Mongo continues to do almost everything wrong as a company and still gets business.
This seems to be more the rule than the exception, doesn't it?
It's even not that hard to come up with explanations for this, main one certainly being that popularity depends essentially upon simplicity.
And simplicity might not even be economically as inept as we would like it to be. Indeed, since only a small minority of all the systems that are designed reach production and stay there for long then it can make sense to use the quickest piece of junk available, at least until proven it will stick.
Easy access to changelogs should be an "easy to access" feature in all databases. Event driven systems aren't rare: the data store needs to be done to tell interested parties that underlying data has changed.
"MongoDB’s default level of write concern was (and remains) acknowledgement by a single node, which means MongoDB may lose data by default.
...Similarly, MongoDB’s default level of read concern allows aborted reads: readers can observe state that is not fully committed, and could be discarded in the future. As the read isolation consistency docs note, “Read uncommitted is the default isolation level”.
We found that due to these weak defaults, MongoDB’s causal sessions did not preserve causal consistency by default: users needed to specify both write and read concern majority (or higher) to actually get causal consistency. MongoDB closed the issue, saying it was working as designed"
What do I use in this situation:
1) I need to store 100,000,000+ json files in a database
2) query the data in these json files
3) json files come from thousands upon thousands of different sources, each with their own drastically different "schema"
4) constantly adding more json files from constantly new sources
5) no time to figure out the schema prior to adding into the database
6) don't care if a json file is lost once in awhile
7) only 1 table, no relational tables needed
8) easy replication and sharding across servers sought after
9) don't actually require json, so long as data can be easily mapped from json to database format and back
10) can self host, no cloud only lock-in
Recommendations?
Depends on what your queries look like, I guess.
Ironically once because mongo was such a pain to work with I dumped the data from it into ES to get the better API, usability and Kibana.
Are companies not supposed to invest money into their product, sales, people etc ?
And why does being listed on the NASDAQ imply being flush with money ?
> And why does being listed on the NASDAQ imply being flush with money ?
It was intended to be a playful reference to MDB's stock price being on a tear right now, not simply being listed on NASDAQ.
Expand the timeline on the graph to "Max", it's at an all time high.
That sounds like a valid redress, or am I missing something ?
Basically, there are a large number of pitfalls that it's very easy to fall into unless you have an encyclopaedic knowledge of the documentation, and you need to ignore some of the words that are used (like "transaction" or "ACID") because they carry connotations that either do not apply or only apply if you do extra work to make it so.
Is the argument that Mongo’s documentation isn’t clear?
"In order to obtain snapshot isolation, users must be careful not only to set the read concern to snapshot for each transaction, but also to set write concern for each transaction to majority. Astonishingly, this applies even to read-only transactions."
"This behavior might be surprising, but to MongoDB’s credit, most of this behavior is clearly laid out in the transactions documentation… MongoDB offers database and collection-level safety settings precisely so users can assume all operations interacting with those databases or collections use those settings; ignoring read and write concern settings when users perform (presumably) safety-critical operations is surprising!"
Yes, I agree as far as the end user is concerned, they are losing data either way.
https://news.ycombinator.com/item?id=23271211
The timing of this is absolutely beautiful.
I don’t have a dog in the Mongo fight. I haven’t done an implementation on top of it in years and probably the next time I do something with “Mongo” it will probably be AWS’s Document DB with Mongo support. That’s based on AWS’s own code and storage tier and doesn’t have the same characteristics as Mongo proper.
> the newer MongoDB 4.2.6 has more problems including “retrocausal transactions” where a transaction reverses order so that a read can see the result of a future write.
If you ask me, if there's something worse than Mongo it's Document DB.
And yes I know most of the data modeling tricks around using GSI’s and LSI’s.
In Mongo's defense, the defaults are similar to what you would likely have with a replicated MySQL/Postgres cluster (single node accepting writes with slaves replicating from there; no concept of write concern). My assumption here is that he is assuming the primary dies before the writes have replicated to the secondaries; that is exactly how master-slave fails too. Maybe there are systems folks can use for having write concern in those databases, but in the companies I've worked for we didn't have them and we definitely didn't have automated failovers
IMHO it perfectly describes the hype-reality disconnect at the early days of MongoDB. Yeah it was that bad.
Mongo has improved since, the hype has toned down and the NoSQL space is more crowded these days.
then a year or two later they admitted that their data model mostly fitted the relational model, and that they spent a lot of time basically reimplementing relational integrity in application code, in ruby.
yeah, diaspora has never been fast. I'm not sure they can blame it on mongodb though.
Reality: Three app rewrites later plus another application written talking to the same DB, and the database is still the same.
I'm glad my gut instinct was correct and that it really wasn't worth the hype. It reminds me of Ruby on Rails.
Regardless of technical acumen, I believe RoR doesn't deserve to be compared to Mongo for one reason: the RoR developers never tried to gaslight their users into thinking they're the reason everything broke; they never said only "if you know what you're doing" can you avoid these hidden pitfalls.
If you set w: majority and r: linearizable/snapshot, both on collection, client and on transactions. Plus assuming you accept snapshot over Isolation. How bad are those remaining cases in reality and how do these issues compare to other databases? The final "read your future writes" error looks quite scary and does not seem to be caused by configuration error, same with "duplicate effects".
- Dwight Merriman, former CEO, and "one of the original authors of MongoDB" [1]
A word to the wise suffices. Sometimes the word in question is implied by other words.
For those who get this oblique post, note that throwing the above bon mot in an interview session for a "distributed systems engineer" and asking for an opinion is a excellent way to differentiate between Peter Principle and Principal Engineer.
[1]: https://web.archive.org/web/20100903213540/http://blog.mongo...
[1] https://community.ui.com/questions/MongoDB-corrupt-after-eve...
https://jepsen.io/analyses/mongodb-4.2.6
... and the corresponding HN thread here:
Data point: I have been running my production system (a fairly complex SaaS) on RethinkDB for the last 4 years.
From my point of view, RethinkDB is not regularly developed and improved. There is progress, but it's slow. Which is a pity, because it's a really good database, and one that tries really hard to be correct above all else.
The only other correct distributed database with strict serializable guarantees that I know of is FoundationDB, which nowhere near as easy to use as RethinkDB is (but it's somewhat easier with their document layer, which pretends to be MongoDB, just done right).
https://news.ycombinator.com/item?id=23253870
(not Mongo obviously)
To repeat my (non)answer:
There is no way to recommend NoSQL database without knowing what you need it for because NoSQL databases are highly specialized systems. If you need general-purpose database use an SQL one.
It's kind of a weird question, now that I think about it. Why would anyone seek out a database based on what it doesn't have?
If you're starting from just "I need to store some data" I'd look to e.g. Riak or Cassandra before looking to an SQL database.
You are never starting from "I need to store some data" you're always going to start from "I need to store and read some data" otherwise /dev/null would work if you are not going to read the data back.
the problem with cassandra and riak is precisely the read aspect of the problem which quickly degrades the performance of those systems.
I've used both cassandra and postgresql at scales most companies never reach. cassandra I'd only touch for immutable time series data and only if that information was large enough to not fix on a single server and i didn't care about consistency. everything else is a SQL rdbms.
The much-vaunted consistency comes at a significant cost: index updates block writes, and more insidiously, it's very easy to be surprised by a deadlock or a stale transaction with a long-running query. I've seen an SQL database stop committing any new writes because someone ran a seemingly innocuous query 23 days ago. And a lot of the time - including every web use case I've seen - you can't actually make any real use of those consistency guarantees.
Writing either a transformation pipeline that serves the same function as a secondary index, or a deliberate map-reduce style aggregation, takes more up-front effort. But it means you understand what's actually going on a lot more clearly and are much less likely to hit that kind of unpleasant surprise.
And the recent change to a restrictive license is worrisome as well. I have been thinking of forking 3.4 and make it back to “true” open source and awesome performance. (If any C++ devs want to help out, reach out to me! username @gmail.com)
Mongo should never be a first choice, but a last choice for edge cases.
* Extract the attributes you're interested in into their own columns, index these. With the extraction happening outside the database, this is the most flexible option.
* Similar to above, use a trigger to automatically extract these attributes.
* Also similar to above, used a generated column[0] to automatically extract these attributes.
* Create an index on the expression[1] you use to extract the attributes.
My use a JSON in PostgreSQL tends towards the first option. This works well enough for cases where documents are ingested and queried, but not updated. The last three options are automatic - add/change the JSON document and the extracted/indexed values are automatically updated.
[0] https://www.postgresql.org/docs/12/ddl-generated-columns.htm...
[1] https://www.postgresql.org/docs/12/indexes-expressional.html
There is always a trade off. If the column is important enough, then you are right, it should stand on its own, but then you lose the json flexibility. I personally almost always only use jsonb if I know I only care about that overall object as a whole, and rarely need to poke around to find an exact value. As a the grandparent comment mentions, if you do need a particular value, then it might be slower if your JSON records are too different (if you think about it, how can you calculate selectivity stats on a value if you have no idea how wide or different JSON records are?).
There are ways around the statistics issue in some cases, e.g. defining a functional index on a jsonb property will collect proper statistics.
Create a functional index.
If you have pid,{name:’val’, others...}. And an index of name with a million John and one Jane. Good luck getting fast results.
Fast results: https://dbfiddle.uk/?rdbms=postgres_10&fiddle=dd9370966b1528....
Postgres does actually keep statistics on json columns, but if you've got a functional index on the table and the query uses it then it doesn't matter if there is one "jane" and a million "johns". You're looking up a key in a btree index.
> FROM pg_stats
> WHERE tablename = 'json_test'
> AND attname = 'json_column';
> {"{\"name\": \"john\"}"}
Hmm. Looks like it does though. Not that it makes a damn bit of difference because if you haven't got a functional index (i.e the stats are next to useless) then you're doing a full table scan, and in that case it sounds like you “expect full table scans to always be fast” :)
And sure, the statistics don't help with the query planner, unless you've got a computed column, but again see "I expect full table scans to always be fast" and re-consider the statement "postgres doesn't keep statistics on json columns" given the fact that it actually does, just like any other column.
Read up indeed!
Yes, that's the thing, it's just a field type. It's not really that different than dumping your JSON in a TEXT column. MongoDB is fun because it's truly JSON - BSON - so you don't have to run migrations, you can store complex documents, and have a more object oriented way of storing your data than SQL.
I'll grant that Postgres probably isn't as much fun as Mongo, what with all its tiresome insistence on consistency and reliability. I would, however, argue that quantity of available fun isn't really a figure of merit here.
Read the docs, you can do a lot of fancy JSON stuff in plain ol' Postgres. It's really powerful and guarantees your data.
That was true of the initial "JSON" type support.
It is very much not true of the "JSONB" type, which was added in 2014 as part of Postgres 9.4. JSONB uses a binary serialization that supports efficiently selecting into JSON documents, putting regular BTREE indexes on specific fields inside the documents, or even putting Elasticsearch-like inverted indexes on complete JSON documents.
if it was just performance than read only replica's in each region would solve most of the issues.
for the legal case generally I just end up with a separate postgresql DB in each region and during login the region is determined by user/company.
usually ends up being like 1k LOCs total.
Please do; someone needs to take that first step, and then many more could potentially contribute.
It's a nice goal but there's likely not much of a commercial market for it, if that's your roadmap.
This corruption is brought on by the stock market.
Have a look also at Shopify. They go and tack on 2% fees when customers use Google Pay or Apple Pay to checkout with. They recently announced that FB would be pulling ecom sales within app, and yet Shopify plans to charge 2% on top of FB fees. That's what I could gather despite the pricing being rather opaque.
Is this a step forward or backwards? Charging 2% / transaction for modern Internet protocols running on cheap hardware across a public network?
</rant>
https://hackingdistributed.com/2013/01/29/mongo-ft/
MongoDB: Broken By Design
And most of those listed in the blog were fixed many years before 2013.
I hope this is a joke.
Well, the warrior has lower upkeep costs. Keep that in mind.
The only thing similar about the two is that they both store data and have the letter D in their name. Otherwise they are completely different, Cassandra being a BigTable style database and MongoDB being a document one.
Really? Which job do you belive needs a "maybe store some of this data, sometimes" kind of database?
For example, climate data gathered from hundreds of thousands of devices every minute can very much survive some data to be lost. Or some astronomical observations data.
I wouldn't choose mongoDB for it, though.