jd_mongodb

If you kept the whole 500GB in spinning disks you can blow through a lot of cost for sure. However as the older data is infrequently accessed you could keep that in an online archive which is much cheaper. That archive can be queried seamlessly with your hot data at the cost of additional latency. Online archive stores data in S3 rather than spinning disks, hence the reduced cost.

You should also be aware that running a two node replica-set means if one node fails and you have set a write-concern of majority (which you must do to ensure you do no lose data) then writes will fail at the client if either node is down. So you are saving money at the cost of durability. It's important you be aware of those tradeoffs.

See https://www.mongodb.com/atlas/online-archive for info on archiving and https://docs.mongodb.com/manual/core/replica-set-write-conce... for more info on how write-concerns are used in a replica-set.

The default for write-concerns on Atlas clusters is w:majority. i.e. each write must have been received by a majority of the members in the cluster before the client recieves acknowledgement.

(I work for MongoDB).

Why are you using AWS SQS but hosting your own MongoDB? i.e. why are you not using MongoDB Atlas?

(I work for MongoDB).

Somebody should break this bad news to Snowflake, SalesForce, Airtable, AWS, Azure, Google, Firestore etc. etc.

Don't feel nervous. There are a lot of MongoDB haters on Hackernews. If you have any issues we can help out at http://mongodb.com/community/forums.

MongoDB has ACID support.

MongoDB hss 26500 customers worldwide (and we are one NoSQL vendor).

These customers include:

Bosch : https://www.mongodb.com/customers/bosch

HSBC : https://diginomica.com/hsbc-moves-65-relational-databases-on...

SEGA Hardlight : https://www.mongodb.com/blog/post/sega-hardlight-migrates-to...

HMRC : https://www.mongodb.com/blog/post/mongodb-microservices-help...

DWP : https://www.mongodb.com/customers/department-for-work-and-pe...

Liberty Mutual : https://www.mongodb.com/blog/post/liberty-mutual-iac-mongodb...

MetLife : https://gigaom.com/2013/05/07/with-300m-earmarked-for-tech-i...

There is a more complete list here : https://www.mongodb.com/who-uses-mongodb

That list is limited is just the customers that are willing to be public references.

Every mature NoSQL vendor has a similar list.

Doh!

The Mongoose github says it's MIT Licensed https://github.com/Automattic/mongoose

MongoDB is a classic example of the innovator's dilemma. In its first incarnation it had no ACID transactions, no joins, no Geo-Spatial indexes, no inverted indexes, no SQL query language etc.

What it did have (which no relational database had at the time) was JSON as a coin of the realm and a distributed database model. Run the clock forward and MongoDB now competes on equal footing with relational databases for all those features but it still has the key competitive advantages of JSON as a native format and a distributed architecture.

Posting old Jepssen analyses is like pointing at old bug reports. Everytime Jepsen finds a bug we fix it lickety-split. I know it's not cool to focus on that fact, but it is a fact. The Jepsen tests are part of the MongoDB test suite so when we fix those problems they stay fixed.

I would love to hear your personal experience of MongoDB as opposed to reposting old Jepsen reports. Perhaps there is something that we can address in 5.1 that is still a problem?

(I work in developer relations at MongoDB)

MongoDB has supported a streaming API since version 3.6

https://docs.mongodb.com/manual/changeStreams/

"I have not seen NoSQL databases replacing important relational databases, I have seen NoSQL databases used for specific business requirements."

Look harder :-) Every single NoSQL vendor has examples of substantial "replace' projects.

MongoDB released the Wired Tiger storage engine in 2015 which replaced the MMAP storage engine. So whatever your read about MMAP is obsolete.

https://www.mongodb.com/presentations/a-technical-introducti...

Feel free to ask questions in the MongoDB community http://mongodb.com/community/forums about your schema design.

That "you must have compromised.." was in relation to the comment by pritambaral not your overall analysis.

There wasn't a lot of information in your previous post. As I pointed out there are a comprehensive set of guidelines for enforcing security. Our defaults make it difficult to accidentally expose your data these days. However if you do add a MongoDB database to a public IP address we strongly encourage you to add a strong password. Better still do not expose your database on the public internet. Put it behind a firewall with auth enabled, secure it with a certificate and only allow access to named IP addresses.

Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-chec...

You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default.

If you want to secure your on-premise MongoDB we publish a checklist here https://docs.mongodb.com/manual/administration/security-chec...

better still use MongoDB Atlas and get our best security practices baked in.

MongoDB 3.6, which was released in November 2016 defaults to listening on localhost only. A user must explicitly configure listening on a public IP address.

https://docs.mongodb.com/manual/release-notes/3.6-compatibil...

AH I see. When we talk about nodes in MongoDB they are the elements of a replica set. That is what confused me. Yes, you are right, to get the equivalent performance of a two node database server you would have to run a sharded cluster with two shards. That is definitely more nodes than the two node design you have outlined for the RAC like design.

I am not familiar with Oracle RAC architecture but casual mention of a SAN fails to recognise the cost and complexity of even the smallest SAN. On the other hand MongoDB can run on dumb disks without a SAN or RAID required (though RAID can help with performance).

With you two node system loss of a node results in a 50% loss in performance whereas with MongoDB loss of a node will result in a secondary taking over the primary role with no loss in performance. You get what you pay for.

Setting up such clusters used to be a PITA until the advent of MongoDB Atlas. Now you can set them up with a few mouse clicks.

As regards shard keys. Yes, they are immutable, but the documents they index are not. So dynamic schema properties are preserved in a sharded cluster.

How do you get four times the demand for resources from two nodes?

The "promised efficiency" of any system is a complex question. The dynamic schema approach of MongoDB makes it trivial to update schema design on the fly so I am confused by "no real easy way to correct once deployed".

BTW : The recommended production setup is three nodes (one primary, two secondaries). This is what you get by default in MongoDB Atlas (you can't go smaller).

(I work for MongoDB)

Love to hear that.

I work for MongoDB. The writer is referring to CosmosDB, the Microsoft clone of MongoDB. You can runa real MongoDB cluster on Azure with MongoDB Atlas. The pricing model for Atlas is per Node size + disk + memory + data transfer. It's generally easier to predict your costs using this model. Users tend to over-provision with this model so we recommend using auto-scaling. This will allow your cluster to scale up or down based on load (price will adjust as well).

There isn't a relationship between the two in terms of API. However the team that designed the Berkeley DB storage engine went on to commercialise it via a company called SleepyCat. That company was acquired by Oracle. The team subsequently left Oracle to found WiredTiger (SleepyCat, WiredTiger, geddit? :-)). WiredTiger was focussed on building a modern database storage engine focussed on high CPU count, high memory footprint servers. In 2014 MongoDB acquired WiredTiger and that is now the main storage engine for MongoDB. Interesting Footnote, Michael Cahill is the primary author of "Serializable Isolation for Snapshot Databases" [0] the paper that introduced Serializable Snapshot Isolation (SSI). SSI is the concurrency control mechanism used in Postgres [1][2]. [0] - https://courses.cs.washington.edu/courses/cse444/08au/544M/R...

[1] - https://wiki.postgresql.org/wiki/SSI

[2] - http://drkp.net/papers/ssi-vldb12.pdf

Postgres is a fantastic relational database but for storage, indexing and querying of JSON documents there is no comparison to MongoDB. MongoDB is built from the ground up as a JSON store (actually a binary encoding of JSON called BSON). I would encourage you to try it out I think you would be surprised by what is in the box.

That is somewhat missing the point of MongoDB. We start with easy to use document storage, querying, indexing, and dynamic schema. We end with horizontal scalability.

I can't reproduce that behaviour on MongoDB 4.2.x using Python and the Shell client.

FYI: Same report update:

2020-05-26: MongoDB identified a bug in the transaction retry mechanism which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8.

If you are aware of anyone who has lost data using MongoDB we would love to hear about it. There have been many examples of constructed scenarios where we a MongoDB cluster can be demonstrated to have lost data and in every case we have fixed those bugs. We take data loss very seriously. If you know of such a data loss instance please make contact.

There aren't "so many horror stories" there are the same small number of ancient horror stories replayed over and over.