Hacker News Reader

I like the concept of MongoDB. My main problem (which perhaps is not totally clear from my post, linked in the article) is that there are certain historical conventions about what a database should be. In particular, a database will not have the following default behavior:

- return from a write call silently, when the data wasn't written and will not be.

If you are going to break conventions on a multi-decade tradition, there should be warnings everywhere. Not just the Downloads page, but during installation (which is harder to ignore, as the Download page is not seen by people like me who use package managers).

The 32-bit issue is worse. There's no excuse for not warning people that the server they just started is a crippled data store limited to 2GB.

Again, I like MongoDB. Given my experience with software development, those issues are telltale signs of a product in its infancy. Hence the "10 years" title of my post, which of course is hyperbole. But it will take time for Mongo to be user-friendly enough.

By the way, those who say that a piece of software has no need to be as user-friendly as possible, and you should thoroughly read the manual before even experimenting with it simply live in a different world. The more of a head start the software can give you, the better. The fewer the gotchas, the better. That was the philosophy of IndexTank, and it worked.

I haven't got a 32 bit version of Mongo, but people pointed out in the other thread that 32-bit version of mongod displays a little warning every time you start it saying it can handle only 2GB of data.

Can you confirm this (because I don't have the 32-bit version installed)? It still stinks, especially because it "silently failed after hitting the threshold", but I personally would feel better about 10gen if this little story is true (i.e. they warn you about it not only in downloads page, but every time you run mongod).

This is true, in the Ubuntu packages this is printed to the default log at /var/log/mongodb/mongodb.log. It is also abundantly clear from the documentation. I struggle to understand how one could deploy a new datastore in production without reading the "getting started" level of documentation or looking in the log at some point.

The 2GB 32-bit limit of MongoDB seems like a complete non-issue to me.

  Wed Sep 19 17:29:21 [initandlisten] MongoDB starting : pid=3765 port=27017 dbpath=/var/lib/mongodb 32-bit host=deepthought
  Wed Sep 19 17:29:21 [initandlisten] 
  Wed Sep 19 17:29:21 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
  Wed Sep 19 17:29:21 [initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
  Wed Sep 19 17:29:21 [initandlisten] **       with --journal, the limit is lower
  Wed Sep 19 17:29:21 [initandlisten] 
  Wed Sep 19 17:29:21 [initandlisten] db version v2.2.0, pdfile version 4.5
  Wed Sep 19 17:29:21 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
  Wed Sep 19 17:29:21 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
  Wed Sep 19 17:29:21 [initandlisten] options: { config: "/etc/  mongodb.conf", dbpath: "/var/lib/mongodb", journal: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
  Wed Sep 19 17:29:21 [initandlisten] journal dir=/var/lib/mongodb/journal
  Wed Sep 19 17:29:21 [initandlisten] recover : no journal files present, no recovery needed
  Wed Sep 19 17:29:21 [initandlisten] waiting for connections on port 27017
  Wed Sep 19 17:29:21 [websvr] admin web console waiting for connections on port 28017

Normally people look at logs when things go wrong. Do you look at the log anytime something starts successfully and seems to be working? You must spend lots of time looking at logs.

This is a message that MUST be displayed on the console when you install the server for the first time. It's too important.

Also, you learn a tool before going into production. I never went into "production" with mongodb. All I did was experiment with a toy project. I never needed to look at the log.

Do you look at the log anytime something starts successfully and seems to be working?

When I first install something new that I'm going to rely on - yes.

How do you know that you're going to rely on it if it's new?

I install something new. I kick the tires. It takes a long time until I decide I'm going to rely on it. I'll look at the log at some point, but not necessarily the first time I install something. There are better things to do at that point.

Maybe you should learn something from your failure.

Maybe you should make constructive comments instead of making assumptions about what people learn or don't (or what they know or don't, or what has worked for them or hasn't).

There are some really disrespectful people in this community.

I didn't say you didn't learn anything. But you seem to be making excuses for why you shouldn't know some very basic things about MongoDB. It uses a memory-mapped file. How can that be larger than 2GB in 32-bit system?

It has async writes...this is pretty well documented by 10gen and is also something noted by a lot of tutorials, blog articles etc. You should have known something this basic about a database so important to your business.

None of that would bother me in the slightest if you were not still here defending such basic mistakes and blaming them on 10gen.

Come on, database (or any other system) that just silently fails to add new data?

2GB limit is clearly mentioned in the logs, that's fine, but anyone that sees this would expect that DB would start "screaming" loudly wherever it can (logs, response to the user on EVERY communication with the server, during any select,insert and others) that it reached this limit.

Async writes are async, so there cannot be a response to the user on failure unless you explicitly check(getLastError). This is very well documented behavior.

> There are some really disrespectful people in this community.

Your article begun and ended with sarcastic remarks about the product. Realistically, what kind of response did you expect? The issues described in your article are very real, and very worthy of repeated discussion, but the article itself eschews discussion in favor of pontification, sarcasm and flamebait.

So important to my business? What business are you talking about? You are just confused.

All this time I've been talking about a toy app that I wrote to kick MongoDB's tires. If you don't understand that, there is no point in having a conversation.

I never had or have any plans to use MongoDB for any business.

Sorry, you are right I conflated your article with another one referenced in the submitted post. I feel at least as stupid as I am, I can assure you.

You have been a "technology manager" at Bank of America in North Carolina since 1998. Do you try new technologies such as MongoDB professionally? What are your qualifications to make such a vague statement? More importantly, what is the lesson to be learned?

I made them in other replies, sorry for being so opaque.

The fact that a 32-bit memory-mapped file is limited to 2GB is basic comp-sci. It is also reported by MongoDB everytime it starts. Furthermore it is noted in the 10gen docs in several places.

Async writes...also extremely well documented.

So the failure here is two things: failure to properly research and understand a technology critical to their business. And then failure to take personal responsibility for the first failure, and instead to blame the vendor for not building a tradition DBMS despite the documentation about the stark differences.

I use MongoDB in a side-project if that is important to understand.

You do know he sold search and indexing startup to LinkedIn, including a custom storage backend?

Then this whole post sounds like a non issue. You couldn't be bothered to even look at the log file for your app because it wasn't important enough, then you shouldn't be irate when it doesn't work as expected.

Every time I start something I tail -f the relevant log to make sure it starts correctly. Think of it as isolating a possible failure point as a sanity check.

It's also on the download page.

http://www.mongodb.org/downloads

The issue is not the limit; it's the silent failure. 32-bit Oracle doesn't do that, it tells you it can't extend the tablespace by raising an exception.

I'm not trying to defend Mongo, but the reason it doesn't tell you is because you didn't ask for it. If you care about the data (i.e. it's not a log or something that's not very important), you ought to always use "getLastError" to see if your data was actually stored or not (some drivers, like mongoose (for Node.js), just let you specify a simple flag ("safe:true") that does this automatically).

A shitty default, no doubt. But it can be changed easily. And "most" drivers offer that. And you usually connect to MongoDB using a driver.

http://www.mongodb.org/display/DOCS/getLastError+Command

> If you care about the data (i.e. it's not a log or something that's not very important), you ought to always use "getLastError"

Wait they are promoting this as a database. It is right there in black on yellow "database". I don't know about you but I suspect most people expect databases to try their hardest to protect data. That means also having safe default. As in when I install it and put data in it, by default it should try hardest to make sure that data doesn't get corrupted. If it doesn't, it doesn't deserve to call itself a database.

> A shitty default, no doubt.

This is not a "oopsie" this is a deliberate lie and misinformation in order to produce fast benchmarks.

Shitty defaults are 90% as bad as hardcoded unfixable settings.

Defaults usually don't get changed, convention over configuration, etc.

The default for the Java driver changed recently to the Safe mode.

So it does happen.

> Can you confirm this (because I don't have the 32-bit version installed)? It still stinks, especially because it "silently failed after hitting the threshold", but I personally would feel better about 10gen if this little story is true (i.e. they warn you about it not only in downloads page, but every time you run mongod).

Just as a data point, I installed MongoDB on 32-bit Ubuntu 12.04 from the Ubuntu repo. At no point during install or service startup does it say anything.

Installing from source may be different.

Just installed it on ubuntu 32-bit (apt-get install mongodb). I now have a 32-bit mongodb process running. Minus all of the noise from apt about setting up xulrunner and the 101(!!!) other dependent packages, here's the entirety of the startup messages I see:

    mongodb start/running, process 13660

Any app that has a warning message will fail to warn you if it's output is rerouted to a log file. This sounds more like an issue with the package maintainers not understanding the limitation and thus not putting in a warning message post install.

Yes that is true for me, at least using the 10g binaries for Linux (not the Ubuntu package). You get this warning everytime you start the engine. You'd probably have to look at your logs to see it. It is also referenced in the installation documentation.

Look guys, MongoDB uses a memory-mapped file. How can it be larger than 2GB on a 32-bit system? There should not need to be "warnings everywhere".

I think expecting end-users to know or care about the implementation detail of a memory-mapped file is even sillier than all the other silly expectations floating around this thread.

You are right I would not expect end-users to know this. I would expect someone who writes articles about DBMS to know this though.

What do you expect end-users to know about oracle or any kind of database tuning?

Maybe stupid question, but isn't there mmap64 and mmap2 with which you can map larger files? You are limited to 2 gigs at a time, but that shouldn't be a complete show stopper.

How could it be larger? You could use mmap64.

Confirmed, like others below. Installed on Ubuntu via APT.

~$ sudo service mongodb start

mongodb start/running, process 7680

Why are people running 32-bit OSes in 2012 again?

I run one on my phone daily? Why are you asking a question with an obvious answer of: not everything is 64 bits yet.

Why would I need that when I have <= 4GB RAM?

When comparing e.g. 32bit firefox and 64bit one, the first runs much smoother on my system. So if I didn't have 8GB I would install 32bit system. (not to mention that all binaries take less space)

Both are clearly stated in the documentation. Anyone that puts any DB into production without reading all the documentation beforehand is foolish and has no room to complain. If there are only "experimenting" then at what point will they read the documentation?

I would guess almost every DBA and sysamdin in the world is foolish. I doubt almost nobody has read the entire documentation for PostgreSQL, Oracle or MySQL.

Still they manage to get those databases up an running without any data loss with just some generic knowledge about OS:es and hardware and reading choice parts of the documentation. Strange, huh?

If a default configuration of a _database_ product doesn't try its best to keep the data safe it shouldn't be just clearly stated in the documentation. It should be a flashing red banner on the front page and everytime its service process starts.

> In particular, a database will not have the following default behavior:

> - return from a write call silently, when the data wasn't written and will not be

There isn't some defined set of rules for how a database should operate. This attitude implies that an asynchronous database should never ever exist. If that's the case, how could I ever use a database for HTTP logging? I can't have every single HTTP request block on a database write, that's absurd. HTTP logging is impossible with MySQL or PostgreSQL for exactly this reason.

PostgreSQL allows you to select this behavior on a per-transaction basis using the synchronous_commit variable: the default (which what we are discussing here) is "don't return until the data hits disk", but you can set it as strict as "don't return until the data has not only hit a local disk but has been acknowledged by a standby slave" or as lax as "return immediately: sync my data when you get around to it, it isn't important"; (so, don't claim something is impossible with someone's tool without first looking into it deeply).

You're right, that's my fault. You can set synchronous_commit to off, which makes this possible. However, this is a database wide variable, so you can't use this database server for anything else, presumably, if you set synchronous_commit to off.

Again, no: it is per each and every individual transaction. (edit:) To be very clear, this means that a single HTTP log table in a single database could have some requests marked synchronous_commit (as they are to a resource that you charge for, and for which you need accurate logs), while for others it is not set (as you just want the fastest performance).

MySQL has INSERT DELAYED which is async. Of course, it's not the default - you have to specifically choose to call INSERT DELAYED when you know the data you're inserting isn't life-and-death.

INSERT DELAYED does not work for InnoDB tables and is now deprecated since MySQL 5.6. I am not a MySQL user though so I have no idea if there are any alternatives to INSERT DELAYED.

The reason for the deprecation seems to be the many gotchas with it combined with the doubtful performance gains. It seems to have been an ugly hack.

http://dev.mysql.com/doc/refman/5.6/en/insert-delayed.html

But even a async database should be able to write something to its log after it discarded the write request. From what I understand from the original article, this did not happen.

> This attitude implies that an asynchronous database should never ever exist.

Is a process that send data from socket > /dev/null a database as well then. Why not call that a database too?

> If that's the case, how could I ever use a database for HTTP logging?

In a normal database, you would possibly switch 'durable writes' and possibly 'time expiration' feature in the configuration file from the default OFF to ON.

I think a lot of the "hating" is a side effect of MongoDB being consistently oversold in terms of its capabilities

A large part of that is that 10gen are much better at marketing than they are at tech. For example, they like to pitch against the Oracle database, and for certain use cases, MongoDB is better than Oracle RDBMS. But if you were to take those cases to Oracle, they would say well don't use the database for that, use our other product Coherence (which was Tangosol before they bought them). And Coherence spanks MongoDB in every possible way. You could write a bestselling novel about it, 50 Shards Of Grey.

Another example here http://news.ycombinator.com/item?id=4533760

And if you hate Oracle, there are a bunch of free things that do what MongoDB does a hell of a lot better. Why waste time with its cheesy MapReduce when you could have ICE http://www.zeroc.com/overview.html for example?

"Oversold" is the key word here. And that's a marketing concept. To dig deeper, the problem has and always will be that MongoDB toes the line between a caching product and a database product, yet 10gen markets it simply as a database product.

MongoDB is flexible and fast, but practically speaking it's a hybrid product that defies simple categorization. That means it must be well understood to leverage it. Ultimately that may also be it's Achilles heel, but in the short term, it's extremely frustrating that 10gen can't embrace this fact. Instead they perpetuate marketing that causes the product to be perceived as flawed by their target audience.

Once you've discovered this, you can use it appropriately (either by overriding the default silent failure to use it as for durable persistence, or only using it for caching or as an eventually consistent store) to reap it's flexibility and speed.

(Largely quoted from my comment yesterday, https://news.ycombinator.com/item?id=4566518.)

Compare with the marketing of Redis. Redis makes it obvious that persistence is a trade off you must make as a user.

http://redis.io/topics/persistence

> I think a lot of the "hating" is a side effect of MongoDB being consistently oversold in terms of its capabilities and architecture.

Agreed, though I think its also coupled with just a lack of understanding of what MongoDB does beyond just "its NoSQL!". So many people are used to working with RDBMS. They just seem to make incorrect assumptions about MongoDB based on what they thought were universal rules about databases.

Mongo's a tool. Its not right for all situations and it definitely has some maturing to go still, but its good to use in certain situations, provided you're properly informed about it.

Totally agree.

I would only add a lack of thinking on the part of developers who jump on the NoSQL bandwagon for the buzzword without understanding the implications of not having a relational database.

I'm not a RoR fan but the ActiveRecord model is probably enough for 75% of people to get their projects started without having to dig too much.

I agree. In some regards it is "oversold", but I also think part of that is from a lot of the marketing efforts we've seen with newer databases like Mongo and even Riak.

I am not saying those are bad, but with more established DBs like MySQL and PostgreSQL, you never really saw the same kind of marketing efforts towards developers, startups, etc. It is kind of a newer concept.

...with more established DBs like MySQL and PostgreSQL, you never really saw the same kind of marketing efforts towards developers, startups, etc. It is kind of a newer concept.

That's funny because MySQL used to do exactly the same kind of marketing fifteen years ago: comparing itself to Oracle, "it's a thousand times faster" when it obviously didn't do one thousandth of what a RDBMS does.

History repeats... and has a strange humour sense.

I think the style is different. I blame the brogrammers.

I don't think that is correct. They never compared themselves to Oracle. I spent some time with Monty parts of the team and also tried working with their business side and they never went after the Oracle workloads. Their huge popularity come with the LAMP stack and their ease of use. They went after this along with data warehousing.

Fortunately, it was public. Don't take my word for it.

I was going to say I remember MySQL doing pretty much the same thing back in the 3.x days. Ok, a different style (I don't remember them comparing themselves to Oracle, but I do remember comments like "who needs foreign keys anyway?").

Actually I think that MySQL is essentially the fore-runner of NoSQL. People who have cut their teeth using MySQL as a single-app persistence store or were trying to code to lowest common denominator in order to be portable now see benefits in ditching the idea of a shared database altogether because, well, they never used a shared database.

I like the simplicity. I can fix something that is broken.