Hacker News Reader

Comment by dkhenry | Hacker News Reader

you lost me here

""" Ordered data

Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. ""

it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially )

There are a few other flags that make me wonder about the exact limitations you found, but I will be anticipating your follow up post to see what your fix was since some of those issues are very common.

View on HN

No kidding. Without details, it really sounds like the author is a bit clueless.

He mentions the lack of joins, but doesn't say a word about Mapreduce.

"MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index.

"Too many databases" sounds a little suspicious. Why not add an indexed field to partition records?

Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting?

MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.

You don't "walk an index". It's an index

If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology

Not sure what you're getting at. "Walk" usually means a sequential scan. An index is sorted, so you can binary search.

See other reply. If the docs say MongoDB "walks", it's hilarious to complain that someone obviously didn't read the docs for saying "walk", too.

It looks like they went with HBase for the replacement which means they can scan an index range, keys are in lexicographical order, so it makes it pretty easy to scan over a series of data with a single RPC call.

With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like <reverse_domain>-<epoch> which would allow quick scans over large amounts of data. ie..Scan from <object_id>-1368536860 to <object_id>-1368540450

HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the cell back, but you could also opt to receive N versions back, which is useful for time series use cases.

The last time I looked there weren't many resources explaining how to design NoSQL databases (how to compose your keys, when to avoid normalization, etc). Has this improved?

Well, NoSQL databases is a pretty broad term. Not all NoSQL databases are created alike, for example, MongoDB is a "document orientated database" where as HBase is a "column-oriented store" based on the Google BigTable whitepaper.

As far as I know, key design is not an important aspect with MongoDB but I could be mistaken. HBase has a pretty awesome book (http://www.hbasebook.com/), which has an entire chapter dedicated to key design. Lars (the author) also has a pretty in depth 1 hour video on key design (http://www.youtube.com/watch?v=_HLoH_PgrLk).

HBase is pretty widely used, I've seen 1200+ node clusters running production tables.

Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the entire log. All of these should be in timestamp order. It's a shame that natural order is not insert order for non-capped collections.

It's a good point that some of this can be achieved with indexing, I should have given more details in the blog post.

Right for that example ,not knowing the specifics, I would index on timestamp and then I could sort by timestamp which once indexed should be a relativly fast operation. I could even go one step better and make the _id a construct of { <timestamp>,<loglevel>,<fuzz> } Then Inserting would be done in order and I would get a magic index for free (depending on if I often query by loglevel I might leave it out) This gives locality of timestamps and helps keep "Hot" sections in memory.

Also, mongo has natural ordering which would do what the author wants without sorting.

No mongo natural order is just order on the disk. It is not always in reverse insertion order

Reply on news.ycombinator.com