Hacker News Reader

I don't know what kind of project it is - but have you actually considered just using Postgres? Its not hip and takes a bit to setup, but after you are done, you have reliable workhorse with all the stuff you need (access control, schemas, etc. pp.) It has a vast amount of documentation and reading about its advanced features never gets old. There is a lot of talent around that knows how to use it. If you still consider Ruby but don't like ActiveRecord magic, don't forget to have a look at Sequel.

I would recommend against DataMapper. It is nicely constructed, but has some weird issues. Also, DataMapper 1 is going to be replaced by DM 2, which is a completely different beast (but better, hopefully).

Don't be fooled by ease of setup of the DB too much. It should be rather straight-forward, but even if a DB can just be downloaded and ran to dabble with it, always remember that this just shifts the moment at which you really have to dig into the details. You don't want to do that 2 days before going to production. Be on the lookout for red flags, though: unreasonable and weird configurations that have to be flipped for no understandable reasons at all before going into production. That shows that the project is not maintained very well or the developers have lost track. Also try to figure out how long it takes you to solve a new problem with the database, with documentation reading and all. Pick the one that you can wrap your head around best.

Easy of setup matters. Because unless you are using a hosted database then you will be managing and maintaing that database day to day. Which is why I would never recommend PostgreSQL for startups. It is far, far too convoluted for many of the basic tasks you will be doing everyday.

That's why everyone chooses MySQL as a relational database. It is easy to setup, easy to maintain and EVERY problem has been solved and searchable online.

Wait... what?! I don't see any major differences between MySQL and PostgreSQL in terms of setup/maintenance. They have slightly different authorization styles, but Postgres can be configured to behave just like mysql.

Ignoring for a moment that postgres was my example, but not my point: choose what you can wrap you head around. I used postgres successfully in multiple projects an no one ever had a big problem with it. I've seen MySQL setups burning sky-high because of some Gotchas the team ran into. That doesn't make either of those bad.

The underlying issue is that (except for the most trivial cases), investing time in properly operating and using your database is a huge gain that many young companies ignore. 6 Month later, they have a burning datastore at hand that they don't know how to fix.

Exactly ALL databases will have issues. Which is why I suggested MySQL. Every issue has either been solved online or worked around in Percona, Facebook or Twitter's implementation.

There are huge benefits that come from being the most widely deployed database.

There's no good reason to use MySQL over PostGres these days.

Setting up Sequel and Postgres is really easy. sudo apt-get install postgresql, gem install sequel... and I think that's about it.

You must be joking if you think that count as 'setting up' a database also I just set up 4 servers:

apt-get install mysql-server redis riak-server mongo-10gen-server

I didn't say its hard. But setting up all users properly etc. (which you definitely should get comfortable with _before_ you have your whole dev environment running under 'postgres') takes a few minutes. It is a pretty unsurprising process though, which is why I wrote about the red flags: it doesn't raise any.

I have a FreeBSD box in my closet (I felt like playing with FreeBSD this weekend) currently running Postgres 9 and PostGIS which is how my current stuff is working.

It's actually pretty rad to have every single location I've ever checked-into via Facebook stored in this database. I can say "find all the spots 2 miles from my house" and it's shockingly quick.

I have PostgreSQL on my mac. It's easy to install and use.

PostGIS was the killer. Version 2.0 (homebrew installs postgis2) does not play nice with Django locally (on my mac) and the ticket has been open for a year or something to fix it. That's a huge red flag for me right there. https://code.djangoproject.com/ticket/16455

I guess I could install 1.5 manually.

My problem with trusting Ruby/Rails is that I've only been seriously developing in it for a month.

Okay, but thats more a Django problem then a PostGIS problem.

Lets put it like that: feel free to try some new stuff, but don't start getting all crazy with it. So, if you tried Rails and weren't that convinced by it, maybe default to something that you know, even if it isn't a love relationship. Learning is a great thing, but do it piece by piece, especially if your goal is getting work done.

How quick exactly?

I've been testing ElasticSearch's geo location search and most queries take 50-150ms.

Standard term searches and filters will take like 5ms, with id GETs taking mere fractions of that still.

Well, compared to your numbers, I guess not that quick. Django is responding in 0.28 seconds. That is going over wifi to my server in a closet though. And i'm downstairs kinda far from the router.

Screenshot: http://wsld.me/J5zu

If I hit it repeatedly for a while it comes back in as quick as 0.14, but usually not that fast.

ping to server:

    michael at Achilles in ~
    ○ ping -c 5 apollo 
    PING apollo (192.168.1.130): 56 data bytes
    64 bytes from 192.168.1.130: icmp_seq=0 ttl=64 time=9.497 ms
    64 bytes from 192.168.1.130: icmp_seq=1 ttl=64 time=83.113 ms
    64 bytes from 192.168.1.130: icmp_seq=2 ttl=64 time=7.335 ms
    64 bytes from 192.168.1.130: icmp_seq=3 ttl=64 time=6.998 ms
    64 bytes from 192.168.1.130: icmp_seq=4 ttl=64 time=6.393 ms

    --- apollo ping statistics ---
    5 packets transmitted, 5 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 6.393/22.667/83.113/30.241 ms

I'd call it good enough, I was of the understanding that SQL database geo plugins were usually miserable. I'll add a mental exception for Postgres.

I used PostGIS in proper GIS scenarios and it was faring quite well. The advantage of PostGIS is that it supports all the index types you need directly. And been around awhile, in a good sense.

If you are thinking of them like they are interchangeable, you need to stop, use a SQL database and quit messing. Seriously, they are not at all the same category of thing.

Mongo is an update-in-place structure store with powerful atomic operations. Mongo will not grow storage if you are just changing the value of existing fields.

Couch is a compare-and-swap document snapshot store with multi-master replication. In replicated mode it's eventually-consistent with manual conflict resolution.

Riak is an indexable, eventually consistent key-value store with tuneable tradeoffs for speed versus safety, and no central point of failure.

Cassandra is a horizontally scalable, eventually consistent key-value column hybrid designed for huge data.

And so on... they are not "databases", they are not something where you replace one with the other, they are specific tools for specific tasks, usually much more so than unsexy but adaptable SQL.

IMO you're worrying about the wrong thing right now. Build your API and application without any backend persistence. Make sure your API works. This will result in a nice data API for your backend that you can then plug into various tools.

Use this to test and play with and empirically figure out what the best persistence layer is for your app. There's no possible way you know at the beginning of the app what persistence is best for your app, so don't make that decision yet.

I agree with your approach. I feel though that the angle to take would be something like mongo... where I do not need to worry about a schema/migrations etc... and rather focus on the API logic itself.

I'm a big fan of relational databases. The problem with them is the high upfront cost of developing a new product: maintaining your schemas, updating data types, getting bogged down in data mapping code and worrying about what the most efficient SQL implementation is going to be, etc.

I like the fact I can whip something up quickly with MongoDB, get everything working and then once my schemas and services operating thereupon have stabilised then I can start looking at swapping my implementations out for a SQL-based back-end if I think it's worthwhile.

I strongly recommend Sequel for Ruby:

http://sequel.rubyforge.org

It provides an ActiveModel-compliant interface, so it works fine with Rails. It also has a lot less magic (associations don't use proxy objects, for example) and has a lot of features that ActiveRecord is sorely missing (such as complete support for composite primary keys). I don't know what your other problems with ActiveRecord are, but Sequel is definitely worth a look. I'm very happy with it.

Tastypie developer checking in. What's don't you like about it? Personally I like how it separates resources from ModelResources and lays out your API (that's why I got involved with the project in the first place).

You are thinking too hard. Just pick one, you can always change later.

All of the options you have presented are "Very Good" - they do so many things right. To steal a pithy quote from my co-worker: "Its a balloon squeeze."

This decision will not determine the success or failure of your business, move along.

I pick Rails & Ruby.

Just a quick note -- if you're working in python, I recommend having a look at zodb[1]. Then you won't need an ORM, because it is an actual object database. The main caveat with zope and zodb is that the move to python3/pypy is very slow, and not highly prioritized.

Another, real world, object database is gemstone -- and can be used with smalltalk or ruby (among other things)[2].

[1] See eg:

http://www.ibm.com/developerworks/aix/library/au-zodb/

and/or:

http://www.zodb.org/zodbbook/installing.html

[2] http://maglev.github.com/

and

http://seaside.gemstone.com/

If the main thing you like about Django is ORM yet are considering a DB layer that doesn't require an ORM (like mongo) - have you looked at Pyramid (previously Pylons)?

Besides giving you only the tools you want, it works very well with a driver like PyMongo. If you're starting out small you can also get free developer hosting with MongoHQ to get you up and running quickly.

Oh and you get to keep doing things in Python which is very nice. I've also had no problems with MongoDB for my web needs and I've worked at a few places that use it without the scary data corruption stories told to developers in order to make them keep drinking their Java.

If you have no basis upon which to decide what database to use, you go with the one with the most domain expertise floating around. This means you use the most popular one, or the one that your tech people have used before. Conceptualizers are the last people on the decision list here, sorry, and there's nothing wrong with ActiveRecord outside of academe.

tl;dr: Reflect upon your assumptions before making a decision more complicated than it might need to be.

how about Flask and SQLAlchemy?

I'm surprised this is buried by people suggesting Rails or Postgres or what have you. SQLAlchemy as good a SQL ORM as there is, and it isn't married to a framework, or a SQL backend.

If you like Python and want to use a SQL backend, SQLAlchemy is a good place to start.

he says he loves the Django ORM. This is typically predictive of a poor response to SQLAlchemy's model :)

I've looked at SQL Alchemy and think it's pretty radical actually. I think that this evening/weekend I am actually going to begin doing this in flask+sqlalchemy. It was a strong contender from the beginning.

Here is my first flask app ever, https://github.com/whalesalad/arbesko-files

No authentication or session management or anything above basic request/response, but it was fun and serves it's purpose inside of Arbesko well.

SQLAlchemy is great, but you cannot say that it is simple :)

I sincerely advice against using node. (I'll get to that in a bit). For the DB, I personally like CouchDB. It's easy to get started with, and it lends itself very well to horizontal scaling.

As for why node is a bad choice: you can think of 2 categories of languages:

1) Those which allow you to get things done quickly but in which large code bases are hard to maintain because you need to carry too much information in your head, and they possibly don't scale very well because they tend to run in an interpreter (python, ruby, lisp).

2) Those which require some amount of boilerplate to get simple things done, but provide structure and modularity so it's easier to maintain large code bases, and they possibly produce high-performance code because they tend to be compiled or almost-compiled (C#, Java, Go).

There's a trade off between "getting up to speed quickly" and "maintainable performant code".

Node is bad because it fails at getting you up to speed quickly AND it doesn't make your code maintainable at large scale. This is because async-style programming is not natural to how we think.

The "performance" should not be taken as a plus for Node because you can get it in other languages/platforms (Go, for example) without sacrificing maintainability.

The node debugging experience is quite nice. Unfortunately, I found myself having to use it.

The issues with carrying around information in your head is true IFF you have a large monolithic app. SOA, which is advocated with compiled languages as well, helps to avoid this issue because each service is limited in scope. Then again, SOA itself is counter to "hello world" latency...

I think this is important because if you find yourself dealing with a (example) rails app that is experiencing cognitive burden overload issues, splitting it out into a SOA can be a nice transition (eventually replacing parts with other languages if it is required.)

Also, if you care about performance and quick execution and want a scripting environment, luvit is much faster than node, much lower memory usage and IIRC you can use lua continuations instead of async style.

IMHO, node.js as an edge server providing connection handling and templating with backend services giving it JSON seems like a reasonable 3-tier architecture, especially because your edge tier can then be maintained by people with front-end experience.

What's SOA? And how does it help with cognitive overload issues?

Service-Oriented Architecture. It helps because each service is self-contained, and only communicates with other services through defined interfaces and known boundaries. Because they are self contained, you then only have to have the mental overhead of the complexity of the service you are currently editing (so long as you do not break any of its contracts.)

IE: Encapsulation and isolation, which is something that "scripting" languages generally stink at enforcing (on purpose!)

Besides shell and perl (which really do excel at a certain kind of task), calling a language a 'scripting language' is usually not much more than a dismissive insult by someone with a heavy investment in something like Java, which seems more serious mostly because it involves a lot of self-important boilerplate and has high levels of adoption by big old companies where you have to wear a tie.

no insult. i am deeply invest in ruby and js. calling a scripting language in my book means no a priori verification of the code before beginning to run it. Of course this is a spectrum from Coq to Ruby..

> IMHO, node.js as an edge server providing connection handling and templating with backend services giving it JSON seems like a reasonable 3-tier architecture, especially because your edge tier can then be maintained by people with front-end experience.

I have yet to try it, but this seems like a great approach. Manage most of the computing intensive tasks in whatever environment you like/need. Use Node to talk to APIs and render templates. With a well developed client framework you can even make only the first render in Node and the following in the client talking to the same APIs.

This leaves the most complicated back-end problems in the hands of developers that like and understand those problems and creates a friendly environment for the front-end developer who usually focuses on building features. The front-end developer even has already some experience with async code because of events in the browser, so he'll probably get up to speed quickly with Node.

Completely agree with the CouchDB choice. It has really elegant and unique features but combined with rock solid durability. You can configure it to fsync after every update it does as much as it possibly can together with the OS to make sure you data reaches the disk before returning a result. That is very important if you value your data.

As for other features, I like the REST-ful interface, a continuous changes feed, master to master replication, and a usable interface to see and manage data -- Futon. I haven't yet found a product that comes with all those features.

Another thing I consider a great feature of CouchDB is the cached map-reduce views. It makes it easy to think and reason about indexing.

I've seen this situation quite a bit with traditional SQL databases: somehow you end up with a giant complicated query, and it's becoming a bottle-neck and it's having a terrible impact on performance. Somehow you need to untangle the mess and figure out where the problem is. Sometimes you find out the query is just poorly written. Other times the problem is fixed by simply adding indexes to some table.

This is a terrible situation I don't want to run into.

I have a feeling MongoDB might also suffer from a smilar problem, because it doesn't have cached map-reduce views; it has a query language.

In CouchDB, views are pretty much indexes, and they're pretty much the only way to do a query. I'd argue that because it's explicit like that, it's easy to reason about.

Yeah, we call it Incremental MapReduce and we _LOVE_ it. I swear, it's one of the biggest "ah ha!" moments whenever I do training for Cloudant's customers.

The basic idea is that the intermediate Map function results are persisted, while most other system (read: hadoop) throws them out.

If you're interested in some of the MapReduce history, where it's going, incremental indexing, etc., then you should read this: http://gigaom.com/cloud/why-the-days-are-numbered-for-hadoop...

Cheers.

Yeah, Couch is seriously underrated. HTTP API, Master-Master, append-only, these are things to love if you cherish your web app's data. I wish Couchbase hadn't abandoned the Apache version, or rather, I wish someone else had stepped in to take their place as "the Couch company" the same way 10gen is for Mongo.

Cloudant is close, but their BigCouch stuff still isn't merged yet as far as I know, so if you go with them you're still using some weird Couch fork instead of the open source version. Which might be more of a psychological problem than an actual one, but it makes me really uneasy about the Couch ecosystem they're the only thing we have that's close to a gold-standard champion.

Yea, it's unbelievable how underrated it is.

I searched Reddit for "couchdb", and there's pretty much no new entry since 2 years ago. I almost thought couch was dead.

It's definitely really hard to find out what's going on with Couch lately. Even checking the bug tracker isn't very helpful for finding out what's going into the next version and when it's releasing.

They lost me when they started to shift "everything" towards mobile a while back, maybe 3 years ago. Do they still focus a lot on mobile?

And what's the deal with CouchDB vs. CouchBase?

Mobile is a focus insofar as the master-master sync working well for mobile apps where the user may be online most of the time, but offline a significant amount as well. I believe TouchDB is where they're focusing exclusively on mobile now, and it's less of a focus with the main branch.

As for CouchDB vs. Couchbase, Couchbase is a fork of CouchDB headed by CouchDB's creators Damien Katz and J. Chris Anderson. Couchbase basically throws a lot of the nice things of CouchDB out the window (such as the HTTP API) in order to integrate Membase feature.

This is worth reading: http://www.couchbase.com/couchdb

To be even more specific while still generalizing, Couchbase is the Membase product with a heavily modified CouchDB swapped in for the storage mechanism (previously SQLite).

As for the leaders, it's a smattering of CouchOne/Couch.io and Membase/Northscale people. Or, as a friend put it, CouchOneBase.io.

Cheers.

Just to clear up one point that was semi-made already, "they" meaning Apache CouchDB isn't shifting anything toward mobile. They're still making the same great product from before.

The mobile stuff that you're referring to is CouchBase's. It's a completely different product and their work should not be confused with Apache CouchDB's, though it often is.

There are all kinds of considerations with respect to node.

But discounting all interpreted languages because they are interpreted is crazy biased. I don't at all accept (and it is not self-evident) that a high-boilerplate compiled language is necessary to have a large codebase or acceptable performance. Unless most of your storage is in memcached, the database is far more likely to be a bottleneck than your interpreter (!)

There are a number of compilers for Lisp.

Between these two issues with your post, I would not be convinced to accept your advice regarding node, either.

I'm aware that lisp can be compiled, that's why I said that dynamic languages "tend to be" interpreted (the implication being, they're not always interpreted).

My point is about comparing dynamic and non-dynamic languages. Dynamic languages tend to have "magic" where complex logic occurs without you explicitly declaring it, at the expense of either a) maintainability or b) performance or c) both

There's a trade off here: varying degrees of magic vs varying degrees of boilerplate.

Boilerplate code can make things very explicit and thus easier to reason about.

Now, NodeJS has the worst of both worlds. The callback mess is not easy to write, not easy to reason about, not easy to maintain, and there's no magic.

That's a lot of sacrifice, it better be for good gain. But exactly do you gain? I don't see any gains. Performance? You can get that with other languages/platforms (like Go for example) without sacrificing code clarity.

This idea that MongoDB is dangerous is just crazy and seems to often come from the same people who endlessly tout PostgreSQL as the world's most perfect database. The fact is that very large companies are using MongoDB in production environments. Yes. MongoDB still has some issues in particular the write locks but the situation is improving pretty rapidly and almost exclusively affects very high end users.

So the answer to your question is simple. Use what matches your data model. MySQL if your data is more relational as it is proven and well known. MongoDB is perfect when you treat it like a document store. For example I fetch one User object (with all the user's data self contained) from the database and do everything client side. So in my case MongoDB works perfect.

But either way. Do whatever gets you up to speed the fastest.

> The fact is that very large companies are using MongoDB in production environments.

Not as a criticism of Mongo but the thing about very large companies is that they can devote a team to just making sure some bit of technology works. Almost anything is suitable for a large production environment if you can afford to have a couple people elbows deep in it tuning and troubleshooting.

> This idea that MongoDB is dangerous is just crazy

The idea that a database was shipping with a mode where it would return after a write without waiting for the result to be written to disk. Or a database where silent data corruption is an easy possibility and yet still called itself a database is what is "crazy"

Firstly that mode is completely valid for many use cases e.g. logging, metrics. Secondly all the drivers handle this behaviour behind the scenes for you. Thirdly it is trivial to set the write concern to specify how many instances of the replica set you want the write to persist to as well.

And silent data corruption is an easy possibility ?

One of the main sticking point for serious mongodb infrastructure is the global lock -- yes, write operations cause the entire mongodb instance (every database, every read operation) to block. This is clearly one of the main issues surrounding high-concurrent mongodb development.

Thankfully with mongodb-2.2 there is now instead a database lock which mitigates the issue.

Personally I feel redis more closely resembles language primitives with simple key-value pairs, hashes, lists (ordered, repeating values allowed), sets (un-ordered, no repeating values), and sorted sets (more interesting).

Redis has no clustering (yet) so is best used when your data is transient and you are able to repopulate.

I have no experience with Redis or MongoDB, but your statement is only true if the storage layer is transient. Some of us use servers that don't go away upon shutdown, backed by a SAN.

Have you tested how long it would take your redis instance to be usable again after it finishes replaying the commitlog (AOF)?