marcua

That's probably the balanced approach. I'll try to do it if I get some free time.

Thanks! My problem is not being able to find a discussion that already exists. The "post to YC" bookmarklet would create a duplicate submission, so I needed something slightly different.

Thanks! I thought about that, but there might be multiple threads for that URL, so I wanted to let the user know there might be multiple. Should I open multiple tabs?

You avoid locks by serializing transactions at any site. Since you're not waiting on disk (in memory DB) and each partition runs on its own block of memory and has its own cpu/thread, you simply don't let two transactions on the same partition run concurrently.

See http://cs-www.cs.yale.edu/homes/dna/papers/hstore-cc.pdf for cases where you want to run two transactions in the same location concurrently in h-store (the academic precursor to VoltDB).

Depending on 1) The storage engine you're using, and, 2) What other queries you run against this table

You could cluster the table on the 'time' column. That would store the data in the table in the same order as it appears in the index, and make time-based range queries a lot faster.

Whereas HDFS is to be used programatically or through a shell, it appears (though the documentation is sparse) that Ceph was designed to be mountable like most traditional unix FSs. There's the MountableHDFS [1] project for Hadoop, and so these could end up being equivalent interface-wise. At that point it's all up to how they implement create/append/delete/seek/replication semantics, which Hadoop has way more documentation on than Ceph.

The Ceph docs also imply that they have designed it so that it's easy to snapshot directories---I'm not sure whether HDFS has facilities for this.

[1] http://wiki.apache.org/hadoop/MountableHDFS

Thanks for posting! I am one of the folks working on FeedMe. If you'd like to see what we're up to now, head over to http://feedme.csail.mit.edu/ to try it out for yourself! Also, feel free to ask questions.

Glad to help:)

I took notes: http://blog.marcua.net/post/211390022/startup-bootcamp-at-mi... .

Hope these help someone!

The notes for each day have a list of readings, including papers and readings from two books.

The canonical undergrad textbook is "Database Management Systems" by Ramakrishnan and Gehrke. I've often been confused by papers we read and gone to this book for a foundation before jumping back into the paper. There are a _lot_ of chapters I haven't read (namely the first few on ER diagrams, etc.), but the chapters on storage/indexing/query evaluation/transaction management/normalization are good if you read them a few times.

The graduate-level book is called "The Red Book" (http://redbook.cs.berkeley.edu/), and is a great source of foundational papers in the field. You can find most of the papers online, but the introduction gives you a good sense for the history and evolution of databases.

I hope this helps!

In case you're wondering, the paper and technology that this is based on is here: http://research.yahoo.com/node/2304