I'd also like to be able to retrieve the key-values for a segment of time in the past.
Does anything like this already exist?
I suspect this is what you are asking for. This page has links to a number of time series database systems.
The oil industry uses them to record data points coming from hundreds of instruments on equipment on their oil rigs (pump pressure, engine temperature, pipe rate of flow, motor RPM, etc). The oil companies that operate them are rich, and there's big money at stake to optimise the flow of oil; not to mention lives at stake - Piper Alpha accident in 1988 cost 167 lives and £1.7 billion.
The one that I saw used was OsiSoft Pi System http://www.osisoft.com/Products/PI%20System/
About two years ago I implemented a database app (relational, Mysql) where tracking versions [of certain tables'] was a requirement. I solved it by adding a ValidFrom timestamp to the table, and now every row has an associated validity period, eg. the customer's address was X between T0 and T1, Y between T1 and T2. Unfortunately this is not some kind of convoluted backup solution, as the application logic sometimes needs to know these values at some prior time (it's a billing system).
Roughly, the basic problem is that simple queries or enforcing consistency requirements are now very complicated, since tables have to be JOINed and WHEREd on the validity periods. People (other than me) are unable to modify the data per command like SQL, imagine 3 normalized tables each with validity periods, and you're UPDATEing them but don't want to break time-consistency, it's frustrating. The client programmer basically failed at creating a usable GUI that was able to manage these versions, and I don't blame him (think of a case where you have a versioned table T1, a versioned table T2, and you're connecting them with a versioned has-a table T3). UI-wise it's kind of a nightmare.
Lesson learned: BAD IDEA. Don't do it. Fight with your manager to have the requirements changed. I've been thinking ever since what would have been a good way to do this, and I can't figure out a good way, hence I think this is a stupid requirement. Also, now I actually have a picture of the business process that drove this requirement, and although this isn't a trivial issue, I believe it could be solved by fixing certain business processes that require application logic to know about past data [in this manner].
I've done something similar with NHibernate, where you only need to add a couple of event handlers (maybe 30-40 lines of code + a bit of configuration XML) to set this feature up. I wish I'd heard of it earlier though, since it's really quite useful (though as you said, horrendous to implement manually).
There's a wiki on NHForge about it, though I don't know that it's really the best way to do it: http://nhforge.org/wikis/howtonh/creating-an-audit-log-using...
I've checked out your link, but I'm not sure this would solve my core problem(s). The problem is that even without versioning, a lot of the system is made up of long SQL commands containing subtables and JOINs and such. In several instances I had to optimize the queries. So I wouldn't want to use some ActiveRecord thing or generators. And this is now further complicated by versioning.
Technical issues aside, the fact that everything is versioned makes it hard to reason about the data. Eg. the people in charge of "controling" are always asking me for help. Many times it turns out that there's no problem, the data is consistent, they just got confused by the versions.
Basically, a 3-dimensional Excel spreadsheet.
Or, you could simply set your database up to use a composite key on the table (so the key becomes the key + revision number), though that isn't the best way since composite keys usually incur some performance overhead.