Paul Graham inspired the creation of Redis
twitter.com
twitter.com
The idiom is "think _of_", not "think at". :)
(BTW, Antirez, your English is very good; I'm aiming for helpfulness, not nit-picking / criticism!)
-- and given your awesome contribution to the world (Redis), you definitely do think of very good ideas!
For languages like Italian (as well as my own native language), the image of 'thinking at' makes sense as an analogue to 'looking at', which you'd indeed do if the idea were floating by in your vicinity.
I think the closest English analogue is 'think about', which taken extremely literally does also place the thinker _about_ (in the vicinity of, around) the idea.
-Benjamin Hoff, The Tao of Pooh
https://www.mondo2000.com/2018/06/18/the-inspiration-for-hyp...
https://twit.tv/shows/triangulation/episodes/247?autostart=f...
Here's a 34 line implementation I use on a node production system. It writes struct-like JavaScript objects that represents events to disk. When reading it back I do a fold (or .reduce) to build the state.
And yes –– it could be way smarter (writing to memory and disk), but YAGNI has been working out pretty well so far.
class EventStore {
constructor(file) {
this.file = file;
this.cache = null;
}
async appendEvent(event) {
// Purge the cache for entries since we mutated the store.
this.cache = null;
return new Promise((resolve, reject) => {
createWriteStream(this.file, { flags: 'a' })
.on('error', reject)
.on('close', resolve)
.end(`${JSON.stringify(event)}\n`);
});
}
async readEvents() {
if (this.cache !== null) {
return this.cache;
}
try {
const data = await readFile(this.file, 'utf-8');
const lines = data.split('\n').filter(line => line);
const events = lines.map(line => JSON.parse(line));
this.cache = events;
return events;
} catch (error) {
return [];
}
}
}You can try it yourself... mmap a file, memcpy some structs there, do the reverse ... and enjoy!
(Obviously depending on the memory layout of one C compiler on one architecture does not make for portable files. But that was never a design goal of this system.)
On startup, and optionally at regular intervals, it "compacts" the database by reducing all events to a single JSON string.
The README links to an article by @antirez, Redis Persistence Demystified [1]. It's been educational studying how it works.
[0] https://github.com/louischatriot/nedb#persistence
[1] http://oldblog.antirez.com/post/redis-persistence-demystifie...
while true, redis predates node, so your example does not support your use of past tense.
The former is something that has all sorts of knock-on effects: backward IP laws "protecting" ideas, perverse incentives within large corporations with outspoken "idea men" being promoted ahead of doers, non-technical founders with "high-potential ideas" sucking up investment and expecting to execute with technical hires on untested theories.
I'm not saying any of the above applies in this case of course, but the fact is that it is you who built Redis, not pg, nor many others who've had similar ideas, and I think writing the above tweet thread lends undue weight to many of above negative trends in our industry (and also in general in recorded history of IP/invention-credit battles).
Ideas are required for execution; they don't have independent value (idea without execution delivers nothing), but neither does execution (you have tohave something to execute.)
In actual fact, even beyond "inception", most execution requires ongoing iteration and innovation. No final product is solely the result of its inspiration.
Ideas are cheap and plentiful if you have the eye for them, and learning the sources of inspiration other makers had can be informative if you also want to make things.
Attributing inspiration takes nothing away from an artist, nor from the work. I don't event think it meaningfully changes the credibility of the inspiring object.
I could randomly paraphrase and quote lines from the Art of Computer Programming, for instance, and if I had a wide enough audience I'd probably bring about a Great Renaissance in the field of computer science and programming. History might remember as the Greatest Idea Man of all time, but that seems unlikely.
On the spectrum on sh#t hn believes, this one, along with meritocracy has got to go. No one is arguing that ideas have _more_ value than execution. But ideas clearly have value, we entire buildings devoted to them. The internet was built to transport them. Getting exposure to them is deemed critical for the development of our young and the future of the humanity.
(Sure, the internet's communication protocols do transmit "ideas" in numerous ways: mainly in terms of shared experience and evolving iterative conclusions from shared knowledge, but it's hardly the intent of its creation).
Getting exposure to past knowledge and experience is critical to our young, and the future of humanity, but—frankly—the glorified fantasy surrounding some popular histories does more to further the idea of the privileged "celebrated few" with the genius of inspiration, than to instill any sense of the true study, investigation, work put in by those who have achieved great things in the past. (think even old storiea like Archimedes' bath or Edison's bulb or Newton's apple or whichever more modern example— placing individuals on sort of divine pedestals to be blessed with such ideas).
You're calling out meritocracy, but it's exactly this kind of championing of ideas that leads to it. Salvatore has put the long, quiet, sometimes relatively thankless hours into making this a reality, and it could seem to some that pg is being put on a pedestal for being momentarily inspirational: is that not the very thing you're calling out?
> pg is being put on a pedestal being momentarily inspirational
I am certainly not doing that. I don't think what happened wrt pg and antirez even qualifies as an idea or inspiration. Chain of events? Catalyst? Memcached was already being used like a data structure server before Redis came along. Redis did put deep thought into the design which is why it works so well for so many applications.
Ideas in and of themselves have value and are needed. Really good ones take a long time to create and hone. Edison's bulb is a great example of brute force. Newton's Apple always seemed like a creation myth. History goes a lot deeper than pull quotes.
https://news.ycombinator.com/item?id=14754
PS: They're basically the same, really, just adding another candidate to yours.
I’m a bit hazy on the details, but I think it involves calling unexec(). https://lwn.net/Articles/673724/
Edit: just noticed a sibling comment mentioned this too...
Despite switching to a dozen new fad languages since then, programmers have yet to get around to replicating this in any broadly-adopted modern system. If you want to put the effort in to engineering it yourself, of course, you can. But it was nice to not have to engineer it at all.
When greybeards seem cranky, it's because of stuff like that.
It's been tried many times, but you die the death of a million cuts. (Not just a thousand.) After you're shutdown, the world moves on, and then you get restarted. Now you're hardware has changed, your network connections have all changed, your version may have changed so the stored data may be all different, and perhaps surprisingly, worst of all, once something gets corrupted, it's corrupted forever. No "reboot" for you.
It turns out that "reboot" step is inconvenient in the short term, but in the long term, enforced a minimum amount of discipline on programmers to make sure they don't get into an unrecoverable state.
You'll note that if anything, the trend continues in that direction. All the recent operational work lately, in Docker, in things like ansible and chef and puppet, in "serverless", in reproducible builds for binaries, etc. can all be read through the lens of taking things that were previously images of unknown provenance, and ensuring that we can always rebuild them from an initial definition. We're in fact headed even farther away from image-based systems that reload in their previous state.
It's a simple pattern to implement. To make it a bit easier to use repeatedly, I've got a small helper class called JournaledCollection. You pass it serialize+deserialize callbacks for your item type, and it takes care of persistence in event logs.
For a while I was thinking about releasing my helpers as a project called LAUF, short for "Lame-Ass Un-Framework". Then one could say: "Most of my projects are LAUFable, I don't need anything more serious." (Awful dad jokes are a solid reason to publish open source, right?)
Never got around to it though, but if you're interested, I could put together an example.
sjl at alephnull in ~/Desktop
><((°> sbcl
[SBCL] CL-USER> (defparameter *name* "World")
*NAME*
[SBCL] CL-USER> (defun foo () (format t "Hello, ~A~%" *name*))
FOO
[SBCL] CL-USER> (sb-ext:save-lisp-and-die "session.core")
sjl at alephnull in ~/Desktop
><((°> sbcl --core session.core
[SBCL] CL-USER> (foo)
Hello, World
NILThat is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.
Likewise ORMs which allow for higher order types etc. have been around since WebObjects i.e. also decades.
One thing I've learned about this "impedance mismatch" is that it isn't a syntax thing, it's a fundamental difference in the way of viewing the world. The way you store data about the world is different from the way you model that world dynamically, with objects. I find it safer to always split out the "business model" from the storage layer, so that those different views don't interfere - and once you do that, you may as well implement the storage layer in a relational way.
The idea that the ORM forces the storage layer to a particular representation of the business layer is only true if they implement the ActiveRecord pattern, which isn't universal.
ABSTRACT. Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). It provides a means of describing data with its natural structure only—that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other.
E.F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM 13, 6 (June 1970), 377-387.
You propose to reintroduce a problem that they absolutely wanted to get rid off 40 years ago. Just imagine that you first have to figure out how to painstakingly parse serialized Python dictionaries before you can access the data in another program written in e.g. Rust.
It clearly amounts to UNSOLVING a problem that is now SOLVED ALREADY.
Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data.
Exist 2 main reasons for the "impedance mismatch":
- Paradigms. 2 different paradigms will be at odds. Example: Functional and OO. This is ok.
- Limitations: The relational model is absolutely superior and more expressive at manipulate data than OO/Functional. You need A LOT of machinery to recover that power.
This is not ok.
However, this not change the fact that OO is ok.Similar how a KV store is fine, but certainly, a RDBMS store is much more capable.
Most people only see the relational model as is inside a RDBMS. That is how judge the OO for what you can do on Mongo.
RDBMS have some weird and well considered restrictions for their case.
But read a little about the relational model, and none of it depend on SQL or say anything about how is the storage.
P.D: Is important to note for where is MORE powerful. Remember that the creator of Pascal say:
https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures... Algorithms + Data Structures = Programs
You can say OO/Functional lean more to the "Algorithms" side but RM to the "Data Structures". OO/Functional not say much how operate on data, most is a exercise for the reader.
Instead, RM give a clear answer and defined operations for that.
You need to "spice up" things to make the one be more useful to the other part of the equation. RM, too pure, certainly is incredible limited, not even can print "hello world!", but that is just too give a solution about how transform data...
This is why I love Clojure (and a particular style of Javascript). Destructuring and a good library of object/array manipulation functions make an environment well-suited to transforming data structures (which is precisely what I want to do in most programs I write), and I find I do not need an ORM where this type of data-focused programming is supported.
Redis loads everything to memory. And doesn't keep the structure in the file system, only the log, recreating it from log+snapshot.
Does merely recalling a language pattern really count as inspiring ?
Anecdotally, I’ve used plenty of in memory DBs (and written some too) but Redis has been by far my personal favourite.
I guess Smalltalk's images are the ur-example here?
Everybody knows RAM became cheaper cheaper, while mechanical disk can't be made faster, and SSD have reliability limits.
It seems entirely logical to expect databases to work on RAM first and then commit on disk for a large performance improvement.
He was not really listening to what I was saying or my arguments, because a system like redis seems like a more than acceptable compromise.
It still seems a few people are reluctant to an in-memory database.
I think people who are deploying redis are willing to tolerate lower durability guarantees for the extra performance. A lot of the time redis is some kind of cache and there is a way of reconstructing the real data from more durable storage in the case of failure. Or people are willing to lose a second of data or whatever fsync interval people are using.
of course, _now_ they are like "FTW" :)
The best ideas often seem obvious in retrospect.