I find it sometimes difficult to link back to Hacker News because unlike reddit, there's no easy API I can use to map links to discussions.
1,291 karma · joined August 16, 2007
I find it sometimes difficult to link back to Hacker News because unlike reddit, there's no easy API I can use to map links to discussions.
I think an appropriate analogy is judging an architect on the hammers he has improved instead of the buildings he has designed.
It's good that it's so easy to figure out how to use, because I touch it less than once a year, so I've completely forgotten about it. That's great software.
A coworker found that calling out to grep can be faster than using Python's regular expressions for large amounts of text processing. My first reaction was to cringe, but it was indeed much faster for this particular problem; I think I was the bad programmer for reflexively cringing instead of saying "nice performance hack".
Incanter is a "Clojure-based, R-like statistical computing and graphics environment for the JVM". Ready to replace R? I don't know, R has a lot of libraries. Worth a look? Definitely.
Emacs.font: DejaVu Sans Mono-12 Emacs.background: AntiqueWhite
You'll need emacs from CVS, but it is a huge improvement over previous versions if only for the better font support.
The tricky part is macros and inline functions. Any time you change a macro or a function you want to inline (CL compilers can't do that automatically because of the language spec), you need to recompile all the functions that use that macro/inlined function. ASDF has some dependency-tracking functionality, but I haven't used it.
The big advantage of changing the storage mechanism from row-based to column-based is that queries over large amounts of mostly-static data go much faster and take up much less space. MySQL, Postgres, and Oracle are all more or less row-based - each row or set of rows is stored whole - while in a column-based store, each column is stored individually, and the rows are held together in some fashion.
Say you have a lot of columns and few tables, and that all your normal queries heavily use a few columns - if you're doing marketing maybe that's zip code and age range, or if you're doing inventory prediction maybe it's item id and sales date. Pre-sort those columns. If you want to get addresses or sales records of everyone in a certain zip code of a certain age - say, the 20-somethings that bought at the local Wal-Mart - that's now very quick, because you can sort a table first on the zip code column, then on the age column, and then on whatever else. It's like addressing an array, as opposed to processing each row and testing a few columns.
That's not a big win on your PHP/MySQL website, but that's a huge win in your 10TB data warehouse.
Once you sort, you can do really simple compression like run-length encoding to get the size down. Maybe you have a million people per zip code, but if you've sorted on zip code, you only need to say "entries 1 through 1 million in the zip code column have value XXXXX". You don't get as good compression on the column used for the next-most-important sort, but it could still be pretty good.
Vertica also does some cool things with replication and redundancy.
Is column-oriented revolutionary? C-Store has been out for a few years, and there are a few commercial and free software implementations out there, and it's in the textbooks, so no.