I'm not looking at solutions that require writing scripts: the most I'm doing is juggling experimental data to look for trends anyway.
Really something to Excel what latex is to Word.
Thanks!
"One can write simple perl expressions to manipulate entire numerical arrays all at once. For example, using PDL the Perl variable $a can hold a 1024x1024 floating point image, it only takes 4MB of memory to store it and expressions like $a=sqrt($a)+2 manipulate the whole image in a few milliseconds. "
There's also SAS, which I like, but took years of brain-twisting to understand. There's always R/splus as well.
Supposedly Tableau is good, but I've never tried it.
But seriously, like with any programming language, once you get used to the syntax it becomes intuitive. The advantage I see for using spreadsheets is that they are portable. You can send someone an Excel spreadsheet and they can look at it, play with it a little, and send it back.
R or Mathematica can do the calculations (and are more powerful than spreadsheets), but you lose the portability.
For simple stuff that I don't need to share I often use interactive Ruby from the Terminal too.
Still, it really doesn't seem to make much difference (or maybe, I'm not using Excel (actually OpenOffice) right).
Haven't tried Mathematica yet; though it has sounded interesting for a long time.
I actually don't use SQL for querying anything (as I said, I'm more looking for seeing how much values deviate from the mean etc), but this looks cool.
One question in relation to your project is why not simply use awk, filter data, pipe to a sqlite db and then query with "real" SQL? I'm curious to know why you choose to write a parser for SQL rather than a lint-thing for CSV..
Also writing the sql "parser" was entirely trivial in that I just match keywords then split it into a hash and build the command accordingly. I dug that approach because it means the sql can be in any order, commas are white space and I can easily extend the syntax i.e. "fields, separator, columns (for setting widths)" etc. without much work.