I am working on a survey programming and cross tabulation engine. It's open source hosted here:
http://sourceforge.net/projects/xtccThere are 2 versions of the Survey compiler, one in production use and a newer version in advanced prototype targeted at tablets etc. The production use compiler is targeted at India and the Middle East and anywhere else where you do Market Research using Pen and Paper format. It can generate outputs for IBM SPSS-Quantum (a specialised software for data processing) and SPSS and and my own cross tabulation engine, described below. Our current offering to clients is a potential 70% speedup in data processing using our system for Pen and Paper data entry.
The cross tabulation engine is on similar lines to IBM SPSS-Quantum but removes many limitations and is potentially much faster. I tested it on 1 million records - each record had about 64000 conditions, it was able to process the data in 20 seconds (I tried this after seeing Evan Miller's post on HN and someone asked about how fast his software was on 10 million records). I compile the input programs to simple C++ fixed size arrays and data is stored on disk as a flat file, fixed length per record. I have been experimenting with sse instructions - there is a lot of potential for further speed improvement.
This is a git repository, active branches are:
nc - this is a stable version of the compiler, but uses ncurses and
web-questionnaire-2: this is the new survey compiler
Active GUIs/Framewors we can compile to are :
1. gtk
2. wxWidgets
3. webtoolkit
I have also separated out the runtime environment so that we can compile with emscripten
Using this, we can compile with
1. Dojo, ExtJs
2. DojoMobile - I could not get Sench Touch to work.
web-randomizer: this is a branch for a randomization grammar
rdg: this is a branch for a Random Data generator. This was used to generate the 1 million records for my testing. I was able to get it to go at a speed of about 170 records a second.
If anyone is interested in playing with it, please write to me - my email should be all over the source code. We are also looking to make a business out of this (I have 2 friends and family investors).