We aim to allow users to understand the socio-economics of any location, from the population of a country to the crime rate of a postcode. Organisations can then use this data to improve their decision-making, whether it's businesses making strategic sales and marketing decisions, NGOs trying to understand a country, or somebody else entirely.
The two main technical challenges we face are:
* Importing data from many different sources, from APIs to Excel files, and understanding their structure. For instance, given an Excel file, we need to determine the structure of any tables, such as which columns represent a location. We also want to pick up as much metadata as possible, such as a description of the dataset.
* Designing a UI that allows users to find what they're looking for, and then making that UI fast (we're currently up to about 10,000,000 datasets and 200,000,000 data points)
Our technology stack:
* Postgres + PostGIS to store the data we've imported
* Elasticsearch for... search
* Python code to import data from the original source to our database (with a tiny bit of Redis for some basic queueing)
* A thin PHP web application to display the data (probably soon to be replaced by node.js)
Things we like:
* Test-driven development and good test coverage (when it makes sense -- sometimes, it's better to manually test a very thin UI than to keep worrying about brittle UI tests)
* Releasing early and often. We deploy features as they're ready, and try to break work down to small pieces.
* Doing the simplest thing that could possibly work. Rather than planning for every possibility up front, we implement something small but useful to start with, and then iterate and refactor as necessary.
What we're looking for:
We were founded in Cambridge last year, and we're now working towards a product covering public data for the UK.
We're looking for a software developer to join us, initially part-time for £1,000 a month + stock option plan. We're joining Microsoft Ventures in London from February 2013, so you can join us in Cambridge or London, or work remotely.
If you have any questions, feel free to get in touch: contact@stat.io. If you'd like to apply, please send us a CV and a paragraph describing why you're interested.
You can find out more about us on our website: http://www.stat.io/