Ask HN: Advice on a public health project.
Question from a very non-technical person (with a bit of a statistics/biology background).
I'm an epidemiologist with a national public health agency. I'm working on a project to try and predict the emergence of new infectious diseases. Lots of data is reported (online or otherwise) by various groups, but not in a way our team can easily interpret or use.
We'd like to scrape data from these reports, populate a database of events, and produce a dashboard that would display events geographically and allow us to perform/display analyses on risk and context. The hope is that with the right data, timely analysis and a highly visual display, the next "big one" will be apparent in the early stages, while intervention and prevention are still possible.
Examples of similar projects (by groups we do or can collaborate with) can be found here:
healthmap.org
www.glews.net
empres-i.fao.org/empres-i/home
These are nice, but do not go nearly as far as we'd like to in terms of analysis.
As a first step, I'm generating a list of criteria that indicate various risk levels (from various experts and the published literature). Next, I'll need to develop the database, and I have no clue how that should be accomplished (I assume copy-past to excel is sub-optimal).
I was hoping HN might have some guidance on how the parsing/database populating ought to be done. I'm so informatically ignorant that I'm not even sure I'm posing an intelligent question, so please set me straight if I'm being dumb.
Basically, where should I start? What are the correct keywords I should be googling? Do I need to get access to the underlying SQL/XML database from my source websites? How should I structure my database? Do I need to hire a hacker? Are these sensible questions, or should I be asking something else?