Introducing the Mozilla Science Lab
blog.mozilla.org
blog.mozilla.org
To be able to read/write/import/export that to and from desktop machines to the web would aid science so much.
AND it seems like a perfect job for the Mozilla foundation. I have no idea from their (uninformative) article what they're actually doing...
http://www.unidata.ucar.edu/software/netcdf/ http://cdf.gsfc.nasa.gov/
http://download.cell.com/images/journalimages/2211-1247/PIIS... http://download.cell.com/images/journalimages/2211-1247/PIIS... http://download.cell.com/images/journalimages/2211-1247/PIIS...
Most science (that really needs hackers' help) is not the big sexy datasets, but the really basic HTML5/CSS viewable graphs otherwise done in Excel. But it must be done in a scientifically rigorous way (retain units and labels, be able to manipulate those units, and be able to show/hide data between different datasets).
Think a way to download and view a melting point curve dataset without ANY user configuration. Or an enzyme activity curve. Or a drug dosage response curve. VERY basic data, but it must be able to maintain and interpret a label and a unit in addition to the number itself.
In the examples in your final paragraph, surely what is needed there is a new tool, not a new data format?
I think the larger 'problem' (if it really is a problem, and not just hackers wanting to believe they know the One True Way) is that there is no such thing as the typical 'scientist'. Every small sub-discipline has its own needs, its own standards that it has settled on and its own toolchain. There is definitely room for improvement within every one of these small areas, but there's no global problem that needs solving.
Unfortunately, many areas of science aren't funded well enough to support a team of CS majors to work with the scientists, and indeed many scientists might not even be able to identify what aspects of their work could be improved through computing.
This project seems to be aimed at educating scientists to overcome those hurdles.
We don't have funding to hire coders/developers/designers. And even if we did, we'd likely spend it on that new microscope instead.
Much of what we need doesn't require computational power, but requires a good input parser, a reasonable UI, and some basic (though absolutely solid) math. And a way to change a whole host of variables used in the computations without too much effort. I have a set of a few projects that a mildly competent coder could bang out in a relatively short period of time that would undoubtedly progress the whole field in which it was aimed. But that's pretty much like 20 people. 20 grad students - their bosses don't care if they count the data by hand over a week, or have a computer do it in 2 minutes. It's not worth anyone's (unpayable) time. So I code it myself - it's crappy, it's rough, but it gets the job done. But no one will ever be able to reuse that code.
You hackers want to do something good for the world? Set up a hackathon thing for your local biology/chemistry grad students. In a weekends' time you can write a few open-source apps which would rapidly and noticeably affect the pace of cancer and other basic research.
What are the low-hanging fruits I can aim for to get used to the problem domain? What are the relevant pain points that justify some hard thinking and coding?
In the end what I have in mind is basically this: http://biomind.com/AI_Against_Aging.pdf
- An open source GUI for sorting spikes. When you put an electrode into the brain, you often end up recording the "sounds" of multiple neurons firing. To make sense of what's going on, you have to identify which spikes came from which neurons. While there are fully automated algorithms for spike sorting, they aren't widely used because people don't trust them to work on real recorded data. Even if you use them, you probably want to check their results later, and you need a good GUI to do so. Right now, the "state of the art" is Plexon's Offline Sorter (http://www.plexon.com/products/offline-sorter%E2%84%A2), which is slow, occasionally crashes, and costs thousands of dollars per license. It's mostly a GUI, and replicating it would probably not require much domain-specific expertise. The fact that it's closed source actively prevents the development of better partially supervised spike sorting algorithms.
- Better fMRI data analysis tools. See https://news.ycombinator.com/item?id=5239530 for some previous discussion.
- An open source replacement for MATLAB that is familiar to users of that language and encourages modular design and has a JIT that can achieve C-level performance. I am betting that this is going to be Julia (http://julialang.org/), but there's still plenty of work to be done before Julia is ready for people without substantial programming experience.
- Better tools for visualizing high-dimensional data sets. I'm not actually very familiar with what's been done in this domain, but I feel it's something that isn't often attempted because there's little overlap between experienced graphics programmers and scientists. A good way to visualize your data can save you days spent performing statistics.
Regarding fMRI, I take your posting as something to talk about with the neuro people.
My main takeaway from your posting is that better GUIs and visualizations are needed. That's what I found (workflow, UI) before and your posting is another affirmative data point here. Thanks for the links.
Any hints how to talk to biologists / neuroscientists? I still need to know more about their incentives (what they have to accomplish or help them doing so) and constraints (what they can't do or is unimportant) and culture. As usual, publications and funding, but what is seen as high status and what is frowned upon?
I want to be able to take a stack of 100 TIFs, look for a few white spots on a black background - and track their centroids. That's it.
State of the art:
- powerful, but written in many years ago in a government-controlled, expensive version of Fortran http://sun.iwu.edu/~gspaldin/rytrack.html
- Growing, but written in Java (burnt out two batteries, a fan and an HD trying to track 128px x 128px frames...). Also very difficult to modify/extend/customize/export http://fiji.sc/wiki/index.php/TrackMate Oh and did I mention it was written in Java?
On the data analysis side. I want to take a CSV file with a large number (1000s) of such 2D coordinates and pull information out. Average distance between hop. Total distance traveled. Duration of travel. Extraordinarily basic matrix/vector manipulations of any other random flavor of basic computation I, the scientist, can come up with.
Fortunately I'm one of very few biochemists who reads HN. I know how to use Mathematica. I've taught myself Python and Cocoa. But I've never taken a course on CS. I still don't know the real difference between a float and a double.
A double is a float that has twice the memory size to store a number, thus it can represent a real number with greater (doubled) precision. Cf. http://en.wikipedia.org/wiki/Double-precision_floating-point...
Most of my peers either use Excel, blindly learn by rote a series of 'voodoo' commands to input into a very specific program running on a dusty old Win98 machine. Get out a calculator. Or beg their CS/physicist girlfriend to help...
That's another issue. I still have no clue what (outdated) OS/software are used in labs. I've heard so many things (that usually point to older stuff) that I can't say what dominates. Hints?
P.S.: At least they don't use COBOL. :)
you could do a lot worse than spend a few hours browsing the faculty websites in departments related to your interests. Look for assistant or associate professors or grad students w/ informative websites (those will typically be the more senior and more independently motivated grad students) and look for people who post their code.
That will help filter out the people who are likely to care about this issue (ie people who want better code in their field) and who might be working on something you're interested in and could contribute to. Then email them, set up a meeting if possible, and ask them your questions in person :)
But, just to be clear, I wouldn't suggest that you volunteer to help them specifically; they will have no idea if you're reliable, any good, etc., and, like the parent said, this is a pretty low priority for them. But they should be able to help you get started in the right direction, and a lot of professors will feel (and may be) professionally obligated to guide you if you show progress and commitment.
s/George R. R. Martin/Mozilla/g