Hacker News Reader

Comment by epmaybe | Hacker News Reader

You probably already know, but there are teams with the national library of medicine that work specifically on natural language processing within the scope of pubmed, but also on new ways to get to the publications. I almost interned there for a summer, very fascinating stuff.

View on HN

yep, we're trying to outcompete them. But it's a hard thing. MetaMap: we beat it, cTakes, we can do it faster. But really how many < 80ms concept recognition systems do you need that work on 28M documents. And if you have that: what are you going to do with it. We have, internally, systems that predict dense vectors based on CUI graphs of concepts in text. Great for similarity search: but utterly useless in the end.

Yeah, I run into this problem all the time in the space. You build something really awesome, an either doctors don't care, it doesn't fit the workflow, or it turns out to be minimally useful in practice. We abandoned most NLP efforts and dump tagged terms into algolia for a lot of use cases and it works well enough to fit a prescribing workflow. The docs already know what their searching for, they just need to know its in the catalog and like the autocomplete.

If I might suggest something, maybe forget about targeting older doctors and researchers set in their ways. Go after the youth, the ones just starting their education in research/medicine, you'll find us far more pliable.

I don't think you need to out-complete these systems in processing speed. Even if you can only do 10 documents/hour, that probably can be scaled to handle a few million. Where you need to out-complete cTakes & co (on a large margin) is information extraction quality (to have a viable commercial product). And that is very difficult...

Reply on news.ycombinator.com