Comment by anon1253 | Hacker News Reader

Comment by anon1253 | Hacker News Reader

1) As someone who does NLP and AI for Evidence Based medicine [1] I can only agree. I build natural language processing tools, together with the team, for EBM. I have /no/ idea what to do next. Please. Help me.

2) Rant: here's the thing, we have SVMs, LSTM, BiLSTM-CRF, Knowledge Graphs, 28M articles, deploy infrastructure, tests, you know all the cool stuff. We can't sell it. It's PubMed this, PubMed that. There's no breaking that spell. So instead of building a better search engine, we focused on better analytics. Is it cool, sure. Is it as good as all the KoL stuff from LexusNexis, nope. Medicine is /hard/, if you think your app is hard: medicine is /harder/. Am I trying to cover my ass, sure. But seriously, deal with everything being non-standard and proprietary for long enough and you'll feel the same way. 80% of my job is saying "no" to people who aren't going to pay in the first place (Sanofi, Novartis, etc). They don't care. I'm just going to ride this out, throw some blockchain in somewhere where it makes absolutely 0% sense, call it a day. Seriously, NLP/AI for medicine: it's crap 100%. And, I'm a "senior developer" with publications behind my ass on this. I don't know what to do instead of riding the hype train. Seriously: anyone out there that knows what to do with 100M+ medical publications in a 100M+ knowledge graph, be my guest: I'd hire you in a heart beat. But it's hard and there is nothing to be gained AFAIK.

[1]: http://growthevidence.com/joel-kuiper/

Imagine you were given raw data from the large hadron collider and you were asked to find interesting things in the data. What would you do ?

Any serious scientist would see the folly of trying to run only machine learning algorithms to find the Higgs boson. What you would need (in addition to ML algorithms for processing) is a good theory of what you're looking at. This also means just applying computer science methods to the problem isn't going to work, you need to inject theory from physics.

For drug discovery this means you need to actually do biology at some point in order to make progress. ML strategies involve tight loop around trying different models, implementing them, and checking to see if it does better than other models. What's needed is then good experiments to test your models and good models to make novel predictions.

There is no getting away from wet lab experiments in biology if you're going to make any significant advances. The theory just isn't there to do purely computational work.

Exactly, and that uncovers the folly in this: what are you trying to do. Often enough it's very specific, and requires actually going out into the world. Sometimes the literature is already there. But nearly always it requires some form of analytical thought. "I want X because I need to prove Y". There is no AI magic sauce around that. It means that the people interacting with the "user friendly" interface need to know, and need to know how to get there somehow. That's rarely the case. And from a product perspective it makes things very hard: you get bombarded with very specific requests that require time and effort to build UIs and models for, that at the end the of the day only solve that specific question. Instead, it would be better to equip people with the tools needed to solve the meta-question so to speak, but that meta question is much much harder (at least, I've been trying but no success yet). As a funny anecdote: my medical knowledge had exploded. But I see that as a symptom of the broader problem: I know how to do /some/ things because I've gained that domain knowledge, but I can't teach the system yet to solve the "general" problem because a) I don't know what it is b) there is no data to get there c) what even is the UI for that.

It almost sounds like the real solution is teaching the users how to program. Possible with a suitable abstraction to make it easier for them.

Is this a fair way to express your problem?

You're being asked to use AI to replace an entire process all at once, rather than to assist with or replace a step in that process.

Instead of being asked "how do I solve this problem" it would be nice if you were asked "can you replace this thing I have to do all the time?"

Maybe this doesn't apply to your work, but if I were in your position I'd talk to the human that does the human version of the process and break down everything they do to get from A to Z into steps. If you can start by just replacing or augmenting a single step, you're adding value right away and you've also built a part of your eventual A to Z machine, right? For example, your search engine idea is a good direction. Can you automate the process of finding relevant articles? That would help the humans but would also help whatever AI you're working up to, right? Then maybe further in there's some other process involving physics modeling or something that is clear enough in scope to do programmatically.

I do much simpler work, but I find it's much easier to automate processes one step at a time rather than all at once.

Well, that's because physics has a model. Biology has nearly no model at all, so people first look for features at the data, and then verify those empirically.

It' actually the obvious thing to do.

Do you know of any good resources to start automating some wet lab experiments, such as culturing, pipetting, etc, in a cost effective manner?

As someone involved in the field, I agree on the need for a better theory, but disagree with your claim that "there's no getting away from the wet lab". The traditional wet lab absolutely has to be gotten away from, because there are huge issues with transferring knowledge gained on organism X onto organism Y.

Reason? Lack of theory. We don't understand enough about what causes something to work in one case, but not another.

There's just no getting away from theory, in the end.

Perfect answer. This is exactly the case, from Swanson's seminal text mining in biology paper on fish oil in the 80's with Arrow, to modern tools like Leach's Hanalyzer: You need to focus on a very narrow target application or have a clear hypothesis you wish to prove. And, wetlab work, human intervention, and tons of labor are major ingredients of those (rare) examples that go beyond information retrieval or very targeted information extraction. (Also note how none of these guys was able to scale that work and produce a whole chain of findings... ;-))

I do AI-related analysis inside the pharma world, on the image/data side. While I employ numerous traditional analytical methods to extract info from images, few are as sexy as deep learning (though that approach is rising in day-to-day utility in bio-imaging) or most ML/PR approaches. How much ROI benefit is there to devising a higher level of performance or adaptation? And do we have enough data to train from?

Often it's more productive to simply throw in a bunch of bias early and use a closed form method that's familiar, sufficient, quantified, straightforward, and interpretable. This becomes all the more likely when the quantity and quality of training data is limited or variable. Scientists we serve aren't especially keen for us to play around with cool new toys when they're on a tight schedule.

That said, there do exist a number of problems that are so noisy or contain so many signals that finding some sort of low bias method that's also domain agnostic could be a big win. Fortunately these kinds of problems are famously hard (like protein conformation prediction, or 3D shape modeling) so if we can make even a little improvement over the status quo using DL/ML, our scientists are usually willing to put up with the higher costs (and often lower punctuality) in our development of novel learned methods.

Often scientists want to play around with new tech too. Working with them to exploring novel approaches (like deep learning) to serve their projects' "stretch objectives" is sometimes an opportunity for us both to get ambitious and try out something new.

I'm surprised by this, since I was coming in here to say "AI in bio research is mostly a dead-end, except perhaps for NLP paper-mining stuff." I can see how medical applications would be tricky (since the field is covered in regulation from the 19th century and doctors have a deep-seated fear of computers) but as a (roughly speaking) biologist I would love to have a program that would let me ask high-level conceptual questions and get answers from the general mass of academic literature. Or, given some experimental result, find equivalent past experiments that corroborate/negate the result. Etc. Things that seem plausible with modern NLP and a fairly conservatively-written corpus, but I haven't heard of tools that could do this. (Even Siri style "here's what I found for your question, now look through these and see if they make sense" would be abundantly useful, if it's sifting out the right ~10 papers most of the time.)

Every scientist I know has a desk covered in papers (presumably, the select few that were important enough to print out) and even for experts it takes time to glean information from densely-written things like that; there's definitely a problem there to be solved.

> I would love to have a program that would let me ask high level conceptual questions and get answers from the general mass of academic literature

AI researchers too would like to have such a tool. It's getting hard to keep up with the torrent of papers. But such a tool would require actual understanding, thus, we get to have a subproblem which is more difficult than the original problem of finding information in text.

Designing a program that can read (all) papers and understand their content conceptually, then comprehend the question it is asked, and finally reason about all that to generate a sensible output is equivalent to creating an AI (a real one, not the marketing bullshit type we've been fed for the last few years). I don't understand why people even imagine this is possible with current technology?

I don't agree with this. Have you not seen the work by Regina Barzilay and David Sontag at MIT? His paper about embeddings of categorical data across 4.5million medical records?

You can build an entire CDS from the data, which is publicly available.

And if you can't find a problem to solve, then why don't you ask where the pain is? There are a _ton_ of low hanging fruit items that NLP and Deep Learning that can help with _today_ in the clinical environment. How about:

Question-answering for doctors that immediately need to find time-critical facts from a patients medical record? My sister, a hospitalist, begs for help in this area when treating seriously ill patients that come into the ER from different hospitals.

How about helping insurers better model their patient populations from unstructured data?

ICD coding for medical billing, our extremely wasteful 16-20billion dollar a year industry with _terrible_ NLP solutions in the current market?

There are problems to be solved. You just need to ask your customers, and find their pain.

I run a startup in this area. Could we have an offline conversation? My email address is in my profile.

You call your problems "low hanging fruit", yet they are real problems that multi-billion dollar enterprises are facing and know of. If they really were as "low hanging" as you say, don't you think they'd have the resources and capacities to already have solved them? (Looking at you, Watson...)

They are, but you have to realize that the companies themselves are focused on healthcare first, and reimbursement. Healthcare in the United States is a fee for service model, not a value based one. So the system isn't aligned for it. But they are doing it, look at the sepsis NLP solution created by Mass General in-house.

You're not addressing the problem area the author is looking at though, which is lead generation (finding new molecules that either have high binding affinity to a receptor and/or might be useful for a given medical application).

A bit removed from drug targets, but have you seen AMELIE - it's for ranking causal variants (genetic mutations) for patients based on literature + patient's phenotype?

https://www.biorxiv.org/content/early/2017/08/02/171322

I'm not in the NLP field, but this does seem like a legitimate use

Just start an ICO, there are not that many medical ones. You already have all the right buzz words to raise a few billion ;)

1) What short-term goals are you trying to achieve ? Recent AI advances have made it easier to transform unstructured information into structured one. But you'll still need to do some boring correspondence dataset construction. There is great value for a better research/cataloging of publication data (but probably not many people will pay for it).

Find what the users do with the data and make it easier for them using modern tools.

2) Medecine is a legal minefield. There is definitely a place for AI to cure illness and improve health. It's hard to get the data, and to standardized in it shape, but I'm convinced that just by the mere fact that it will have more personal data and vastly more general knowledge that AI generalist doctor will trounce any generalist doctor even with simple predictors. The real value is not to cure you when you are ill, but preventively to keep you healthy by continuous monitoring and adapted personal recommendation. Look at performance enhancing equipment for athletes optimal training for example.

There are plenty of things you can do with your graph database (probably more with the metadata than with the data) which can provide value, you can use it to make recommendations of paper to read (like arxiv sanity). Use it to connect researchers, for example if one of your co-paper author work with another researcher then you can suggest a work relation. You can cluster researchers. You can rank researchers according to a multitude of criterion (like an ELO score for researcher). You can create recommendation so that you reduce isolation between clusters and promote information dispersion.

I appreciate this radiologist and current ML PhD student's perspective, as a trainee myself: https://lukeoakdenrayner.wordpress.com/2017/04/20/the-end-of...

If you're a basic research scientist, you can't just rely on PubMed, simply not everything is indexed there.

Google Scholar is pretty much the only place you can go for that sort of thing, and it's distance from things like Microsoft Academic Search are leagues.

I'd kill for a non-Google academic search, and smarter Google Scholar/PubMed alerts would be appreciated.

I think there is a lot to be gained by creating better systems for searching and synthesizing diverse pieces of biomedical knowledge. But indeed this is mostly a field of research, rather than something that could be directly marketed. Did you think about going (back) into academic research?

Absolutely, always wanted to finish my Epidemiology PhD. But eeh the pay is awful. Not that I /really/ care, but that's getting into personal details. Still, whatever I could come up with now would not be much different than what I would come up with during academic research. The machine in my head is working regardless.

What is your use case?

Based on the link you provided, it appears to be a cloud tool that allows someone to generate end reports comparable with uptodate.com. Is this the case?

A domain specific vertically integrated search engine seems like a great idea but probably requires being bought by Bing or Google to incubate while both the world and the technologies develop.

You probably already know, but there are teams with the national library of medicine that work specifically on natural language processing within the scope of pubmed, but also on new ways to get to the publications. I almost interned there for a summer, very fascinating stuff.

yep, we're trying to outcompete them. But it's a hard thing. MetaMap: we beat it, cTakes, we can do it faster. But really how many < 80ms concept recognition systems do you need that work on 28M documents. And if you have that: what are you going to do with it. We have, internally, systems that predict dense vectors based on CUI graphs of concepts in text. Great for similarity search: but utterly useless in the end.

Yeah, I run into this problem all the time in the space. You build something really awesome, an either doctors don't care, it doesn't fit the workflow, or it turns out to be minimally useful in practice. We abandoned most NLP efforts and dump tagged terms into algolia for a lot of use cases and it works well enough to fit a prescribing workflow. The docs already know what their searching for, they just need to know its in the catalog and like the autocomplete.

If I might suggest something, maybe forget about targeting older doctors and researchers set in their ways. Go after the youth, the ones just starting their education in research/medicine, you'll find us far more pliable.

I don't think you need to out-complete these systems in processing speed. Even if you can only do 10 documents/hour, that probably can be scaled to handle a few million. Where you need to out-complete cTakes & co (on a large margin) is information extraction quality (to have a viable commercial product). And that is very difficult...

As John Ioannidis has shown, most Published Research Findings Are False.

Are the inputs to your knowledge graph weighted by reliability? If not, the knowledge graph is unuseful.

Yet proving that reliability in a fully automated fashion is hard. And it's even harder/more costly when the core problem is reproducibility...

Reply on news.ycombinator.com