The business of extracting knowledge from academic publications
theseedsofscience.pub
theseedsofscience.pub
A pharmaceutical company that heavily leverages computation is called a pharmaceutical company. All modern pharmaceutical companies heavily leverage computational tools - including some powered by deep learning.
Any company building a computational platform to accelerate drug discovery / development is not a pharmaceutical company. They are a software company who wants to sell software to pharmaceutical companies, which is a terrible business to be in.
The product the author is selling already exists for free. All of the big knowledge bases use automated ML tooling for curation and extraction. And the people who run them and QC them are world class experts in their domains. I mean, just take a look at uniprot.[0]
And the only types of pharma companies who would want a big knowledge graph would be the large ones with active programs across multiple therapeutic areas. Most of the small companies / start ups tend to be focused on getting one or two assets to market in a therapeutic domain. And the big companies possibly have their own ML teams doing literature extraction - because a team of 5-10 FTE ML / software engineers is like a rounding error in their R&D budget.
The other thing is that the most valuable knowledge is the stuff that is not in the literature. That’s why we do experiments.
Still, it's a tough business, and they were bought by Qiagen (to make an end-to-end solution, with moderate success, as the market is strangled by Illumina's control over NGS sequencing). And with exponential growth in information, their early relative advantage likely has waned.
Also note Veeva software, which provides infrastructure for pharma, is a public benefit company, legally devoted to its clients. That's how much power the customer has.
That said, there's something of a revolution in drug discovery as more is being done by small companies with ultra-focused expertise. If they get a viable candidate, a bigger company buys it to marshal through approvals, and then it can be sold again to a marketing company. Focusing on these pop-up drug-discovery companies that evolve out of someone's PhD could be a good niche; I would expect Pharma VC's to prefer funding companies using their favorite computing vendor, for reliability if not insight. Here computational discovery assistance would help a bit, but validating findings and science could be golden.
The problem is that the market is just not that big. Assume that every single pharma company buys your product - like Schrödinger - where does your revenue top out?
You can beat that as a small pharma company with one or two assets that make it to market. So the question is: if the computational platform is so good then why not just be pharma company?
If anybody with an AI pharma start up tells you they plan to make money by licensing their platform, then run the other way because it is a red flag.
If you want to see what a real “AI” pharma company looks like check out Vertex pharmaceuticals.
A very common feature across verticals is that in house data analysis is siloed.
While it's a difficult sell, getting multiple siloed data sources to agree to 3rd party analysis for shared gains can be extremely successful for everyone involved when pulled off.
So looking specifically at pharma, while analysis of published papers is meh, an often discussed component of research is the bias against publishing negative results.
So a hypothetical product like using ML across multiple firms' data of failed products and research in order to establish a model that could more quickly identify dud research avenues by leveraging industry data could only exist with the broadest dataset as a 3rd party product and would deliver gains that could be quite profitable for some of the largest companies out there.
Also, I think anyone who has worked with larger corporations on in house tech knows that even if the individual efforts are quite large and sophisticated, the sheer amount of bureaucracy that goes into every single thing at a sizable firm can mean significant advantages for startups vs in-house efforts, particularly when related to fast moving fields.
I'd agree that "moving into a niche without knowing it extremely well" can be fraught with issues and that attempting to get buy in from B2B firms for a startup is a nightmare, but I'd disagree with "large company does X in house so creating a startup to do X is a bad idea."
Third party data analyses is a thing, but this is often bundled with conducting experiments by CROs.
See my other comment about market size. The problem is that you can make A LOT more money selling drugs than you can selling software to pharma companies. So if your software is any good, then you should use it to make drugs and just be a pharma company.
0. https://www.opentargets.org/
1. https://www.citeline.com/en/products-services/clinical/pharm...
As for your other point, I'd imagine there's a pretty big difference between the capabilities and infrastructure between bringing a drug to market and analyzing the data of people who bring drugs to market.
During my Ph.D., my PI wanted to do some work on signalling pathway analysis. He was interested in knowing when and why sometimes body starts assuming that the sick state is the right state and start acting against drugs given to patients. This making treatment very hard.
I don't know what is status of that but one insight I got from him that most (all?) Pharma companies do not trust acdemic data. I didn't ask reasons but I'd assume because of replication issue (crisis). Knowing how data is gathered and published in signalling domain, I'd not blame them for having low trust in academic data.
this looks like one narrow niche knowledge base. I am not expert in this domain, but there are probably some other use cases not covered by existing offerings.
> literature extraction - because a team of 5-10 FTE ML / software engineers
I think this problem is so hard and open ended, that your 5-10 non-star avg salary ml ftes likely produce very mediocre and likely not usable results.
But they are just one knowledge base. There are others. Each with their own focus area. And they are all associated with prominent bioNLP / biomedical AI research labs and employ human SME curators.
> 5-10 avg salary non-star ML FTEs likely produce very mediocre and likely non usable results
LOL!
So, is your claim that all topics/niches/verticals/steps in pharma development and productions are covered by all those databases with absolute quality and perfect UI/workflow? I follow some studies about trials meta-analysis, and my impression is that end result is mainly produced by manual work of some low paid postdocs, which makes them not trustworthy.
> LOL!
I actually claim non-trivial expertise in this area (facts extraction from untrivial niche documents), and my observations is that even SOTA (aka results from top researchers) are hardly useable in real life, because this area is very hard. Could you support your "LOL!" with any references I could check?..
Meta analyses? Automating meta analyses is not a real need. They have their place, but it’s like a quaint cottage industry type thing - like custom haberdashery.
Also the most valuable knowledge is not in any publication. If you are reading about it in an article then you are already 2-3 years too late.
I search https://www.google.com/search?q=site%3Ageneontology.org+nlp+... and don't see anything meaningful.
> Automating meta analyses is not a real need. They have their place, but it’s like a quaint cottage industry type thing - like custom haberdashery.
my opinion is that it has to be top level tool in modern science which could easily sort out lots of bs and contradictions in reported results.
Long:
This is the bit that makes biology awesome to me, so excuse me for the small essay ;-)
Proteins are basically extremely advanced nanomachines which work together in larger systems to ultimately form a cell. Having a listing of all the proteins in a cell (the sum of the parts) is insufficient to grok the whole, but it's pretty darn important. The abilities and limitations determine and constrain what a cell can do, and ultimately influence what organisms and ecosystems are and are not capable of. Which is a big chunk of the science of biology.
Uniprot lists many/all of the genes/proteins that have been decoded so far. It's a bit odd to call that a "niche" in the context of the field of biology.
I'm not going to discourage you though: Proteins and protein systems are pretty darn awesome!
Example:
(using KEGG rather than uniprot, since it's got graphical maps, which is handy to get an intuition)
For instance, if you want to know why you need to eat and why you need to breathe (flowchart for what your cells do with starch and oxygen):
* https://www.genome.jp/pathway/map00500 start by finding starch on this map (gets split into glucose)
* https://www.genome.jp/pathway/map00010 which gets broken down into 2* pyruvate
* https://www.genome.jp/pathway/map00020 which gets processed
* https://www.genome.jp/pathway/map00190 and ultimately "burned" with oxygen.
Each step is a 'chemical reaction catalyzed by proteins'[1] (in the rectangles). You can dig in deeper to find your actual source code: Say we click on a random step (in this case near the top of glycolysis on map 00010)
* https://www.genome.jp/entry/K01810+K06859+K13810+K15916+5.3....
At the bottom you can find the gene listed for Homo Sapiens (HSA)
* https://www.genome.jp/entry/hsa:2821
And this lists the amino-acid (AA) sequence for the protein, and the nucleotide (NT) sequence found in humans. Since this is highly preserved functionality, that's probably (almost) exactly the source code that you have in each of your cells.
KEGG is nice to get an overview of some of the pathways that are fully understood with the maps.
[1] calling it a "chemical reaction" is sort of underselling many proteins. Proteins can have moving parts and can work together. I prefer to think of them as sophisticated nanomachines.
I actually checked uniprot, yes, it lists proteins (probably most of them), but ontology is raither narrow, it has few dozens properties, you can't for example query that DB with question: give me diseases which can be attributed to broken pathways synthesizing protein X, you would need to do a lot of manual work and check external databases of uncertain quality.
Another question is quality of that dataset, why it is so obvious that all those millions of pathways for hundreds thousands proteins are researched and described with 100% accuracy?
nod
Uniprot is more useful if you're looking for the actual "bare metal" NN and AA sequences. Which is rather important in its own right, obviously: Sooner or later you DO need the actual sequences if you're going to do something with them in real life.
But uniprot doesn't -itself- give you an understanding of what that code is then doing.
UniProt is not niche. It contains curated information about all proteins across numerous domains.
post is not about pharma specifically, but about general bio-medical literature, including genes, diseases, symptoms, etc.
> proteins make up more than 95% of the targets of the pharmaceutical industry.
do you have references to support this? Some google search says it is 280B market out of 1.5T total pharma market: https://www.alliedmarketresearch.com/protein-therapeutics-ma... https://www.statista.com/topics/1764/global-pharmaceutical-i...
Also, it is likely important steps how to research and produce protein drugs, but the end target is to cure diseases, so you need lots of additional data about diseases of all types, symptoms, pathways, trials, etc.
People are being dismissive of your comments because to say that proteins are niche in the context of pharma is like saying advertising is niche in the context of Meta and Google.
its all about how you define word "niche", for google, main revenue stream is supported by several pillars: search tech, infra tech, ads tech, ecosystem+network effect, human management. You remove one pillar, and everything is destroyed, so one can say ads is one of the niches in their food chain. I suspect with proteins it is about the same.
> in the context of pharma
there is no context of pharma. Post is about more broad bio-medical publications.
I'm now not entirely sure what your experience is with bio sciences. You're definitely coming at it from an odd angle though!
You never know, maybe you'll end up contributing to our understanding of life, maybe (indirectly) even save a few lives!
life science is one of potential applications if there is an interest and money.
But if you mean in general if you're capable of looking at metabolic pathways where each protein catalyses a step in the pathway, that's definitely interesting. If a certain person has a flawed gene coding for protein X, that could indeed cause a problem.
To find valid answers, you might need to eg. track nodes and states in a graph, to figure all the consequences of a break. Not all types of storage systems/engines are equally good at that.
[1] https://en.wikipedia.org/wiki/Transcription_(biology)
[2] https://en.wikipedia.org/wiki/Translation_(biology)
edit: s/protein pathway/metabolic pathway/
Yes, I built system which traverses paths in graphs with 1B nodes and 10B links in 1h on affordable server. But that's only one part of the puzzle.
Literally the first line in the comment that started this thread.
that's one of the themes (and you are already working hard to stretch drugs pharma to "biochemistry"), if you can't see other themes in his examples and screenshots, I think this discussion is not interesting to me.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6314433/ Check out this article for example.
Even when a drug target is unknown, or remains mysterious, very high chances are that the target is a protein. DNA or RNA as targets are niche (DNA is often avoided as a target on purpose and it’s hard to be specific to it without DNA-like material, and RNA is still hard to target effectively, though things are improving). There is not much else of use in the cells (lipids, sugars, cofactors, and metabolites, some examples of which have been targeted by a couple drugs each over the long history of trials, often unintentionally.
Small molecules are an excellent modality for an eventual approved therapy. They almost always (so far) targets a protein. They are hard to design but when they are done well they expose the target to something nature hasn’t seen before in order to get a desired effect. Sometimes people don’t care about the target itself (think recreational drugs, or phenotypic drug discovery), but the target typically remains a protein.
Proteins make up the machinery of the cell. You jam or modify them to achieve desired effects.
The article describes two separate things likely to appeal to HN readers. The first is that there is a lot of tacit knowledge not captured in scientific publications. The second is that the author and his team, despite best efforts, never found product-market fit.
To the first point: it wasn't until I got to graduate school that I realized that the scientific literature isn't exactly 'An accurate record of true facts.' It is instead the paper-trail of a slow-moving conversation among researchers, where old ideas are slowly jettisoned and new ideas are evaluated and tried on.
To the second point, a few reactions:
* If your primary market / audience is graduate students or post-docs, good luck! You can't sell to people who don't have money: grad students are paid around minimum wage, postdocs its slightly better. If I had to sell a science or research-adjacent product, I'd either sell to entire departments or colleges / campuses. This is likely a pretty protracted sales process and doesn't seem pleasant.
* I wonder if Law would be a better place to start. Either selling to law-firms directly (building tools for generating internal knowledge graphs when they are given 80k documents in the discovery process), or particularly for IP lawyers. IP lawyers have the money and expertise to have their skills augmented by AI-powered literature searches. I have to imagine it's not just patents they read when looking for prior art.
* I wonder if the author and his team tried to solve too broad of a problem: they never seem to have gotten hyper-specific, and built something from the bottom up.
To be honest, while the author's depiction of academic publishing is mostly not wrong, they make it sound much worse than it actually is. Folk knowledge is a thing, but papers do contain most of the valuable knowledge if you know how to read them.
I think 95% of this person's failure to monetize their product comes from trying to sell it to an audience that is just quite broke, and the rest is probably mostly post hoc rationalization. Not only grad students and postdoc wages are low, in many countries (not the US) professors aren't well paid either (and buying software subscriptions from grant funds is often not allowed or difficult due to crazy bureaucracy).
As a full professor myself, I almost don't buy software for work. I suffer the torture of Microsoft Office, which my institution is subscribed to, I'm subscribed to Overleaf with grant money (for now, but I might be forced to cancel depending on how the funding goes) and I pay for ChatGPT out of pocket because trying to use grant money for that is bureaucratic hell. That's all. It would take a really transformative piece of software for me to subscribe to something else.
- Bureaucracy. Writing silly boilerplate, e.g. data management plans or gender perspective statements in grant proposals.
- Cutting or expanding text (we routinely have lots of forms and submissions where you need to write a text in a given word or character range).
- Polite emails in English to people I don't know much (e.g. "Write a polite professional email reminding this person that the deadline for reviewing paper Y expired yesterday...")
- Brainstorming. "Give me 10 ideas about research direction in topic X". It won't give great ideas, but it's good to set the mind rolling.
- Routine scripts/code used in experiments and papers: write a Python script to make a box plot with such and such data, or to take a file in this format and strip this unneeded content, etc. The typical kind of code that appears a lot in research, is trivial to code but consumes time and ChatGPT does it in seconds.
- Suggest titles (paper titles, grant proposal titles, etc.).
- Suggest ideas for exercises or exam questions (e.g. write an assignment that can be solved with the coin change algorithm but involves no coins or currency).
- How to do X in Excel (although the problem here is that my Excel is in Spanish - why, why did they decide to translate function names? - and it's not that good at that - but anyway, it's very useful).
The productivity boost is very noticeable, well worth the cost, even if it hurts to pay out of pocket for a tool used at work.
It is a hoot that they sell access to individual scientific papers for $35 because if you think one will help you with some commercial problem you have the odds are the real value is $0.00.
also, lots of competition already
"Business intelligence" is very broad term, maybe it is possible to find market fit there, since area is moving very fast, but hard to judge without seeing specifics.
to me every filling has maybe 20 essential numbers which are interesting: balance sheet, income statement and major sectors, everything else is some generic boilerplate, and there are dozens of services which will already sell it for cheap.
Not sure what else you can sell to your clients..
Think of a global aircraft manufacturer turning maintenance documentation into a knowledge graph, a global clothing and shoes retailer building a model of what social media thinks about them, etc. I told other employees that our product could generate enough value for one customer that it would be worth it for one to buy us and... that's what happened.
I think this is kind of what the author attempted to build (i.e. something that spits out the Literature review portion of a thesis.)
I think that's probably why graduate students were excited - they are the people who have to write a thesis at the end of the day.
In regards to the article, parsing academic publications and spitting out a word cloud or k-nn graphs of topics isn't going to be useful to a professional. They've already built up a working model in their mind that they've honed over the years. They have years of filtering information and the ones contributing to these knowledgebases have the experience to curate that information to professionals which is what's lacking from these NLP experiments.
I do think that ML and tools like SemanticScholar can be used to identify new literature that may affect knowledgebase articles and flag them for review. I'd be surprised if that doesn't already exists to some extent.
Previous discussion: https://news.ycombinator.com/item?id=29481061
But neither of these are enough to combat the plethora of issues that the author identifies and it makes me so sad, particularly once you get into publishing and see how insane peer review is from every corner of it (paper author, reviewer, conference chair). Academia truly feels like a cartel at times.
In others you need a hardware like a microscope X with a lens Y and a light Z, and use cells of W cultivated with nutrients V by the graduate student U that is the only one that can keep the cells happy. You can't just git clone & config & make it.
I saw a blog post by a fellow from OpenAI earlier today which basically said "when you refer to a model like Bard, or Claude, or Llama -- you're moreso referring to the dataset, than you are the architecture."
His meaning was at training time, but perhaps a similar observation can be had in this context: The value of a retrieval / organization system is only as good as the average information quality in its target corpus
But yes, more to come out next year obviously.
> Divergent Tasks are Hard to Evaluate and Reason About By “divergent” I mean loosely defined tasks where it's unclear when they're done. That includes "mapping a domain", "gathering evidence", "due diligence" and generally anything without a clear outcome like "book a hotel", "find a barber", "run this assay", "order new vials"...
This is not just a problem for outsiders, or building a map, it’s actually true. I can’t count how many projects I was told to “find some kind of meaningful evidence” for something that is super tenuous.
1) Identification of all the major research groups working on the problem, including their physical locations and resources, lab standards and data standards, history of grants and proposals, etc. This will not be in the academic literature, and is fairly difficult and possibly expensive to acquire, and if you don't know what to look for, well, you need to hire experts with relevant experience. Think of it as due diligence (Theranos investors got burned because they didn't do this).
2) You have to aggregate a lot of papers to get a good picture of what each research group is up to. A single coherent project might generate a dozen papers scattered here and there, and importantly, they won't have published their failures in most cases, even though that information is just as valuable to an outsider attempting to replicate or advance their work as the successes are.
3) A common mistake is to neglect materials & methods and instead focus on results and discussion. A large fraction of the literature is based on poor methodology and so the results can't really be trusted, and fraud is remarkably widespread in academia, for various reasons from PhDs desperate to graduate to PIs who've made the practice their bread and butter for decades. Clearly written materials and methods sections that include all the information needed to replicate the work are an indication that it's fairly trustworthy. Deliberate obfuscation is a bad sign.
But it took decades of expert hand-curators going through piles of articles to get to the current state. Also, proteomics articles report very simple outcomes that relatively easy to annotate. For instance, the subcellular localization of a given protein isoform.
I appreciate the author didn't find a good product-market-fit. But the above claim makes no sense and isn't supported in the article.
The author doubles down on the idea later:
> All that is to say: discovering relevant literature, compiling evidence, finding mechanisms turns out to be a tiny percentage of actual, real life R&D
The author admits to not having worked in research, and this is one place that lack of experience shows. This is the kind of thing that takes years, or decades to develop an appreciation for.
Discovering relevant literature, efficiently and well, is a thing that can make or break you as a research scientist.
Over time I've sensed the devaluation of literature search skills in science, but I've also noticed those that can't do good literature searches do bad research. They waste time, sometimes years, in re-discovery of old results. They commit resources to experiments that don't need to be done. They have objectively worse ideas because they don't actually understand the field they're working in. They can't see the holes in the literature that lead to opportunities.
I know this because I put it online in before I left them in 2012 [2] however I'm unable to use their new data explorer to extract any usable information.
In the past we provided direct access to the malaria endemicity surveys and georeferenced anopheles occurrence records but I'm now struggling to find that for some reason.
[1] https://malariaatlas.org/ [2] https://pubmed.ncbi.nlm.nih.gov/23680401/
2. How much would you pay?