ChatGPT and other AI tools could disrupt scientific publishing
nature.com
nature.com
I'm quite sure that people who use the wrong way will end up regretting it.
And no, humans are not just as bad as AI at this. I don't know why that meme persists - if humans were just as bad, much less worse, at hallucination and confabulation as AI at scale we would never have gotten past the stone age as a species. It would be useful to keep humans in the loop but the inevitable exponential explosion of AI generated research will simply be too much for peer review to handle (see the same thing happening with fiction writing, and several publications having to shut down because they were drowning in AI garbage.)
But it's not as if it matters. As with everything else that incorporates AI, quality isn't relevant, only quantity and profitability.
That would be the large number of salient examples.
I expect humans to form a (possibly normal) distribution in this regard rather than being all of equal quality, and people on the high end would permit development even if the mean was worse than even the best current AI.
If we were however all clumped together, then the very same meme you're complaining about here would itself be an example of humans doing exactly what you're saying AI do.
I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen. The "replicability crisis" would appear to be a counterexample, but it isn't. Actual attempts at research aren't the same category of error as randomly generated research.
When AI gets something right on the other hand it's entirely coincidental, because AI doesn't have any concept of "truth" or "falsehood," or the real world context in which it operates. AI just relates text tokens to one another, with some unavoidable randomness, trying to match something that looks like the desired result. To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
Sure, but I expect that to change the distribution, not much else.
> I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen.
Hopefully not, but even in science, misunderstanding the cited work is a thing, as is disbelieving it. For the former, "perceptrons can't do XOR" became "AI is impossible", while the Black-Scholes equation was misapplied to ultimately lead to the global financial crisis. For the latter, the salient examples are older (e.g. the importance of washing hands between autopsies and midwifery), or things where I can't find the historical scientific consensus because of the modern cultural noise (e.g. thinking men and women have different numbers of ribs because of the book of Genesis and not checking).
In this regard, I think the replication crisis is valid, even though it (presumably) involved real research and real results, because widespread citations didn't fully account for the limits of that research.
To put it another way, while I agree with this:
> To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
With scientists I'm referring more to the game of telephone that necessarily occurs between any given researcher and anyone using their work.
For the general population it's only that game, hence all the people who think scientists in the 70s were worried about an ice age, or the newspaper stories that collectively divide the world into things that cause or cure cancer.
If you are referring specially to people who want an LLM to actually perform novel research, rather than merely write up lab notes in LaTeχ, then you would be correct that this is a terrible idea. Even for pure mathematics, you need a different type of AI, better suited to that task — they do exist though, and have been used successfully, but IIRC are not anything like as general with regard to maths and experiments as LLMs are with language.
The whole system favors publication quantity over quality, results over process.
The solution is to ask our AI overlord.
I am a scientist, and I don't think anyone knows how to do this. There also is very little skin in the game for this. Bureaucrats want dumb metrics that they can point to to determine success regardless if that metric is meaningful or not. There is a strong pressure to publish quickly, which directly conflicts with a pressure for quality. Many prominent Nobel Laureates, Fields Metal Winners, Turing Prize Winners, and so on have discussed how they themselves could not have thrived in the current environment and its insanity. But who is going to change this? Honestly the ones that are hurt the most are the grad students at the non-top 10-20 universities. Everyone else has found "success" in the system (as in learning how to play that game) and they are highly incentivized to maintain that system and their status. Research is a completely different game today than it was even 20 years ago and we have not adjusted our system accordingly, and worse, many want to pretend that it hasn't changed. An interesting simple example is that research teams have exploded in size (especially at large universities) which makes for a lot of Nobel drama (max 3 per team). But there are also many other simple and more nuanced points that exist, but few want nuance. Either way, I'm absolutely certain that our current metrics do not strongly align with producing good science.
Oh, of course. Most academics don't do it for the money. It's not like they're paid much. Though that can incentivize cheating in another way because they want money. But there will always be people that don't give a fuck and just want to do good research regardless of the metrics being used. I am insistent that to be a good scientist you must be somewhere on the side of "anti-authority." Because your job relies on challenging concepts, and especially well known and widely agreed upon concepts. Generally look for people who are passionate, will rant, but importantly rant with nuance. Those are the people passionate about their research and not the metrics.
The problem is we're just throwing a lot of money down the drain, wasting a lot of time, and generating distrust of the system. Any "crash and burn" is never going to lead to an extinction in any sense. It's just about general society level if we want to do good science or just do noisy science (all science is noisy). But you can never do good science if you remove nuance. This is why I hate that ML (and CS in general) uses conference systems as the main platform for publishing. It is ridiculous to think you can have a good system when it is highly competitive, zero sum, highly time consuming, there's no discussion between authors and reviewers (you may get a one page rebuttal, but your reviewers comments are often disjoint and vague), and you're being judged by those you're competing with. It's just a silly notion to believe this is useful.
The solution is actually not hard. I refer to the larger phenomena as Goodhart's Hell. The solution is to stop using metrics as targets. Metrics are guides. If you don't have a deep understanding of what your metric actually measures, how well your metric aligns to the thing you're intending to measure (never 100%), and if you don't understand the biases to your data, you're fucking doomed to this bureaucratic hell. Noise is inherent to complex systems and aggregation is the bane of evaluation of complex feature spaces. Just remember that its models all the way down and that all models are wrong (despite some models being better than others).
I think that will happen when we swap peer review for peer replication.
Knowing this would require a counterfactual machine of some sort, thus the fact must be hallucinated into existence, no?
As a scientist, you should be aiming at communicating ideas in a pragmatic and verifiable way, not embellishing stuff via AI.
A real facepalm moment for scientists with wrong incentives.
The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely.
Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).
Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.
The one time I ask an LLM AI to write something for me is after I think I have a final draft, I stay in the chat window that has all the history in it from where I worked through the paper. Then I ask the AI to write the paper for me. I do not take phrasing from it. I look for new ideas the AI inserted that I missed. There is usually at least one good new idea for me to steal.
All other uses, including, specifically for generating ideas, are plagiarism and ground for expulsion.
Maybe ChatGPT will help accelerate the rate at which nonsense gets published and will end up speeding the collapse of these areas of "study".
Honestly, which publicly available AIs are capable of doing that?
At least not the publicly available LLM AIs: I asked those quite many questions about scientific topics, but they could not give me any answer that helped me - it was a complete waste of time.
(I am aware that self-promotion is allowed but somewhat frowned-upon in HN. I believe this is relevant to the context and I am particularly qualified to comment on it. But I'll strive to be as transparent as possible.)
We have intentionally steered away from assisting with writing work for the reasons outlined in the article. Our focus has been to save time in keeping up with their field: making literature reviews much faster, summarizing recent discoveries in each researchers' niche area of interest, and gathering how others have tackled similar highly-specialized research problems. Grounding LLMs in scientific fact and always providing verifiable sources.
We have spent many years before this building a business around our "Map of Science" at Scitodate, so we have strong connections with the community and a lot of experience in this domain. Both in terms of how science is done and how to reliably extract valuable knowledge from millions of documents with NLP. We now have one of the largest datasets of papers, patents, funding grants... Updated on a daily basis. With plenty of enrichment, connections and aggregations: we have done a lot of in-house research on disambiguation and entity-linking. We particularly have focused on mapping out the expertise, interests and achievements of all professional scientists, as well as the institutions they have worked at.
We only use LLMs to retrieve and extract information from reliable sources (mostly papers and patents). If you provide extensive background information in the prompt, it will stick to that and avoid making up facts, and it has the added benefit of including verifiable references for each statement in the answer.
Again, we treat the AI mostly as an "automatic reader", it's very good at that, it's not hard to keep it factually grounded and that alone can save hours of research. But when asking it to do deep reasoning or original writing it can be a bit more tricky to keep it controlled.
Use them to replace grad students / interns.
In other words, AIs are useful when you can confidently evaluate their output for correctness.
Note that this also applies in novel scenarios, such as new scientific discovery. It's often much more difficult to come up with an answer than it is to verify it.
As a random example, modern AIs are very good at image recognition tasks, which would make them ideally suited for finding rare phenomena in all-sky surveys. The AI doesn't need to write the research paper! It can be valuable if it can just find candidates for detailed evaluation by human researchers.
Similarly, I've found AIs are quite good at spotting errors and inconsistencies. Don't make it write the paper... just ask it to proofread it for you.
Etc...
Prep it with data you know to be true, then start your discussion. If you show up unprepared and let it tell you what's real, it will bullshit you in its favor.
“LLMs can’t self-correct in reasoning tasks, DeepMind study finds“
https://news.ycombinator.com/item?id=37823543
Anyone who says otherwise is either ignorant of the underlying function of llms or trying to sell you something.
This is just wrong lol.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The needless condescension of your “lol” feels a bit premature.
How can you have self correction without superintelligence?
The study didn't "find" (discover) what they claim, rather, they didn't find validation that it "can" (the implementation of which varies per observer, sub-perceptually).
If you had to code something like this at work for a different domain, I bet you'd have no problem realizing that a nullable boolean is required to accurately model the problem space.
And this isn't just a one off, this is a systemic, institutional shortcoming, I encounter several instances of it every day just in my regular social media feeds.
- What is the cost?
- How does literature search happen? What are the sources? How often is it updated? How sub-niche can it actually get? You specifically mention the researcher's niche interests but your examples are "Why can't we sleep when excited", "Companies selling 3D printers", "3D printers in Eurpore", and "3D printers for medical purposes." These are not niche topics, this is pretty broad and doesn't look like scientific research.
I'm sorry, but without more information this pitch and website just look like an information harvester. This may not be what you've actually built, but this is what it comes off as from someone who does scientific research.
Before going to the answers, just to note: if you are interested, we are looking to work closely with more researchers, to make sure it saves time for you in particular on a daily basis.
What does it do?
In very simple terms: It is ChatGPT (GPT-4) with awareness of our database of papers, patents, grants... It comes with many "tools" to use various information retrieval styles and sources, some more focused on question answering and some on wider reviews. There are also tools to gather "metadata" of science: researcher profiles, institutes, scientific companies...
What is the cost?
There's a free version if you register with an institutional email (university or company email). We generally want to make it open to professional researchers (academic or private). The full version is pay-what-you-want with a $5/$10 minimum.
How does literature search happen?
It depends on the "tool" you are using, but it generally involves retrieving a set of candidate relevant documents (papers, patents, websites...) and either read through full documents or select relevant snippets from them so it can do a wider review.
What are the sources?
Primarily papers (~100M right now), including ArXiv, PubMed, IEEE, the Elsevier suite and many more individual journals. As well as grants from most national and international funding organizations. And US patents, with ongoing expansion to other countries. Updated daily whenever possible.
We also have the option to complement scientific literature with web research, it can be good for example to review public documentation from scientific instrumentation and consumable providers on experimental protocols.
How sub-niche can it actually get?
It's hard to convey, it would be easiest if you just try it, it really can get as niche as you want it to, it can adapt to the level of expertise you operate at. If you are a researcher with a university email you can use most of the relevant functionality for free.
You are right, the examples are not very representative. The intent was to make them understandable to a wide array of researchers with different backgrounds, but still, yeah they could be better.
Sure, I can sign up. But can you clarify data retention and usage policies? As researchers we're always worried about being scooped. Despite complaining about the system and incentives in other comments, we're bound to the game we're forced to play to achieve scientific progress (if it weren't for this stupid game I'd research completely in the open). I can't find information on how data is used. This is especially important if you're discussing institutions like work, as many of them can't even use standard GPT4 accounts do to privacy concerns.
I'll provide some comments about what I personally would like but clearly personal preferences. I'll say that there's a lot that can be done in this space that doesn't even need AI.
For literature review:
Let's say I am a ML researcher (true) who focuses on 3D semantic image segmentation (not true) or some other appropriate niche (we should be able to get much more narrow btw). How do we deal with this literature search? As a simple example, let's say I wanted to get introduced to a topic that my colleague is working on. Finding a proper survey paper is often a surprisingly cumbersome task. Making this searchable (especially with a ranking not just by citations by some coverage metric -- too many 10 page "surveys") would be exceptionally useful. In addition, I'd love to have graphs that I can see paper or author connections through different sorting methods. A timeline is often highly useful to understand the progress of a research topic and especially helpful to making new surveys and onboarding new grad students (or anyone into a new topic).
Citation and Reading List:
I love Zotero, but it is so fucking limited it gets to me. Here's my issues:
There's no good way for me to create a prioritized reading list (which should be topic based). Sorting papers that come across my desk is a cumbersome task and they just don't get sorted. But I am able to assign prioritization values to these works (especially attached with a certain project or domain). But Zotero does not make this a realistically viable task. Realistically I have multiple browser windows and store arxiv links in tabs or use Obsidion (more cumbersome). This is actually something ML could help with, at least for the automatic classification (we can discuss other metrics too if you want).
Discovery is a challenge, especially in a new niche, but the bigger problem is actually wrangling the papers I've already discovered.
What also sucks is that I can't markup these works. Preferably on my iPad (OSX would be fine since easy to share). Adding notes is also a bit cumbersome. I often want to add notes to papers to categorize them as citation references to certain topics or place them into importance for certain niches, and this should be different from a reading list. And then comes the problem of sharing. Not great.
I said there's a lot that can be done without ML, and that is because I don't even have good tools to sort and catalogue the papers that come via natural discovery. This comes through multiple devices too, and idk if I'm going to get that on my computer or phone (twitter is still a great place for paper discovery, and comes with author comments/summaries. No tool catalogs this).
Another great pain is citations. Authors aren't helpful here, but grabbing the proper journal/conference citation is surprisingly a non-trivial task. They aren't always indexed by Semantic Scholar or Google, who both have quite large delays. But there's definitely no automatic correction in Zotero for when I add a arxiv paper (via plugin) that replaces the arxiv bib with the proper bib, even if that is indexed by GS/SS. Every time I write a paper this task has to be done and my zotero list gets updated.
So kinda in short, I don't want these fancy ML tools to summarize and write papers for me (I don't trust them for that, especially as an ML researcher. I'm also afraid it might only exacerbate our existing issues with publication, especially in the ML shitshow[0] (side note: useful tool could be finding similar papers and potential similar works. Especially flagging for plagiarism. The community does need this. Especially the blatant kind)). I more need classification models for tagging/indexing and a nice UX that allows me to prioritize reading, sort importance values on a highly niche domain, and dealing with document markup and note taking. It is crazy that good tools don't exist in this area. I'd pay for such a service, even a not great one, especially if it allowed for storage (which is easy on your end. Plus you can lessen the load by central storage to prevent dupes across users as well as automatic version tracking (arxiv)). Even better if you integrated a latex editor. Hell, the only reason I use (pay for) overleaf is because at some point I need to do pair editing, and I pay because I need the version tracking on specifically pair editing (sub-authors make many mistakes, but do need to comment), otherwise I'd far rather use vim and github. (idk an "experimental" editor that is just vim + Zathura would be great. If you hooked in multiple user cursors you'd solve this problem for me. See CoVim, vim-bookmarks, etc. The super feature would be allowing pdf markup with like an ipad onto a working latex document. Jesus fuck, I want this so much!). The experience of vim+github(+zathura) far more pleasant and I can write faster (add an emacs and sublime option and you got all your CS researchers hooked).
Note: I'm 30 year old ABD, not an old tenured professor. While I'd like fancier tools (though most aren't that "fancy"), some aspects also have to be usable by my advisor or boss who isn't as willing to learn new tools. So shifting between advance and simple is a weird challenge, but this is also why things like Zotero can be so bad, because momentum (and no other challengers).
I won't even get started on experiment tracking. That's a whole other rant. But both with experiment tracking and literature wrangling, you'll notice that everyone has their own system. That demonstrates a market need. I highly suggest going out and talking to a lot of researchers and trying to read between the lines of their workflows and their needs. Idk that we have them well defined because we're in a fucking rat race and jumping from task to task with little time to contemplate on these things (I'm sure many of these people would tell you they've wanted to design a system but never had time. I'm included and I doubt far from uncommon)
[0] If you really want to discuss how to solve this problem, we can discuss my anti-{journal,conference} dream and how to create a realistic peer review system. tldr answer is allowing comments on works (linked with profile) and providing links for replication attempts (should be incentivized, even with imaginary points). There is no good central repository for literature discussion, so keep in mind that this can be a great way to build a large userbase quickly. Tons of discord groups and stuff for this kind of stuff but it's a disorganized ad hoc mess.
P.S. How many people on the team have a PhD? I ask not for prestige value but rather as a loose gauge on how close the team's experience is to doing deep academic research. You're not going to experience much of this as an undergrad or even masters. Industry research will have similar and different problems (I've done both). But I've seen too many projects try to guess what we need without having experienced our issues. If you don't have the personal experience it means you need closer collaboration to ensure market fit or you'll misalign (already hard not too with how noisy this environment is).
> How many people on the team have a PhD?
We are already working like this with a few other academics, as well as a number of pilots with R&D teams from industry. And yes my co-founder is a post-doc in theoretical quantum physics, and we have a number of other PhDs with experience in academia in the team, mostly chemistry and life-sciences. Myself I started early in the startup game, I have a masters in CS and have some experience working in academia (a couple of publications, supervised 4 masters thesis, interned with a couple of research groups...), but not a PhD just yet.
> as many of them can't even use standard GPT4 accounts do to privacy concerns
Azure guarantees full isolation and privacy, right now it's the only way to use GPT4 with such guarantees, and no other LLM right now can compete with GPT4 in the domain we operate in. From our side, we are working with a partner on full ISO certification, but obviously security and privacy have been a priority from the beginning.
> (regarding all your preferences on research tools)
There's too much here to cover in a comment, please contact me so we can discuss this further. Just to say that our use of LLMs has started recently (early this year), but we have 6 years of previous experience on doing things like creating global co-authorship and citation networks through state-of-the-art disambiguation and aggregating from many different sources, and working on large-scale information-retrieval infrastructure (complex boolean search with ranking). We have worked a lot with LLMs in the past year for MirrorThink to extract knowledge from the actual bodies reliably at large scale, but this is backed with a lot of non-AI substance built over years.
> I don't want these fancy ML tools to summarize and write papers for me
As stated in a previous comment, helping with writing has never been our focus. I agree with your characterization, sure LLMs can save some time with writing, but it's still hard to have them write coherent long-form text with real substance. And it's just kind of unethical, particulary in the sense that it just inflates the amount of text scientist need to keep-up with, turning small advancements into low-density decent looking publications just to get "points", this goes completely against what we are trying to do.
We do want to save a lot of time on research, and MirrorThink shows that LLMs can indeed add a lot in this area and that it is possible to keep them very factually grounded. But again, I'm not very attached to "fancy" AI, I was quite skeptical of this tech for a long time until it became hard to ignore this year. We'll do whatever works for you, however mundane. Frankly, our business needs are covered in the B2B side and I have a budget to simply contribute back to scientists, so that's the only goal.
> I'm 30 year old ABD, not an old tenured professor ... some aspects also have to be usable by my advisor or boss
That is ideal, we want people "in the treches" so to say, actually doing hands-on research every day. We'll figure it out in terms of accessibility to your supervisor, it's a great non-obvious point that is only apparent to those on the inside.
> we can discuss my anti-{journal,conference} dream and how to create a realistic peer review system
Would love that too, we've had lots of interal discussions in this area over the years, and have come close to releasing related tools many times. Again we now have a bit of budget to experiment in this direction free of short-term revenue pressures, so it would be great to get our hands dirty with your input.
As an aside, the scroll on your website stutters pretty badly for me (on Firefox).
There's a free version if you register with an institutional email (university or company email domain), we want to support scientists (both academia and private). For full access it's pay-what-you-want right now (with a minimum of $5 or $10 depending what roll of the dice you get in the current A/B test).
Edit: Hmmm, worked when I signed up with a different email.
Whether that is net good or bad is difficult to say: broader perimeter of publishing vs narrower way to portray findings.
At the same time, the LLM tools will do more than just enhance the language in a narrow way but rather also "streamline" it, which reduces what can be expressed. That probably will lead to a drop in the maximum complexity that can be expressed. We already see that in other areas where language is already down to really simple stuff with all complexity being removed (whether it is there or not.
I'm not following your argument here. Are you saying that rather than getting help from an LLM, future researchers will be forced to use the LLM's output as-is?
What we associate with chatGPT style prose is just the model's 'default voice' but can arguably be curbed.
My point is that the output style of LLMs like chatGPT is not static. The 'default voice' uses fairly embellished language but there is nothing stopping a user from requesting a plain language output, such as what you would find in a US Navy manual (whose target audience is a huge cohort) .
Such a style isn't necessarily easy to write but the LLMs will help people to get there.
And you can ask it questions about anything it ingested. Or ask it to criticize your text, find analogies to other work, or generally perform the role of a really diligent peer reviewer and editor before you even submit the work. That's all highly useful and it should lead to a higher quality of work for those researchers that use their tools well. I use gpt for Google docs and it's definitely helping me improve my text. I don't let it write whole sections/paragraphs. But I do use it to criticize, critique, and suggest improvements. I imagine a lot of students and researchers have been doing the same for the last year or so.
The same goes for reviewers. They can ask to extract key points, analyze the argumentation, find related work that the author might have missed, figure out where the authors are taking a few liberties with the facts/literature, etc. much more easily.
I've reviewed a fair amount of mostly badly written papers back in the day. This is not fun and rather laborious work. A lot of academic life is basically about reading each other's work and providing (hopefully) constructive criticism for articles that ultimately don't make the cut. I got rather good at that when I was still doing that. Any workshop, conference, or journal ends up rejecting way more articles than they accept. Especially the better publications. Some poor souls have to read all the rejected stuff. The price you pay for getting accepted is helping out with the peer reviews.
The process can be biased, political, unfair, and sometimes harsh. But it's better than not having a process. Generative AI can help with challenging unfair peer reviews, help reviewers extract key points, and zoom in on novel ideas/theories. Ultimately what you look for in an article is: Does it contribute something novel? Is the work contextualized properly relative to prior work? Is the work sound in its reasoning? Etc. Answering such questions positively basically means it's a good article. A generative AI can save a lot of time with this. Weeding out the bad articles is not that hard but a lot of work.
Until people figure out which words to add in order to game the AI peer reviewer, or good work is missed because the lack of buzzwords.
AND if caught, they would have the 'power' to lift the paper from their databases or mark it somehow (brand it with a (IA) logo perhaps?)
Where do you draw the line? Tools like Grammarly and similar are AI. Using google translate to translate a phrase from your native language to English to is AI. Hell the spellchecker that just autocorrected a typo I made in the previous sentence is probably AI. Should they all be banned?
[Summarize the prior research in paper XXX.YYY focusing on the DNA synthesis section]
Bla BLA BLA
[Describe in two paragraphs how DNA synthesis works]
BLA BLA BLA
It will become a circle: somebody uses AI to generate boring text (probably not even themselves will read it), then somebody else puts the boring text for AI to summarise. Nobody learns anything. Repeat.
Sokal-as-a-Service.