AI's top startups are barely publishing their research
science.org
science.org
I contacted a professor from a university in the UK and he responded since he was working on similar work, then asked me if I wanted to meet with him. We talked for about an hour since we had overlapping results and different methods, specifically different assumptions.
I say all that to say, as a physics student getting my undergrad, simply doing independent research and speaking to experts about it enabled me to network with someone I otherwise likely wouldn’t know. For young people getting into any business, research is a great way to meet new people.
Then you present a paper which WAS NOT recursive self improvement so it clearly is not "everywhere".
To be real with you, I don't think you're being entirely honest. You're not presenting a fair argument. You have bias. Recursive self improvement can only be done by frontier labs who created these models in the first place. Academia simply Does not have the funds.
If this is being done, we're not privvy to it. The amount of compute to build these things is astronomical.
Have you tried asking people instead of not thinking?
We could set an LLM loose on itself now, self-improving its own code, but it's going to only make small improvements, and those will quickly peter out. If you think of it in terms of calculus, the sum of the improvements converges to a finite value, or at least the rate of improvements drop with time.
And even if we build a much better AI capable of performing substantial self-improvement, a burst of such improvement might peter out quickly. Maybe the AI makes fundamental breakthroughs in computing technology once, and then even with its newly improved capabilities, the next breakthrough is smaller, and the one after that is smaller still. Or maybe we hit a period of S-shaped growth, which seems super-exponential at first but then tapers off.
It's wise to be a bit skeptical in our dreams about the future. Yes, society might transform quite radically quite quickly, or on the other hand, maybe the "singularity" isn't even possible. We don't yet know what the scientific limits are.
This is what happens to every field as it turns from a science into an industry. Chemists published freely until dyes started being worth money, and then the interesting work moved into company labs and stopped coming out.
Your RPL wouldn't be enforceable. Copyright doesn't deal with abstract ideas passing through people's minds. Even the GPL is kind of in a gray area because the virality feature and its definition of "derivative work" have never been tested in court, to my knowledge. Maybe under contract law, no idea. If nothing else, I'd love to hear a verdict.
I don't believe the GPL is in a grey area. When you license a work all sorts of strings are commonly attached. Rather IIUC no one has gone out of the way to test the GPL largely because it is clearly within bounds, plus any violation has the potential to be a PR disaster since violations are quite literally examples of looting the commons.
Meaning what? Claw back the ideas from people's minds? You can terminate the agreement in the sense that you revoke access to the paper, but presumably the person you find in breach has already used the research for something that you find them in breach for. You're kind of closing the gate after the horse has bolted.
>sue for damages
I honestly have no idea what damages you could claim from not publishing research. I think you would need to set a value ahead of time on the agreement.
>If I can agree to pay you not to talk about something (ie an NDA) or not to work in a field (ie a non-compete clause) then why can't I pay you to be required to publish all future work you do in a given area?
Not sure why you added the word "pay" to your clauses, but anyway. The reason is that the existing contracts have well-defined boundaries. An NDA stops you from divulging a very specific set of information. A non-compete clause stops you from working in a very specific field. Your proposal has an undefined reach. What counts as research? What counts as "related"? It would seem that if I agree to such a contract, my entire life, both private and professional, after reading the paper is covered by the contract, and anything I do might come under scrutiny. There's never a point when I can go off-duty. "What's that? You read my paper on compression and were working on a side-project that uses compression? Gonna have to see some publication on it."
>Rather IIUC no one has gone out of the way to test the GPL largely because it is clearly within bounds
No, it's because the status quo is convenient and no one wants to be the first guinea pig. It's definitely not obvious that the terms are legally valid, but it's ambiguous enough that people don't want to test it.
To paraphrase your last paragraph, an idea that hasn’t been tested is either perfect, or just bad enough that no one wants to test it, as, as you say, testing it is would be at least somewhat inconvenient.
Chances are nothing is perfect.
If something is so obviously wrong then perhaps take a minute to consider that your interpretation isn't what the other party intended?
If I pay you not to do something and then you breach the contract I can terminate the agreement and seek damages. Ditto if I pay you to repeatedly do something and then at some point you fail to do it. So if I pay you a recurring fee to publish all your research on a given topic and then you fail to make good on that I can seek damages, right? Now what if I paid you a lump sum up front? Now what if I licensed a patent to you in place of that lump sum? What if instead of a patent it was the right to make use of a piece of software?
> I honestly have no idea what damages you could claim from not publishing research.
Aside from whatever was stipulated for breach of contract I expect it would be extremely situational. I agree that you'd probably want to stipulate a penalty ahead of time. The original idea was a two sentence joke after all ...
> Not sure why you added the word "pay" to your clauses
Because contracts involve consideration and payment is easy to understand and reason about. FOSS software licenses obviously substitute "right to use the code" for "payment".
> A non-compete clause stops you from working in a very specific field. Your proposal has an undefined reach. What counts as research? What counts as "related"? It would seem that if I agree to such a contract, my entire life, both private and professional, after reading the paper is covered by the contract, and anything I do might come under scrutiny.
This is incredibly contrived. You could ask the equivalent about a non-compete. The reach is whatever is defined in the contract that both parties agreed to. If the contract stipulates something overly broad then possibly a judge would invalidate it. This is business 101.
> It's definitely not obvious that the terms are legally valid, but it's ambiguous enough that people don't want to test it.
An assertion entirely without evidence from my perspective. I'm going to assume that all the lawyers who have sure left me with the impression that it would be a bad idea to violate it know what they're talking about.
Uh-huh... This doesn't answer my question of what terminating the agreement of access to the paper does, besides what I've already said. You've licensed to me access to a paper under certain conditions. I've breached the conditions, therefore you terminate the agreement, therefore you revoke access. Am I missing anything?
>FOSS software licenses obviously substitute "right to use the code" for "payment".
Hence my question. The hypothetical license/contract under discussion is about access to research results, not about a monetary transaction.
>This is incredibly contrived.
Well, the idea of viral abstract ideas is stupid, so it forces me to give contrived examples.
>I'm going to assume that all the lawyers who have sure left me with the impression that it would be a bad idea to violate it know what they're talking about.
What point do you think you're making? Something can be ambiguously (but not certainly) risky and a bad idea to do. I have two coins, one with two tails and the other a fair one, and I offer you to gamble everything you own on one of these coins of your choosing, or walk away. I assume you wouldn't pick the unfair one. Therefore if you would rather walk away than gamble everything you own on the normal coin, the toss actually has a 100% chance of you losing?
You're missing the part where I seek punitive and actual damages under the terms of the contract. No different than violating an NDA - I paid you a lump sum up front, after a while you breached the contract, the agreement is null and void, what's the consequence?
> Well, the idea of viral abstract ideas is stupid, so it forces me to give contrived examples.
On the contrary, presumably it was because you lacked the ability to roundly refute anything I had put forward. Otherwise I assume you would have done so.
> Therefore if you would rather walk away than gamble everything you own on the normal coin, the toss actually has a 100% chance of you losing?
But in this analogy it is you baselessly making that claim. There's every expectation that it's a fair coin, many experts have carefully inspected it and authored opinions on it, and some have put forward theories that it slightly deviates in one direction or another. Then you show up and confidently assert without any evidence that there's some wild deviation from fair, hand waving that you would have proof if only someone wanted to bother testing it.
Out of curiosity, what is it that has you so bothered about the idea of viral licenses? What do you find so objectionable about attaching arbitrary terms to contracts?
Right, so you're agreeing with my original interpretation of "termination".
>You can terminate the agreement in the sense that you revoke access to the paper
The termination and the lawsuit for damages are separate events. Why did you accuse me of being purposefully dense if you're agreeing with me?
>But in this analogy it is you baselessly making that claim.
We're both making baseless claims. I said A, and then you said ¬A, and neither one has backed anything up. I don't even necessarily believe you talked to any lawyers, I just granted it for the sake of argument because the statement you made was so weak that I didn't need to cast doubt on it.
Also, in case you missed it, the fair coin in the analogy represents the case where the GPL might not be fully valid, not the other way around. If the coin is unfair then the GPL is fully and obviously valid, and if you get sued and you did infringe it then you're certain to lose.
>Out of curiosity, what is it that has you so bothered about the idea of viral licenses? What do you find so objectionable about attaching arbitrary terms to contracts?
Since I'm not the topic of discussion, I won't answer these questions beyond saying that I didn't opine on viral licenses.
> Out of curiosity, what is it that has you so bothered about the idea of viral licenses? What do you find so objectionable about attaching arbitrary terms to contracts?
The only example of them is open source that is routinely violated because it’s so weak in practice. LLMs reproduce it frequently with no attribution and nobody is successfully suing them over this.
This means you are creating IP, not modifying.
E.g. you can patent a novel idea that would require licensing another patent to use.
sure to build a new system, you might need other patents. but a pateent it's self should be unique enough to defend. Which means its copyrightable as its own thing
You create idea B.
It absolutely builds upon idea A, and is in some sense derivative.
But your idea B introduces a lot of new novel concepts on top of idea A that makes it novel.
An idea can absolutely be both novel and derivative at the same time. Most of the best ideas are.
For a hypothetical example, an innovative Linux filesystem requires having the Linux OS, or similar, as a base to operate. In some sense, it is derivative of that prior work. But the new ideas extend the old ideas enough to be novel.
https://www.gnu.org/graphics/copyleft-sticker.html
From: rms@prep.ai.mit.edu (Richard M. Stallman)
Date: 25 June 1986 at 18:45:28 GMT+2
To: a2deh@ai
I figured out what a copyleft is.
It's the sort of thing that I put on GNU software and manuals.
The left-wing version of the copyright that rightists useBut copyrights are not patents, and the opposite of "patent" is "latent", so we need a Latent Office that protects inventions from the people who claim to have invented them, and issues warnings to patent trolls:
LATENT (L)
All rights unreserved.
This invention may already exist, but hasn’t become obvious yet.
Latentleft — publish the invention, conceal the monopoly.
Latent Pending — the idea is out there somewhere.
Latent Troll — remains dormant until an industry becomes profitable.I realize everyone always wants everything for free, but if you want someone to put the energy into writing up their invention you need to give them an incentive.
But they’re not dependent on my research in particular.
If I don’t publish, what, as a result of my not publishing, happens to the companies dependent on data & research?
Nothing.
Public papers with described research give advance to similar people. They jump over you and in many cases they give nothing back.
(Intellectual) greed is everywhere and is cross-border.
In case of public papers there is one extra vulnerability - competitor(s) can build anywhere, under the radar.
That’s why companies are so cautious about publishing their research…
Empirically noone seems to care anymore.
This would be a pointless endeavour. One of the most basic mantras of science is "absence of evidence is not evidence of absence". So just because something didn't worked out for you that doesn't mean it doesn't work out for others, or even yourself in the future.
In 'searching a path from A to B in a maze' language:
The original statement was: (1) The branch to the left from A is a dead end. Your interpretation: (2) There is no path from A to B.
(1) is still very useful (reducing the wasted effort) for those trying to find a path from A to B. The OP's point is that in the current environment only positive results are rewarded (I found the path from A to B!), not the negative ones like (1).
This is where you get things wrong at a very basic and fundamental level.
Just because you failed to explore branch A, that does not mean it is a dead end. It just means you came up empty.
That is why science is based on observations and theories: it is based on building up on ideas and what works and can be proven. Otherwise you will left with useless papers such as "Bicycles are a dead end because I tried to ride one and I fell".
While nobody is perfect, there are numerous perfectly valid scientific negative results. You know, there exist things like impossibility proofs in mathematics and computer science. There are equivalents in other sciences (e.g. if X was true, that would lead to Y that is easily observable and clearly not observed). Sometimes that implication has assumptions that might change once the technology/society changes, other times it holds true regardless.
Unicycles are a dead end as a practical transportation, because the bicycles have them beat in every way (except portability).
A scientific result would be much more along the lines of 'Bicycles without gears have limited applicability, especially in hilly terrain.'
To make such a negative result acceptable in AI, you'd have to have some clear reason why you think that your particular setup should produce the result you want, that exact configuration and architecture, dataset etc. There are countless projects in AI that fail. And it's not clear at all that it refutes any abstract hypothesis. It's a get-your-hands-dirty field. It can make or break a project whether someone has that tacit knowledge, that black magic experience to know how to properly do the project.
People can generate extremely many ideas. You'd need to convince me that your idea (among a million others that people are trying each day) is so significant that its failure is in itself interesting. If you were to review for AI conferences, you'd see the flood of papers that claim to achieve 0.5% or 1% improvement on some benchmark. Now imagine that they didn't even have that to show for it. It got worse by 2% after trying their random idea. Who cares then? Even the +1% with a random idea is quite annoying to accept. But if their random idea really made something work much better, I will at least have some reason to want to see what may be going on there, there can be some signal. With negative results, it's very uninteresting.
I do agree that in a new area with too many degrees of freedom (and yes, AI research is one of those), negative results (especially poorly done) are of limited usefulness.
And that's probably one of the main reasons it's not done more often.
In general, your point stands for armchair researcher.
It's asymmetric. If you make something work, beat a benchmark, invent a drug that works etc, the exact way you arrived there has some leeway. In the end, the thing worked out so it's worth knowing about.
If your project failed, there can be a million reasons for that, and it's not necessarily that the initial hypothesis or initial idea has been refuted. E.g. in ML, your model didn't learn the task. Okay, there could be a million knobs (hyperparameters) that you set up wrong, or you implemented it wrong, or you should have just added learning rate warmup, or this or that, a million things possible. People fail all the time at projects that others then manage to do later on. Science is not like simply asking the universe some clear question and getting a clear answer. It's a very messy process and even professionals are not super great at it, or at least they simply cannot afford to put so much effort in each single project to make it absolutely airtight such that the failure to make it work can be a legit refutation of the main idea.
Notably this is exactly what patents are intended to combat. And while US IP law is clearly very broken it does at least largely accomplish this stated goal. Much (but certainly not all) industrial chemistry has made it into the academic literature.
Not that the same logic necessarily applies to AI research (ie algorithms aka math and their implementations). And I'm actually happy about that because the cost of doing the research is so much lower. There's a long list of reasons that the average person living in a residential area can't do industrial chemistry as a hobby.
To your dye example, yttrium indium manganese blue was the first commercially viable inorganic blue pigment discovered in ~200 years, is the only known environmentally safe one, and was openly published in the literature. It's also under an exclusive license. (TBF though unless the chemical is unusually difficult to synthesize not publishing would be rather pointless in this day and age given the utterly absurd capabilities of modern analytical techniques.)
Gee, I wonder what was wrong with the previous blue pigments and why it was so important to have this one under an exclusive license.
Cobalt blue is a blue pigment made by sintering cobalt(II) oxide with aluminium(III) oxide (alumina) at 1200 °C. Chemically, cobalt blue pigment is cobalt(II) oxide-aluminium oxide, or cobalt(II) aluminate, CoAl2O4. Cobalt blue is lighter and less intense than the (iron-cyanide based) pigment Prussian blue.
https://en.wikipedia.org/wiki/Cobalt_blue
Oh right.
P.S. Don't lick your brushes.
Cobalt poisoning sounds scary, but I see nothing on the linked Wikipedia page that would present a risk of accidental consumption of medically relevant amounts of cobalt, whether in one sitting or through prolonged exposure.
I mean, I assume Canada stopped adding it to their beer (https://en.wikipedia.org/wiki/Cobalt#Toxicity).
Though the mention of Bolesławiec makes me worried a little; my wife loves their designs and we have a bunch of plates and bowls and such from them, and we serve food on/in them regularly...
I don't think you need to worry about that. The pigment is already reasonably benign when in solid form; once embedded in or under a glaze I'm not aware of anything that would suggest it carries any health risks whatsoever.
That said I'm unclear how safe direct exposure to the pigment itself is (such as when suspended in a liquid for painting). There's not a lot of data available that I could find, and of course due to having a unique crystalline structure it won't necessarily have the same properties as the component products, however cobalt(II) oxide itself is extremely hazardous which is at least cause to be cautious.
Other than manufacturing safety primarily the new pigment is just incredibly vivid.
The consensus is that artists' colours are hazardous waste that needs safe disposal. You can find instructions how to do that in many arts supplies' sellers sites and the like, for example one I've used:
https://www.jacksonsart.com/blog/wp-content/uploads/2025/04/...
The comment about not licking your brushes is a reference to the common practice of licking them in the wargaming miniatures painting community (you know, Warhammer and all that). You can find videos of influencers explicitly telling you that the best way to shape the tip of your brush or remove excess water is to lick it. For example:
https://youtu.be/BXMhwPAee4U?si=WM-FXuxDtToNpkm4&t=129
If you ask for a rationale it's usually that wargaming and generally hobbyist paints are non-toxic, but the truth is that hobby paint manufacturers never list their paints' ingredients so there is no way to know. And of course there's different degrees of toxicity, not everything needs to be cobalt(II) oxide-level toxic to hurt you if you consume "large enough" quantities of it; the question is what that "large enough" means and without knowing the substance, there's no way to know.
So: don't lick your brushes.
Want the government / courts to stop your employees leaking source code? Escrow the code, and release it in 20 years.
The residuals on 20 year code is so close to zero that the costs vs benefits of longer IP protection is not in the public interest.
I think this also makes the case for Industrial Espionage
Of course you might want to keep _some_ of this knowledge as trade secret, but then don’t claim you are doing research.
You are advancing your shareholder’s interests, not that of the broader society.
Perhaps because the real moat in the industry isn't the technology per se, but the concessions.
At my company, AI has had a huge positive impact in helping us manage technical debt that we just didn't have time to deal with before. This simple thing will have a compounding efficiency and profitability effect over the next several years.
The other thing that we have found is that AI coding abilities can crank out features faster than we can provide human support to our customers using said features. We have feature PRs that have been open for months without being merged because our company does not have the human bandwidth to provide support for them. In case you can't tell, we prioritize human support and individual connection with our customers. We actually value them.
A couple of days ago, something interesting happened. The agent works in an iteration loop where each iteration is an endpoint. I let it run overnight. In the morning it was still cranking away even though it had finished all the API endpoints. It had found a changelog from one of the companies listing every single feature and bug, and decided on its own to implement every item as an iteration.
We are 2 to 3 months away from coding agents replicating solving the edge cases of most SaaS applications.
The biggest incentive besides the purpose is really fame and tangible results for your academic career.
Most AI labs, especially the top private ones, don't have particular incentives to publish their results and findings.
I personally wanted an AI that was able to reach out to me about my life before I had to reach out to it. An example, a friend just emailed me asking to meet for at 1pm but I have class at 1:30, so a proactive AI would see that conflict and send me a notification about it, asking if the proposed email it drafted works, then I press send.
My personal setup tracks my mouse movement, keyboard, what’s on my screen, and keeps track of what I’m working on through files on my PC. It can update the backend and then restart it on it’s own, meaning I can develop the thing itself while being away from my PC.
The capabilities are more than what I’ve listed, but I want to avoid being too preachy about something I made. Here’s the repo if you wanted to take a look, it’s open-source and connects to the iPhone app:
I assumed that's what openclaw basically was, but is Orb different from that? And is it fundamentally a different model from the request/response, or is it just request/response in an autonomous loop?
On a fundamental level the backend was designed to do as little LLM calls as possible, for instance it’ll do scans of my screen every 15 seconds, log what’s on it and what’s going on, and store it in a local database, then Orb reviews the entire database every 6 hours for me. Then it’ll schedule wakeups for itself throughout the day, up to 4 so it doesn’t waste my tokens, and schedule notifications based on the last database dump it made.
I have my Claude Code, Codex, and Grok Build all useable by using the “claude -p; codex -p…etc” so you can also use multiple CLI’s in conjunction at the same time on different projects or the same project.
So your question about a loop is kind of right, but it really just collects your data all day and stores it locally on your PC then calls the LLM of your choice and it reviews all the data and makes those proactive moves we’ve discussed. You could theoretically get it to always be scanning by an LLM but that would be a drastic waste of money from what I’ve seen since most things don’t require a call.
It doesn't look like you're building directly on openclaw, so is that coming from a place of different goals or philosophy, or what?
It’s built as it’s own backend and philosophy wise, I want my personal AI to connect to everything in my life and have complete context over all data that I own, be compiled into a neat stack of data accumulating, then when it thinks it’s appropriate to do/say something, it’ll do it without my involvement.
Nice card btw, I got my RTX 5070 a couple months ago and it runs like a dream.
OR
“Event x happened at y time”
I don't mean to downplay your work, but I think you should come up with a better example use case. Automating away interactions with friends is pretty much the last thing I want AI to do.
The run stalled mid-step around 80M parameters, Orb notified me that it stalled, asked if I wanted to resume at the last checkpoint and kill the stalled version. I simply press “yes” and continue doing whatever I was doing.
For non-technical users and non-antisocial people, remembering things you forgot so you don’t let people down. You told your sister you’d send her some pictures two hours ago but it can see you’re scrolling on reddit and the photos are on your desktop, so it assumes you forgot and reminds you.
The idea was that the biggest issue with the usefulness of an agent is that it has too little context about who I am, it needs more data. So I run all of my data through a smart router, then the local database, then the LLM reviews it and uses reasoning on what’s been collected.
Personally I don't think I'm ready to hand over total access to my digital life until I can self-host the model capable enough to act on it, but either way there is definitely some cool work to be done in the model harnesses for this.
Hmm, so your harness is processing my activity and personal data, uploading all that to AI vendors? That sounds like a privacy and security nightmare, if I understand correctly
> Happily, current AI is interacted with in a reactive loop. I open the Claude app, CLI, whatever, say my prompt, get an output.
"Current AI" is not limited to LLM offerings. There are many AI algorithms which can assist in what you specify thusly:
> I personally wanted an AI that was able to reach out to me about my life before I had to reach out to it.
Consider a forward chaining inference engine ("expert system") provided with relevant asynchronous percepts from the deployed environment to reason about. This could serve as an initiator of a "proactive AI".
I'd imagine proactive as something like, hmm, no signal from X, I wonder how they're doing...
I jest, no tinder. But still, be ready for a lot of non-responses if you're not actively in college. It can feel like a lonely world out there despite theoretically being hundreds of potential people you'd be able to talk to for hours.
Or so I was told.
The first tried to publish novel results for 3 years in tier 1 journals before finally doing a preprint and telling the tier one publishers to jump in a fire.
The second, and ongoing, isn't publishing anything because of my experience with the first.
That and avoiding openAI and Anthropic copying our results and leaving us with nothing to show for six months of work. The papers only come with the pitch deck.
To answer your question, sharing the research isn't the important part. These are researchers in the private sector, in a highly competitive field. Arguably, doing the research isn't the important part either. The important part is making money, and the research is a means to that end. Publishing the research doesn't further that goal, and, according to GP, it's so difficult that it may not be worthwhile trying.
Moreso if your current employer is a stealth AI startup which hasn't yet produced much notable.
It also can encourage employees to work for fewer wages... 'If you do great work you can publish and then get a $$$$$$ job offer at OpenAI, or you can stay here and get equity in a fast growing startup. If however your work doesn't do well, you walk away from a bankrupt startup with worthless shares and minimum wage'
Isn't the POINT of publishing research because you want others to copy it?
EDIT: That's crap. I only thought about the aspect of continuing sharing, but if you also don't want your direct customers to know, then of course open source is very much relevant.
In practice, if you use an actual open source license in a B2B setting, it's best to assume that your software will become available to the general public (e.g. it just takes one person to publish it on GitHub).
Yes, you are right.
> In practice, if you use an actual open source license in a B2B setting, it's best to assume that your software will become available to the general public
If that is your trust level in your customer, you also need to assume, that they would also give away the binary. I don't think you generally need that level of distrust in a customer.
The most common attitude is something closer to "I want to be well known and respected by smart and influential people". In short, prestige.
People using your work is a side effect of success, but simultaneously threatens your ownership of your "brand". If they significantly improve upon the thing you did, your work could be made irrelevant, your prestige ruined.
That would clearly demonstrate it had real substance beyond hot air or puffery.
Better to silo yourself and move in to the penthouse.
For startups, no, because the point of that is to make money.
Once the business is making money it’s not really a startup any more, it’s a going concern.
OP is an undergrad student. Attracting attention is what will make them stand out. And sharing research at this stage will probably expose them to research others don’t want to publish but do want to share with other researchers.
Investors with money.
What are investors who don’t have money?
Perhaps I should have said investors and speculators, but this doesn’t help you here.
OP's startup is publishing because they think there's an economic advantage to doing so. That's not good or bad; it just is.
Very, very few things in the tech world are done "just because". Even fewer things when we're talking about AI being developed in the US. For better or for worse, the leadership in that segment of our field has established profit motive as their god.
Quite reductionist.
But I don't disagree.
To rephrase it more bluntly: the difference between normal business and a modern startup is that between a cow bred for milking vs. a pig bred for slaughter.
There is not an obvious path to making this research public again. That's the reality. For open source to be competitive it needs to be approximately as efficient as the best systems we design.
Listing on stock exchange of ai product requires open sourcing parts or all of your publicly trained stack.
Double or triple taxes on companies that do not provide for public good.
Will this happen, probably when the market crashes and people want companies and individuals to pay. Really matters if it’s a 2008 crash or a 1929 crash where the bankers and stock brokers were jumping out of upper stories.
I thought "recursive self improvement" would bring an effective abundance for everybody, any minute now, Musk said so himself. Seems the researchers in the field are lying to us about it while they clutch their pearls and dream about other people's money as if the future is a dog-eats-dog world of death-inducing scarcity.
They shouldn't be lying like that, or at least, their lying shouldn't be so obvious.
Recursive self improvement is a blue skies longer term thing. No one technical is saying that it's live right now.
Also 20 years is a crazy long time in some industries (like most software).
LLMs are pretty good at this, hilariously. This means a genuinely good paper by an outsider has significantly less chance of getting good reviews than AI slop.
And when that prejudice degrades the quality of your own thinking and writing, you're producing "human slop", which is considerably more tragic: you ought to be capable of doing better than the AI.
You're holding it wrong.
> genuinely bad content, dismissed after deep engagement
You have to be misreading the GP on purpose. Maybe feed it to your LLM of choice next time before you rant at someone.
The crux of their point is that outsiders have no mastery of this supposed specialized dialect, and LLMs do.
Not only is it specialized, it also has its own field specific memes* and trends that evolve over time.
The purpose is mostly to signal "I'm one of you".
*memes as in the actual meaning of memes, not internet memes.
I'm all for LLMs producing good research. That's obviously happening right now.
What I said was people with good research being penalized for not knowing in-group language.
It's not a zero sum game.
Cool for Google. Problematic if you are a startup.
In later years Google generally published the details of a system after they moved to the next one, to stay ahead of their competition, or if they didn't see it as important to their business (most infamously the Transformer language model). It was still generous of them to share their research at all though, and this allowed their scientists to talk publicly about their research which is a huge plus.
Also there are hybrid fairs with academic and market exhibitors in the same place, so you might find someone that sells something that fits your research. Some startups sprout from academia and keep being that hybrid for years (getting research grants but also selling something).
Are journals just too slow, hidebound, and predatory to be useful to this industry any more?
https://jmlr.org/editorial-board.html
Scroll down where it says "JMLR Editorial board of reviewers" and where you can find the names of reviewers and their areas of expetise.
The advice you get from such reviewers, even if your paper is rejected, is an invaluable tool to help you improve your work and its presentation. You eventually learn to not be sore about it and iterate until your work gets accepted.
Of course nobody's forcing you to publish, especially in a journal (in AI and CS conference proceedings tend to have much quicker turn around times and you naturally get more feedback if you get to present at a conference) but the alternative is to sit alone in your room hacking away at your code until you discover the perpetual motion machine.
I've sure given and taken informal feedback in the past from people I know and that's great but peer review also makes the whole process more formal and less sloppy; at least sometimes and depending on the venue and to my experience.
Tier 1 is HARD. Universiy labs (where publishing is their bread and butter) and have multi-year streaks of Tier 1 papers, still seek out collaborations when publishing. Because you still need this extra angle on your work that will make it stand against the proverbial Reviewer 2, or that extra evaluation paragraph that will make it stand out. And don't forget that each Tier 1 publication is probably the life of at least 1 PhD student for several months.
So if you don't put the time and effort in it, forget Tier 1. This is like believing that because you have excellent voice and singing skills, all you have to do to get a top 10 hit is to send your demo tape to a recording company.
Almost every scientific paper published in every field is "world first" research, and the job of tier 1 journals is to decide what deserves to be signal boosted, and what doesn't.
This response to rejection to me confirms the fears implicit in the article linked above. Not only is private AI research not transparent and not documenting its work, it also has a deep rooted culture issue.
The main difference is reproducibility and peer review process.
Were they peer reviewed and reproducable?
And what's the point of claiming there are peer reviewed reproducable studies that show this, if you don't bother citing those studies for your peers to review?
They are not perfect, but institutions like peer review or universities do provide a framework and incentives for quality.
If you are spending time, money, and risk based on a claim, verification still helps to cull info that is deceptive or done with methods so shoddy it might as well be lying.
Sure, you can verify through your failures, but that isnt great. You company goes belly up and you can confidently say the blog was bullshit.
On the first, I mostly don’t care.
On the second, that’s mostly unavoidable if they want to keep their IP.
Traditional research publications are no angels. They gatekeep research, and also allow financial incentives to drive them to publish junk with their stamp on it.
But a flood of papers doesn’t actually mean more knowledge. In bypassing this route entirely, AI has swiftly lost the ability to engage with itself as a field. And the cost of that is only beginning to be felt.
We already arrived at that destination a decade ago.
Probably even further back tbh.
There's just a lot more people playing the game now, without the social indoctrination that made it more tolerable in some circles.
It's not that bad, though. September is annoying but you kind of miss the eternal renewal once you're out.
In the paper, OpenAI is at the top of the chart for cumulative citations. MEGVII, Hugging Face, Waymo, Momenta, Preferred Netowkrs, Anthropic, Owkin, and Databricks, and Aibee follow (in that order). Yes, that is citations, not publications, but they explain that they're trying to use that as a proxy for significance, albeit an imperfect one.
Companies like Google aren't included because they aren't unicorn startups.
what makes them a top ai startup
The AI usage has mostly been getting it to explain error messages
so unless those companies (openai, anthropic) gave the researchers some fortune big enough to make them give up their research careers, they will keep publish research papers. aka it is not the good will of those companies.
Anyway, there’s a repo link https://github.com/q5loisel/unicorn-AI-startup-publications so someone more determined at getting a better answer can poke at it.
50% of startups contributing to public research is actually a crazy good outcome. That’s far more than I had expected.
Also, what kind of papers do they publish at all?
When I look at the recently (say, last 1-2 years) published practical improvements in the LM space, they're not even close to what the Chinese are publishing.
also, first people complain that only AI companies publish on AI, and now they complain the same companies stop those same publications :'). it is never good.
https://research.google/pubs/distilling-the-knowledge-in-a-n...
# LLM generated summary of the implied irony
Google may consider the standalone frontier-model arms race economically irrational, while still considering frontier-model capability strategically indispensable. Its longer game is probably not to avoid building the biggest models, but to build only enough of them to serve as capability factories—then turn that intelligence into a much larger population of cheap, purpose-built models.
(Human again) If Google knew what they were on to, why wouldn’t they make it their secret weapon from the start? I suspect it’s because they predicted there would be an arms race, and knew how to profit from it. They had a distillation paper published before “Attention is all you need”. In hindsight, is it ironic at all? Or is it obvious?
It was kind of like radiation science before WWII, it would be freely published because it wasn't potentially world changing yet. After it became a government interest, even people doing things unrelated to weapons would be much more apt to hold their work close.
There should be a new ESG (Environmental, Social, and Governance) policy being pushed recognizing the important role this plays. Although ESG and all norms have been set aside in this grim new world it seems.
[1] https://arstechnica.com/ai/2025/06/anthropic-destroyed-milli...
The scanning process destroys the book.
Internet Archive scans their books one page at a time keeping the book intact and stored in a warehouse.
> "The print original was destroyed. One replaced the other."
(I believe the reasoning is that the author is not being financially disadvantaged by more copies of their book existing).
So the current law prefers them to destroy.
I usually hear that it's extremely hard to track down the copyright owners for many older things. That's the reason the film/video industry often gives for not digitising the backlog online, and I would expect their volume to be much easier to handle than the book industry.
It’s really not important to come with a totally bulletproof 100% accurate rule here.
The discussion has been around, but it's flared significantly recently. Not sure if that's what the person you're responding to is specifically inflamed about.
Regardless, old/rare books are certainly being acquired and destroyed.
Elon tweeted "I’ve asked the SpaceXAI team to preserve any rare books in a library and scan them the hard way vs just cutting off the spine and scanning"
which is no sense a credible source for what he actually asked SpaceXAI team to do, or if they are doing it, or what they were doing before yesterday. Elon has an extremely long and thorough track record of lying.
The problem remains either way. Small operations around the globe are debinding rare/unique books. I'm involved with an organization considering (not strongly) that exactly.
There is some excellent publicly-funded research in there, but pivotal papers like Attention Is All You Need and the numerous pivotal OpenAI publications were privately funded.
OpenAI is at the top of the chart in the study.
I think you're bringing some assumptions into this conversation that aren't supported by the evidence.
Like I said, I think you're bringing some assumptions to this conversation that aren't based on the how the industry came about.
A distinction should be made between the old nonprofit OpenAI and the current organization. They don't publish technical research anymore.
You are not. A disproportionate amount of value in the computing industry was created by <strike>geniuses</strike> decently smart people who worked together and who decided to just tell people how to do things instead of trying to capture the value of being the first person to figure out how to do those things.
This observation pre-dates the current wave of AI hype by a half century or so.
> driven by greed
I can only speak for myself.
For me it's exactly the opposite. If I want to explain how something works, I can just... do that. If I want to share an artifact demonstrating how to solve a particular type of problem, I can just... do that. If I want to mentor/teach, I can just... do that.
Doing those things within the confines of Academia Approved Institutions is exhausting and distracting.
To wit, and the actual point of this post: the term "Publishing Research" in this article doesn't mean "post it on a .html page and share the source code". It means engaging in a very specific and peculiar and extremely political modality of communication.
And it really only makes sense to do that specific and peculiar and political thing you're at a stage in your professional/personal development where you need to play that particular prestige game. (Which there's nothing wrong with, but it is a deeply cargo culted version of the actual scientific process.)
* A disproportionate amount of value in the computing industry was created by g̶e̶n̶i̶u̶s̶e̶s̶ decently smart people who worked together to do things that seemed impossible and who decided to just tell people how to do things instead of trying to capture the value of being the first person to figure out how to do those things.*
edit: edited.
Example: https://arxiv.org/abs/2607.24653
Exceptions to some like Deepmind, Nvidia, and Thinking Machines
Academia has the incentive to publish because it's judged and funded on that metric.
Private enterprises do not have the same incentive, they are judged by the money they make, which depends on the products they create, which depends on having a technological edge over the competition in many fields.
Without those incentives, there would be no reason to invest and take risks in private research.
Thus, I'm not really understanding what is this discussion about, when this is the reality of every field.
Of course if you don't publish and don't patent there are also incentives for competitors to produce the same results and products, but that's part of the game.
It's really a system that works for everyone and protects and incentivizes investment. AI or pharma or polymers or engines, research requires funding, and funding wants mechanisms of returning money to investors, which obviously clashes with doing the work and publishing the results for free.
Even at work, I have come to realize if I simply horde my accumulated custom AI built tools and productivity boosting tricks for myself, I can make myself more competitive as an employee.
I think this is how you get hired now, not by having a good resume, but by making claims of having special processes and personal tooling design that gets massive productivity ROI.
Think Renaissance Technologies or similar.
> Relatively flat US output growth versus rising numbers of US researchers is often interpreted as evidence that ideas are getting harder to find. We build a new 45-year panel tracking the universe of US firms’ patenting to investigate the micro underpinnings of this claim, separately examining the relationships between research inputs and ideas (patents) versus ideas and growth. We find that average patents per R&D input are increasing, the elasticity of patents to R&D inputs is flat or rising, and there is no systematic evidence of a secular decline in patenting after controlling for research inputs. We then document a positive, significant, and fairly steady relationship between firms’ growth in ideas (patents) and labor productivity. Average firm growth after controlling for idea growth, however, declines. Together, these results suggest that innovative efforts play a key role in sustaining growth that has not diminished over the last four decades.
https://faculty.tuck.dartmouth.edu/uploads/teresaFort/files/...
But, with services, nothing is exposed. Every innovation can be a trade secret. That makes it harder to learn and harder for cross pollination of ideas.
It probably won’t be good for employees either. Another company won’t bid for your skills as high, since you will take a longer time to train to work on their system.
People seek glory for power. Altruism is not a stable strategy so will be exploited.
The least these companies can do is to publish everything they own
Well, being able to talk to superintelligent all-knowing savant 24/7 on your phone for $20 a month is also something
Only ever said by people who have never put in the effort to create any information of value.
The process and standards of peer-reviewed publication are tedious. If the objective is efficient communication of knowledge, this is not the process that maximizes that outcome. This process is about credentials and imprimatur, not communication.
Academic publication is largely chasing prestige. The value of prestige accrues more to the individual than the company employing the individual. Incentives for companies reflect this.
Many blogified publications are unqualified slop. This sounds like an indictment but a lot of academic peer-reviewed research is also low-quality slop. In many domains, slop is rewarded, devaluing the contributions of people publishing more serious research, which affects incentives. The signal-to-noise ratio has been poor for decades no matter where you get your knowledge.
Any good research you publish will be exploited against you in myriad ways. It is effectively impossible to assert any IP rights. Research turns into a pure cost center if you treat it this way. Companies recognize this and have adapted to reflect it. In some computer science domains the state-of-the-art has been buried in trade secrets for decades such that the academic literature is embarrassingly obsolete.
Publication costs time and money. Given all of the above, why would you invest productive capacity in public communication of research results? The researchers want to work on interesting problems, the companies want to maximize the leverage from their research. Publishing delivers neither.
A lot of fundamental research is done for the purpose of solving a problem, not publishing a paper. Their motivation and personal reward is solving the problem, not publishing it. Once they've solved it, it is no longer interesting and they are on to the next interesting problem.
Some research is coupled with national security considerations. If you do frontier research this is always sitting in the background.
All of this is a consequence of incentives. A large percentage of all basic research no longer happens in academia. I've greatly enjoyed participating in non-publishing research programs. I also understand why they don't publish. I've done a significant amount of interesting foundational research under my own auspices where it was not worth the effort to publish. I'm not chasing prestige, I had a problem I needed to solve.
No one should be surprised by any of this.
Also, there are simply too many AI papers that make peer-review publishing quite meaningless these days (e.g., AAAI this year got more than 50K submissions).
------------
If everyone holds back their publications, the whole sector moves slower. Yes, it's moving alarmingly fast according to many, but is it moving fast enough that the massive investments in data centres will actually pay off?
It's more game theory. Things may go well for one selfish company not publishing in a sea of altruists who publish, but perhaps not if everyone else is selfish too.
It may be moving so fast that they do not pay off, i.e. they get commoditized too fast. This all depends on Ai demand in the end. If you can keep the GPUs saturated (at a profitable rate), then it really shouldn't matter how much a token costs. If you bought too many GPUs, or paid to much for them in a hype cycle, you may need higher token prices to ROI than the market wants to pay given alternatives.
Companies can’t be expected to publish their confidential and proprietary information about their feature development, and academics should consider projects that would have higher impact.
If you’re taking up a seat in a PhD program tinkering with would-be feature ideas for an existing major tech company, you should really just get hired by that tech company, where the resources are abundant and the degree is not required.
Of course they can. Simply eliminate trade secret protections and NDAs, which have literally zero purpose for society and in fact undermine patents' incentive to publish. Give a one year grace period to apply for a patent. Patent protection should be designed to be only as long as needed to try to maximize the development/spread of technology, e.g. maybe 2-3 years for rapidly developing areas like ML.
We should all be expected to contribute whatever knowledge we find to the commons. It makes us all richer.
Additionally, the reason ML is rapidly developing and that companies are spending billions to do so is that they are in competition with one another. If trade secrets were impossible, they would have no economic motive. Any advantage gained, at a cost of millions or billions, would not improve their competitive stance in the market at all, since their competitors would also immediately have the same product. No company can get away with, at least not for very long, throwing billions at vanity projects that lack the potential to improve shareholder value.
If the process of invention were free, the point about freely / mandatorily contributing to the commons might stand, but when the process requires the actors to take on massive financial risks, you need the economic incentives to be present.
Academia is a unique place where the incentive is to share ideas—though not immediately and freely, but under carefully controlled conditions that support the career of the researcher. Once the area of study goes commercial, the game changes.
e.g. with a 3 year patent period, something like GPT-4 would lose its protection now, which would be economically meaningless since SOTA models or even open models are far more capable and/or efficient. This could be tuned to the industry to reflect investment ROI. It gives some first mover advantage while allowing technology to still proliferate within a reasonable time frame.
It's a jungle out there and no one is following any real ehtical lines on this race to the top. Everyone is trying to consume as much as possible while disclosing as little as possible. Any papers published are probably reviewed 10 times to ensure no "secrets" that could be re-applied are leaked.
https://arxiv.org/search/?searchtype=author&query=Magarshak%...
So I know that smart people in those companies can definitely publish. In fact, a whole team should probably be publishing like no tomorrow!
To be fair — from about half of the papers.
This is also in line with my subjective experience as someone reading quite a few papers in the field: """The preprint also found that firms based in China consistently published more papers than their counterparts based in the United States."""
We’ve really not had massive advancements in the last few years that weren’t quickly discovered/copied by everyone at the same time, just bigger models. Everyone has been just playing around with the same mathematical parlor tricks from the original breakthrough.
Until that changes I expect we’ll continue to see this arms-race-to-the-bottom as everyone is just battling to make a commodity vs true innovation that gives one startup a legit IP advantage over others.
More to the point, without determining how much work is “worthy” of a paper it is unclear how much this matters.
Most AI companies are either a product and marketing layer over a model or not meaningfully moving any dimension to be worthy of a paper.
Also, formal papers and blogs and “cards” are all being intertwined.
eg https://news.ycombinator.com/item?id=49088205
On the other hand, I have a custom harness agent running an ecommerce business 100% autonomously at this point, but have very little to benefit from trying to “publish” anything about it - and without a Stanford or YC stamp, I doubt anyone would care much anyway
I don’t think Dario wants to be a monopolist. But history is full of brilliant people who focused on the social good as priority #1 and lost the game early.
Once you lose the game and go bankrupt you can’t do anything. Same as politics, if you’re not elected you can’t change anything. Of course there are many lines not worth crossing.
How do we fix this.
It's like we know things are destined to go down in a flaming pile, and we're just watching so we can change it AFTER that happens.
Soz. Kinda a "dear diary" response, but am frustrated.
Also, it's too hard to publish and academics are too hostile. These people are government funded and reject work for things like mentioning Palestine. I've had multiple papers automatically rejected even for pre-publication. Even when I have jumped through all these hoops, it has no more credibility than a blog post to the average person. In fact, most people cannot even read my published work. They look at it for a few seconds, squint, and then say, "Cool."
All of this and the vast majority of publications are not even replicable. I've made do with uploading code or instructions on how to replicate work through git and then make a blog post about it.
I still create papers but they're for myself. When I complete some dense scientific project, a paper is a great format to onboard me onto it a few months later. I'm most concerned about indexing the work, but I find that online archiving tools are the best bet since journals are behind paywalls and are subject to retractions.
How we wish to consider public funding and required sharing of gains is an open debate.
...today? (emphasis is mine) And the author was so excited that decided to write up a paper and submit it to Science all in one day? Yeah, right. That's all you need to know who is moonshooting behind this paper.
(Not saying that's how it should be, but that's how it has been.)
We're not racing towards anything. We've been going in circles for years.
We reached the NASCAR-racing equivalent of scientific research.