Hacker News Reader

Comment by williamcotton | Hacker News Reader

Favorite, because it is hilariously wrong?

Functioning code and tools are available for almost all of the relevant AI and ML work in academic publications.

Yes, it took a big leak from Meta for a certain type of model, but the techniques to use LLMs and other transformer based tools are not vaporware. Most have code repositories and functioning open-source implementations.

We can certainly be critical of the datasets used for testing but we can’t accuse people of selling snake oil when so many of us are implementing and gaining value from the techniques described in these academic publications.

View on HN

The GPT4 "paper" that's been talked about so much is little more than a marketing press release.

I certainly disagree that it's the case for the whole field of ML. But the AI that's in the news right now is LLMs with OpenAI in the lead who have done their utmost to imped reproducible research and even research built on top of their already released tools.

It's been a trend that started with Google years ago and their increasingly large models which are unreproducible for most of the scientific community and often not released (if I recall correctly).

Extrapolation from one GPT4 paper to all papers from all sources is a huge mistake.

It's funny the article makes its own overbroad claims without data or references, in the name of asserting that's all that's going on in AI.

> without data or references

Can you clarify what you mean? There are almost two dozen footnotes pointing to references and papers.

I mean, there is no data support in the claim that this pattern is essentially universal in AI research. Citing a few instances doesn’t prove the point. I’m open to the idea that AI research is generally more closed than other areas of research, but there is no quantitative analysis of how open AI research is, or how open other areas of research are.

pick any other area of research, and I will happily find you some footnotes citations that support exactly these claims.

> Citing a few instances doesn’t prove the point.

I haven’t read every linked reference, but for a relatively short article there are more than a few. It's certainly more annotated than the average article I find on HN. Shouldn’t the quality of the sources matter most? And if what matters is the number, how many would be enough in this case? What would prove the point?

> or how open other areas of research are.

Other areas of research are irrelevant to the point. Even if other areas suck, that doesn’t excuse this one sucking too.

> pick any other area of research, and I will happily find you some footnotes citations that support exactly these claims.

Even if that is true, it doesn’t make the argument automatically false or irrelevant for any particular case.

First you objected the author had no data or references. Now you’re saying that for any area of research you can find references to prove the point they’re making. Which is it? Is the problem that they didn’t provide enough sources, or that the sources aren’t believable?

I’m sorry I was apparently hard to follow.

I meant, the author cites anecdotes (“here’s a paper that’s not open!”) rather than data (“of the 500 most-cited papers in the AI space, 82% were not open, compared to only 36% of papers in anthropology being non-open”).

And, I meant if you pick any other discipline that produces thousands of academic papers a year, I will be able to find 20 that don’t count as open.

Anger often makes it hard to understand things.

> for almost all of the relevant AI and ML work

The word “relevant” is an important qualifier which may be doing a lot of lifting. Who defines what is “relevant”? How do you interpret it for this particular case?

I haven‘t looked at the matter myself, but the author did provide a source for their claim¹. Do you have any specific information on why it is wrong? It is not my intention to put you on the spot, I’m trying to understand if your point is based on data or gut feeling so I can myself get a better understanding of the validity of each argument.

¹ https://www.baldurbjarnason.com/2023/beware-of-ai-snake-oil/...

I’d say the two most relevant models right now are Stable Diffusion and GPT-4.

Stable Diffusion is open in all senses of the word.

GPT-4 is closed but the techniques it enables are reproducible by anyone with a credit card.

GPT-like things like LLaMA are, well, available. Direct, individually reproducible, these tools and their related academic publications qualify as computer science.

Nullius in verba certainly applies.

> GPT-4 is closed but the techniques it enables are reproducible by anyone with a credit card.

Requiring a credit card to verify scientific claims is a large deterrent. And in addition one needs to provide them with a phone number, which many people will not be comfortable with when considering their CEO’s history.¹

Even ignoring the above, using the output of ChatGPT is definitely not the same as having “access to any of the data or code used in the research”, which is the author’s complaint and what the original commenter quoted.

As for the other cases, we could only properly assess them if we discussed specific studies which I don’t think we’re going to do in the limited context of HN comments. Thank you for taking the time to explain your view.

¹ https://www.buzzfeednews.com/article/richardnieva/worldcoin-...

I had to give apple my credit card to install xcode...

e: and I'll never forgive them for it

You actually don’t have to pay to use Xcode. You only need to pay to build for real devices and submit to the App Store.

I think the OP means is that even for free (as in beer) Apple software you still have to have a credit card attached to your account (or at least you used to), even if it isn't actually charged, because of the way the App Store works.

correct

This is silly, though. Are NASA and friends not advancing science? You need way more than a credit card to replicate their work.

I would prefer more open advances, but hard not to feel gate keeper for a term to claim they aren't advancing the science.

There's a difference between capital/operational expenditures to reproduce work, and directly paying the owner of the work.

The former is science. The latter is a commercial product.

Stretching their business model to its most charitable interpretation of "open", OpenAI is essentially offering "Here's the cost for the lab and labor for the work you want to do. We'll do it and send you the results, but won't give you any of our lab procedures or intermediate data."

I mean, sorta? Science is advanced in many ways. Not all are open. Some for good reasons. Some not.

Fair. But there's a difference between "openly accessible scientific information" and "trade secrets".

The US military has pushed state of the art in many fields, but I don't think anyone would say it pushed stealth composite science forward with the B-2/F-22/F-35 programs, because all the details are classified.

The risk is that non-shared research is eventually lost.

Also mostly fair.

I think the catch is Open AI is risking turning "open" into a weasel word? That is, still science, but hardly open. Which is fine, but then don't call yourself open?

As near as I can tell, there's nothing really "open" about OpenAI. I think that they include that word in their name is about marketing, and really does help to make the word meaningless.

OpenAI originally was a nonprofit nominally focussed on AI that was actually open, then they branched off a for-profit subsidiary with heavy involvement with Microsoft, came up with “safety” as an excuse to abandon openness, and created the “Open”AI of today.

I'm uncomfortable with your conflation of science, openness, and financing.

> There's a difference between capital/operational expenditures to reproduce work, and directly paying the owner of the work.

But you can reproduce the work with a shitton of compute and a massive dataset. They aren’t sharing their exact strategy but the technique is fairly well known.

IMHO it really doesn’t matter, advancing science comes from trying different things and not from reproducing someone else’s work to prove it works.

> advancing science comes from trying different things and not from reproducing someone else’s work to prove it works

I'd say yes/no.

Efficiently advancing science definitely leverages well-reported and -reproduced results to skip to building on top of them.

> advancing science comes from trying different things and not from reproducing someone else’s work to prove it works.

I disagree with this. If scientists aren't reproducing the work of other scientists, then science cannot advance. Reproduction of results is one of the main mechanisms by which science separates reality from fantasy.

Of course, if reproducing other's results is the only thing scientists do, that also does not advance understanding. But that doesn't take away the extreme importance of reproducing the experiments of others.

> I disagree with this. If scientists aren't reproducing the work of other scientists, then science cannot advance. Reproduction of results is one of the main mechanisms by which science separates reality from fantasy.

Are you or have you been a professional scientist?

Because while that's how it should work, it generally doesn't anymore as it's really, really difficult to get straight replications (which are desperately needed) published.

> Are you or have you been a professional scientist?

Not directly, but I've worked as a research assistant in a number of labs for real, professional scientists and I'm a coauthor on a few papers.

> it's really, really difficult to get straight replications (which are desperately needed) published.

This is a fact. But nonetheless, such replications do get done. Not enough, certainly. But pointing out that the way institutional science is done is flawed doesn't invalidate my point at all.

I think a lot of this is because people are constantly conflating "science" and "engineering".

"Science" is a method of determining objective truth. "Engineering" is taking the truths science uncovers and using them to make things.

Commercial enterprises, including OpenAI, are about engineering first and foremost. They may also engage in science, but that is only to serve the engineering. (Saying that is not saying the there's something wrong with the science they engage in).

This is a bit of a false distinction, all told. More than a few of our "scientific discoveries" were the results of studying the best engineering could do at the time.

I think the distinction is valid and useful, but science and engineering do each inform the other.

The point is that the scientist is primarily intending to learn something solid about reality, and the engineer is primarily intending to create something that is practically useful.

I get the intent of the distinction. I just don't know that it has ever been as clean of a taxonomy as would be desired.

Maybe we have had some solid "pure scientists." Largely we would label them closer to philosophers? That said, the vast majority that you know the name of, were probably employed closer to what we think of as engineers in today's parlance.

Or am I off here and the likes of Feynman, Faraday, Fourier, Maxwell, etc. would not have been said to be working on engineering problems, by modern eyes?

My gut is that it gets difficult as looking at things with our modern lens of corporations today makes it hard to really consider institutions of the past?

Arguing that academic papers for AI are snake oil by using one CEO’s previous and unrelated venture as an attack on his character is not convincing.

The methods for reproducing the LLMs in question have been published. Much like with the LHC, one hindrance to reproducing claims is the cost of building the machine. Unlike the LHC, mere mortals can interact with the machine and draw their own conclusions of the claims published by the academic journals.

This the the foundation of the issues you raise with regards to science in the 21st century: the cost of reproducing a claim is at odds with “taking no one’s word”.

The leak of the LLaMA models changes this dynamic. It is now possible to run the models locally and with the same datasets published in the journals. This is hardly egalitarian but science has always had a material cost. Competition from both public and private sources almost guarantees that these models will be cheap and abundant in not too long of a time.

> Arguing that academic papers for AI are snake oil by using one CEO’s previous and unrelated venture as an attack on his character is not convincing.

That’s a wild extrapolation and an absolute mischaracterisation of what I wrote. Please respond to what people write, not what you think they believe.

I only claimed many people are wary of giving OpenAI their phone number due to their CEO. I did not say AI papers are snake oil. On the contrary, I explicitly said I don’t know and have respectfully asked for clarifications.

Also, Worldcoin is neither a previous venture nor unrelated. They just pivoted to collect your eyeballs to prove you’re not an AI, “solving” the problem they created.

https://worldcoin.org/blog/engineering/humanness-in-the-age-...

Models like GPT-4 are highly functional, so they are not snake oil in that sense, but they are certainly not scientifically reproducible (or understood for that matter).

In fact there's generally a reproducibility crisis in machine learning, since things like models, datasets, training techniques, hyper-parameters are often only documented in the vaguest of terms.

Reproducibility doesn't mean I have a black box and when I push the button (after swiping my credit card) the light comes on as expected. Reproducibility in science means I can reproduce the experiment from scratch and get the same results.

Practically nothing is known about GPT-4. OpenAI's "GPT-4 Technical Report" doesn't say much more than the fact that it's "Transformer-based".

https://arxiv.org/pdf/2303.08774.pdf

>GPT-4 is closed but the techniques it enables are reproducible by anyone with a credit card

No, you really can't reproduce it this way. Science is not a carnival show. This is why the term snake oil was appropriate.

Well, crypto and blockchain have real, verified, science behind them, and that hasn’t prevented the crypto industry from becoming a ponzi garden of delights.

I suspect we’ll be seeing some highly entertaining stuff, coming down the pipes.

Might not be as bad as the crypto snake oil bazaar, but it should have some interesting hucksters.

I saw a billboard yesterday that was advertising some AI thing that probably had nothing to do with AI other than jumping on the hypetrain. Wish I could remember what it was selling, thinking it was something about lawn care.

> Might not be as bad as the crypto snake oil bazaar

I think it will be as bad as with cyrptocurrency, if only because so many of the people involved come from the cryptocurrency crowd.

We can’t be critical of the datasets used for testing if we don’t know what’s in them.

The same applies to training data - it should be disclosed.

Most of the datasets are available.

TriviaQA: https://huggingface.co/datasets/trivia_qa

HotPotQA: https://huggingface.co/datasets/hotpot_qa

Those two are used in most of the question-and-answer based metrics for LLMs in academic publications.

The same applies for other datasets used for other metrics. HuggingFace has most of if not everything of any relevance.

Reply on news.ycombinator.com