Hacker News Reader

Comment by ml_basics | Hacker News Reader

From the paper:

> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

I'm curious whether they have continued to scale up model size/compute significantly or if they have managed to make significant innovations there.

I just skimmed the paper but seems they are also omitting details about how they actually feed the images in too, which is a shame as a curious outside observer.

View on HN

What about the glaring safety implications of the custody of this power being in the hands of a relatively small number of people, any of whom may be compelled at any point to divulge that power to those with bad intentions? Secretly?

Conversely, if all actors are given equal access at the same time, no such lone bad actor can be in a position to maintain a hidden advantage.

OpenAI's actions continue to be more than merely annoying.

> What about the glaring safety implications of the custody of this power being in the hands of a relatively small number of people, any of whom may be compelled at any point to divulge that power to those with bad intentions? Secretly?

What you are looking for is a publication known as "Industrial Society and Its Future"

More commonly known as “ The Unabomber Manifesto”[1]

> 1995 anti-technology essay by Ted Kaczynski… contends that the Industrial Revolution began a harmful process of natural destruction brought about by technology, while forcing humans to adapt to machinery, creating a sociopolitical order that suppresses human freedom and potential.

[1] https://en.wikipedia.org/wiki/Unabomber_Manifesto

Available for free online in many places, for example:

https://theanarchistlibrary.org/library/fc-industrial-societ...

I agree very much with Teddy about the problem but I don't condone his solution. I don't have a better one though.

I'm sure you can come up with something that doesn't involve murdering innocent people

I would sure hope so, but so far I haven't seen anything convincing. The industrial machinery keeps marching on.

At this point I'm predicting that the transition to renewables will fail due to the enormous costs involved (aside from transportation there are also things like converting metal industries to electric), combined with increased EROEI of fossil fuels eventually making extraction too expensive to maintain expected outputs.

It's still somewhat far into the future but it's seems to be happening, which is a comfort from the perspective of Ted's insights, but on the other hand it's not going to be any less violent, even though it would happen as an unintended side effect rather than through conscious effort.

People will once again need to become skillful in multiple areas, compared to the current specialization economy where every person is pretty much useless unless part of the "machinery".

> murdering innocent people

If you are refering to the bombing campaign, that was a publicity campaign for the manifesto, not related to the content of the manifesto.

I don't think the manifesto itself advocated violence.

Indeed.

193. The kind of revolution we have in mind will not necessarily involve an armed uprising against any government. It may or may not involve physical violence, but it will not be a POLITICAL revolution. Its focus will be on technology and economics, not politics.

> 172. First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary.

> 174. On the other hand it is possible that human control over the machines may be retained. In that case the average man may have control over certain private machines of his own, such as his car or his personal computer, but control over large systems of machines will be in the hands of a tiny elite-just as it is today, but with two differences. Due to improved techniques the elite will have greater control over the masses; and because human work will no longer be necessary the masses will be superfluous, a useless burden on the system.

I always thought a good addendum to 174 is that the ai will be compelled to generate extremely effective propaganda to convince the non elite that this situation is good.

I don't really understand.. Pretty sure he wasn't worried about "safety implications" in that. Is this just like a snarky thing? Like having any kind of critiques about technology means you must be allied with the unabomber?

People have spilled a lot more ink than that on this subject! And most of them weren't also terrorists.

That doesn't make sense to me. Would rather you have it in the hands of people who think a lot about safety, but might be compelled to give it to bad actors, or would you rather just give it to bad actors right away?

It's not a zero-sum game where you can level the playing field and say everything's good.

People who think a lot about safety are the bad actors when 1. there are incentives other than safety at play and 2 . nobody actually knows what safety entails because the tech is so new

I'd rather have it in the hands of everybody so that we can decide for ourselves what this means for safety, everyone can benefit from the new technology without restriction, and so that we are not dependent on someone else's benevolence for our protection or for access to powerful new technology.

Leveling the playing field won't instantly make everyone safe, but leaving it uneven certainly doesn't either.

It's not clear to me how your argument would work for GPT-4 when it's clearly not reasonable for nukes.

We elect the people with the nukes (in theory). Don't remember electing OpenAI.

Dito for the sewage/water system or other critical infrastructure.

Not saying OpenAI needs to be elected or not, just expanding on what (I think) they meant.

This is the same argument people use against the 2nd amendment, but it fails for similar reasons here.

If we accept that the public having access to GPT-4 has the same level of risk as the public having access to nukes would than I'd argue that we should treat GPT-4 the same way as nukes and restrict access to only the military. I don't think that's the case here though and that since the risks are very different, we should be fine with not treating them the same.

The counter for nukes is nobody should have nukes. Anybody trying to build nuclear weapons should be stopped from doing so, because they're obviously one of the most catastrophically dangerous things ever.

At least with ai you can cut the power, for now anyway.

We can use nukes to generate EMPs to take out the AI

Nonproliferation is practical with nuclear weapons.

With something that can be so trivially copied as a LLM that isn't possible.

So in this scenario, one could argue that ensuring equitable distribution of this potentially dangerous technology at least levels the playing field.

It's not practical. The NPT is worthless, because multiple countries just ignored it and built their nukes anyway.

North Korea is dirt poor and they managed to get nukes. Most countries could do the same.

It does. Mutually Assured Destruction (MAD)

https://en.m.wikipedia.org/wiki/Mutual_assured_destruction

That's not everyone. That's major strategic powers. If everyone (in the literal meaning of the term) had nukes we'd all be dead by now.

The nuke analogy only applies if the nukes in question also work as anti-nuclear shields. It's also a false equivalency on a much broader fundamental level. AI emboldens all kinds of processes and innovations, not just weapons and defence.

It's not a false equivalency with respect to the question of overriding concern, which is existential safety. Suppose nukes somehow also provided nuclear power.

Then, you could say the exact same thing you're saying now... but in that case, nukes-slash-nuclear-energy still shouldn't be distributed to everyone.

Even nukes-slash-anti-nuke-shields shouldn't be distributed to everyone, unless you're absolutely sure the shields will scale up at least as fast as the nukes.

AI of course has the potential for good—even in the hands of random people—I'll give you that.

Problem is, if it only takes one person to end the world using AI in a malevolent fashion, then I think human nature there is unfortunately something that can be relied upon.

In order to prevent that scenario, the solution is likely to be more complicated than the problem. That represents a fundamental issue, in my view: it's much easier to destroy the world with AI than to save it.

To use your own example: currently there's far more nukes than there are systems capable of neutralizing nukes, and the reason for that owes to the complexities inherent to defensive technology; it's vastly harder.

I fear AI may be not much different in that regard.

I wonder how this would work for nuclear weapons secrets.

I think it's okay to treat different situations differently, but if someone were able to make the case that letting the public have access to GPT-4 was as risky as handing the public all of our nuclear secrets I'd be forced to say we should classify GPT-4 too. Thankfully I don't think that's the case.

But if this tool is as powerful as Microsoft says, then an average nuclear physicist in a hostile state will now be more easily able to workout your nuclear secrets (if they exist)?

I'm actually starting to wonder how long these systems actually stay publically accessible?

On the other hand, people might be able to use these machines to gain better insights into thwarting attacks...seems like we're on slippery slope at the moment.

My guess is that eventually our devices will get powerful enough, or the software optimized enough that we can build and train these systems without crazy expensive hardware at which point everyone will have access to the technology without needing companies to act like gatekeepers.

In the meantime, I expect our every interaction with this technology will be carefully monitored and controlled. As long as we have to beg for access to it, or are limited to what others train it on, we'll never be a threat to those with the money and access to use these tools to their full potential.

I think universities might help serve to bridge the gap though, as they have in the past when it came to getting powerful new technology into the hands of the not-quite-as privileged. Maybe we'll see some cool things come out of that space.

Without paper and architecture, GPT-4 (GPT-3+1) could be just a marketing gimmick to upsell it and in reality it is just microservices of existing A.I models working together as AIaaS (A.I. as a service)

At this point, if it goes from being in the bottom 10% on a simulated bar exam to top 10% on a simulated bar exam, then who cares if that's all they're doing???

If they are overfitting, then its not very interesting.

Humans overfit when they go to law school.

OpenAI writes in the post:

> A minority of the problems in the exams were seen by the model during training

A minority can be 49%. They do mention they tested against newly available practice exams, but those are often based on older real exam questions which may have been discussed extensively in forums that were in the training data. Now that it is for-profit ClosedAI we have to somewhat treat each claim as if it were made adversarially, assuming minority may mean 49% when it would benefit them one way and .1% when it serves their look better for sales pitch to the Microsoft board, etc.

There's no need to be quite so adversarial in this case though. The methodology is explained by the report:

> A minority of the problems in the exams were seen by the model during training; for each exam we run a variant with these questions removed and report the lower score of the two. We believe the results to be representative. For further details on contamination (methodology and per-exam statistics), see Appendix C.

I hadn't seen the technical report: it is good they made an attempt to remove them, but they only use substring match of 50 characters to find duplicates. Forum discussions after an exam are usually peoples more fuzzy memories of the question (it is impressive if it can convert back at test time from people's fuzzy memories, but still potentially taking an exam from the past where it has had access to the questions, especially the hard ones which get discussed the most).

From the results before and after removing some of the duplicates it doesn't seem to have hurt its performance badly though. Sometimes the score increases, so the substring approach may be helping it by excluding question variants with matching substring that it memorized but then the real test varied somewhere outside of the sampled substrings and had a different answer (or it random chance that the extrapolated score increased with some questions removed).

This approach to safety reminds me of The Right to Read, the famous short story by Richard Stallmann. He predicts a dystopian future where private possession of a debugger is illegal. https://www.gnu.org/philosophy/right-to-read.en.html

It is unsafe to not release the source along with the service. That incentivizes competitors to sacrifice their own safety research in favor of speed to market. Instead of getting shared safe tools, we get a bunch of for profit corporations pushing their proprietary unsafe tools.

Preventing this situation was the original reason to setup OpenAI. Speed run to the dark side.

I bet they use CLIP to caption the image and feed the text of the caption into GPT, but that's just a guess.

Did you check all of the samples provided? It can read an entire research paper and understand the figures just from the images of the papers pages. This seems to be a much deeper connection than extracting captions.

Are you sure? Sounds too epic

It's SOTA on DocVQA[1] so yeah it is able to read text/graphs/tables from images

[1] https://www.docvqa.org/

See the real examples for yourself, starting on page 34 ... mind-blowing.

https://cdn.openai.com/papers/gpt-4.pdf

The extreme ironing image example has a bullshit explanation in the paper. The extreme ironing on back of taxi is a popular photo with lots of text associated with that picture: https://google.com/search?q=extreme+ironing+taxi&tbm=isch

Give the model new images that are not in the training set (e.g. photos not on internet, or photos taken after model trained) and ask the same question and see how well it does!

The paper says: “Table 16. [snip] The prompt requires image understanding.”

I think the explanations (in the paper by OpenAI for the images) are probably misinformation or misdirection. I would guess it is recognising the images from it’s training and associating them with nearby text.

It seems like they used some unknown images in the livestream, see replies to: https://news.ycombinator.com/item?id=35157940

However, I still think they should not have used images from the internet/training set in their paper. And to be safe, neither should they use “generated” images.

I am looking forward to taking photos of some paintings by friends and seeing if ChatGPT can describe them!

There's no need to round-trip through text, you "just" need to train an embedding space that captures both domains.

They almost certainly generate tokens directly from the image. It would be extremely hard to generate short english descriptions which sufficiently describe the images to pass some of those benchmarks.

CLIP doesn't do captioning, it just generates embeddings. And it's contrastive, so it would work poorly for this kind of task: anything 'relational' falls apart immediately. (See for example the DALL-E 2 results for these kinds of captions/tasks.)

It's almost certainly a VQ-VAE-style encoding of the image itself into a sequence of tokens, as was done by DALL-E 1, CM3, Gato and a whole bunch of more recent models. It's the very obvious thing to do, and their context window is more than large enough now.

This way the model would also be able to generate images, I would also be curious how they handle images with different aspect ratios (and maybe resolution so it can read well on papers).

You can look at Google's recent PaLM-E model for a possible approach. They use a vision transformer to tokenise the image (or to generate embeddings and then tokenise those?) and they also tokenise detected objects so the model can reason at a semantic level. Either way, it's been shown that these massive LLMs can handle images in tokenised form if you pretend it's text. In Google's case, the model is trained to look for sentinel values in the prompt (i.e. <img>) that denote images/objects are being sent.

These are all good reasons, but it’s really a new level of openness from them.

Open AI more like Closed AI

Safety has nothing to do with it. It's an easy tack on for them because of popular fear of AGI.

It's all about power over the market.

Cringe.

I'm assuming they scaled up the model significantly, given the limited availability of the trained model and the increased pricing. Seems like they don't have enough clusters of A100s to go around at the moment.

Or perhaps the usage restrictions allow openai to improve the "safety" of gpt4 before too many people have access to it.

We don't trust you with it. You don't get a choice whether to trust us with it.

> Given both the competitive landscape and the safety implications

Let's be honest, the real reason for closeness is the former.

> this report contains no further details about the architecture (including model size), hardware, training compute

As a beginner in the NLP world, this may serve me a purpose which is to hide the complexity behind building such models.. numbers like xyzB parameters, 12K A100s.. are scary, so I still can dream of building one system one day. This story [0] and this one [1] hide some extremely complex edge cases that a beginner will never though of or had the courage to start if he knew what is the real cost.

We may, however, still be able to infer some details [probably in the future] knowing how Microsoft had re-arranged its infrastructure to welcome OpenAI training [2]

_________________

[0]. https://www.construct.net/en/blogs/ashleys-blog-2/simple-sof...

[1]. https://prog21.dadgum.com/29.html

[2]. https://www.theverge.com/2023/3/13/23637675/microsoft-chatgp...

Reply on news.ycombinator.com