Hacker News Reader

Comment by m3kw9 | Hacker News Reader

OpenAI getting rich now everyone wants a piece and everyone fighting over it like a billion dollar inheretence fight. In a round about way, if GPT5 is nearly as advertised, watch the govt swoop in under national security guise

View on HN

> if GPT5 is nearly as advertised, watch the govt swoop in under national security guise

Re: GPT5, are there any .. reasonable/credible sources of information on the subject? I've become deaf from all the speculation and while i am very curious, i'm unsure if anything substantiated has actually come out. Especially when considering speculation from Sam himself.

There's barely any credible information about GPT4 (i.e. OpenAI hasn't said very much about what's going on behind the curtain) and there's absolutely none re: any releases beyond that.

It’s unlikely to be any time soon. Despite productization by OpenAI, LLMs are still an active area of research. Research is unpredictable. It may take years to gather enough fundamental results to make a GPT-5 core model that is substantially better than GPT-4. Or a key idea could be discovered tomorrow.

Moreover, previous advances in GPTs have come from data scaling: throwing more data at the training process. But data corpus sizes have started to peak - there is only so much high quality data in the world, and model sizes have reached the limits of what is sensible at inference time.

What OpenAI can do while they are waiting is more of the easy stuff, for example more multimodality: integrating DALL-e with GPT-4, adding audio support, etc. They can also optimize the model to make it run faster.

There are diminishing returns from compute time but it looks like even though they are diminishing there's still a fair bit on the table.

Though my guess is that GPT-5 will be the last model which gains significantly from just adding more compute to the current transformer architecture.

Who the hell knows what comes next though?

I keep keep hearing people claim we are at the end of corpus scaling, but that seems totally unfounded. Where has it been proven you can’t running the training set through in multiple epochs in randomized order? Who’s to say you can’t collect all the non-English corpus and have the performance transfer to English? Who’s to say you can’t run the whole damn thing backwards and still have it learn something?

When you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still decreasing, then yes you keep going with a second epoch. On the other hand, if your loss has already converged then starting a new epoch on the same data is just burning money.

In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go.

We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.

You are missing the scaling on context size.

I know ChatGPT is trained on Wikipedia and other open datasets. I wonder what would happen if you gave it the entire Bing index?

They've already swooped in.

I mean the conspiracy argument would be that the $10B isn't a normal investment. It's a special government project investment facilitated by OpenAI board member and former clandestine CIA operative and cybersecurity executive Will Hurd through his role on the board of trustees of In-Q-Tel the investment arm of the CIA. It's funneled through Microsoft instead of through Google in part because of Google's No-Military-AI pledge in 2018 demanded by its employees, after which Microsoft took over its military contracts including project Maven. The new special government project, the Sydney project, is the most urgent and ambitious since the project to develop nuclear weapons in the mid twentieth century.

Of course I don't necessarily believe any of that but it can be fun to think about.

Who has time to come with this stuff?!

ChatGPT

Most of it seems to come from /r/conspiracy, which unsurprisingly has a lot of overlap with another subreddit that starts with /r/cons*

I'd love to go through all the things Liberals labeled a conspiracy in the last 5 years that actually became true, but I don't have that kind of time today

I'd settle for three examples with sources.

Well at least one: in the year 2000, I used to work for Verizon and a picture from one of the local networks hubs was circulated showing a bunch of thick cables tapping into the network and alleging that the government was listening to all calls Americans made. People made a lot of fun of that photo until Snowden brought the details to light.

Please stop spreading FUD and unsubstantiated rumours all over this thread

It would be irresponsible for intelligence agencies NOT to involve themselves in AI. LLMs have the capabilities to catalyze economic shockwaves on the same magnitude of the internet itself.

Notice how OpenAI is open to many Western friendly countries but not certain competitive challengers? https://platform.openai.com/docs/supported-countries

Out of the BRICS, Brazil, India, and South Africa are there. Russia and China aren’t, but that’s not really an issue of “competitive challengers” so much as dictatorships who are invading or threatening to invade democracies

Realise that America has invaded plenty of countries, overthrown leaders, been a huge driver of claimed change and oil industries, basically done whatever it wants and continues to do so, pretty much based on being able to print as much money,as it likes and if you don’t like that, you’ll face the full force of the military industrial complex.

Look at Snowden and Assange. They tried to show us what’s behind the curtain and their lives were wrecked.

The rhetoric on here about Russia and China = ”bad guys”, no questions asked is overly simplistic. Putin is clearly in the wrong here. But what creates a person like that? I believe we are somewhat responsible for it.

People cite possible atrocities of Xinjiang, but what about Iraq and Siria, North Korea, Vietnam whole entire countries destroyed. Incredible loss of life.

American attitudes are a huge source of division in the world. Yes so are China and Russias.

We cannot only see one side of a story anymore, it’s just too dangerous. As we have more powerful weapons and we do, we have to, absolutely have to learn to understand each other and work through diplomacy with a more open mind and peaceful outcomes which are beneficial for all.

No I’m not advocating for dictators, but you cannot pretend that Americans invasions have been always positive or for good intention, or that American interests are always aligned with the rest of the worlds.

The arms races need to stop. Very quickly.

Why wouldn't the US Government invest billions of dollars in a technology that it sees as essential? What's FUD-y about that? Most of our industry itself is the result of the US Government's past investments for military-related purposes.

Later edit: Also, article from 2016 [1]

> There’s more to the Allen & Co annual Sun Valley mogul gathering than talk about potential media deals: The industry’s corporate elite spent this morning listening to a panel about advances in artificial intelligence, following sessions yesterday dealing with education, biotech and gene splicing, and the status of Middle East.

> Netscape co-founder Marc Andreessen led the AI session with LinkedIn’s Reid Hoffman and Y Combinator’s Sam Altman. The main themes: AI will affect lots of businesses, and it’s coming quickly.

> Yesterday’s sessions included one with former CIA director George Tenant who spoke about the Middle East and terrorism with New York Police Department Deputy Commissioner of Intelligence & Counter-terrorism John Miller and a former chief of Israeli intelligence agency Mossad.

So, yes, all the intelligence agencies are pretty involved in this AI thing, they'd be stupid not to be.

[1] https://deadline.com/2016/07/sun-valley-moguls-artificial-in...

Now seven years later Will Hurd and George Tenet are currently the managing director and chairman respectively of Allen & Co! More facts worth considering are in the mysterious hacker news comment from the other day: https://news.ycombinator.com/item?id=35366484

Allen & Co was also the “boutique investment bank” (as the media was calling them back then) that was involved in the acquisition of WhatsApp by Facebook. Archived WSJ source for it [1]

[1] http://web.archive.org/web/20140221144525/http://blogs.wsj.c...

Wait holy shit this is at least partially true though? https://openai.com/blog/will-hurd-joins

Is it not enormously more likely that Microsoft decided to invest a small amount (to them) in a technology which is clearly core to their future business plans?

Let's also note that it is an investment with an expected return (capped at 7x), and gives Microsoft an ownership stake in OpenAI. It's not like MSFT is just throwing money away! Seems a very savvy move by Nadella, and a great partnership for OpenAI too - really a win-win.

Undoubtedly OpenAI already has some very close ties and/or contracts with the DoD.

Any evidence for this, or are you just assuming that it's the case?

To me it overstates what is achieved. This isn't AGI. Unsure how much the government would care about this.

Don’t now about the DoD, but one of the board members of the nonprofit is Will Hurd, who appears to have had a 9 year career in the CIA before turning to politics. No hard evidence of anything of course, but enough to raise an eyebrow imho.

Will Hurd is/was a managing director at Allen & Company, who runs the Sun Valley conference where this article talked about Satya Nadella and Sam Altman "bumping into each other" to secure the deal. It's looking more and more like OpenAI is the Manhattan project as AGI is to nuclear weapons.

Reply on news.ycombinator.com