Generative AI is good at cooperating with people and bad at full automation | Hacker News Reader

Generative AI is good at cooperating with people and bad at full automation

skybrian.substack.com

98 pointsskybrian70 comments

It used to be that classical software was useful for verification, but not so good at generating ideas.

With LLMs we have the opposite: machines are now good at idea generation and exploration, but their ideas need to be verified by a human (or classical logic-based software).

So all we need to do is have LLMs generate classical logic-based software to qualify its own ideas

I’m quite confident they’re using GitHub copilot or similar inside of OpenAI while working on improving GPT-4. So yeah, that’s happening.

But isn’t copilot itself a fuzzy llm? Many of its solutions don’t seem formally verified.

Not entirely, but it will already be doing pretty well when it reliably outputs code that compiles and passes the test cases it generates.

There are plenty of different ways this is happening. 1) already happened with Galactica - the model included python scripts as part of it's "working memory" as part of a <work> token, which could be used in a container 2) via copilot, and copilot 365, as someone stated elsewhwre 3) GPT-> Wolfram alpha as a plugin, which will be out very soon

Galactica is the only one that is already (technically) available

But, by virtue of how they work, are they any good at generating novel ideas?

That raises an interesting point. On the one hand, novelty is in the eye of the beholder, if you will. What has been done before might still be new to many. On the other, randomness is built in and language has a remarkable capacity for uniqueness (see statistics about novel queries on Google). So it's possible and likely for LLMs to generate novel sentences. Whether those are valuable is a harder thing to think about.

Given that even an army of monkey typists can generate a Shakespeare, why wouldn't an army of llms? Whether we call that "generating" or "creating" might be only philosophical.

An army of monkeys can only generate Shakespeare if you have enough manpower to sit around monitoring the masses of output, which makes it effectively not possible in any meaningful sense.

Some combination of users interactions with the endless output might create a decent enough opportunity. For example, Spotify discover combined with auto generated music (of a certain level) maybe is a possibility for this in that genre?

in now condition, some company rely on copy artists' style to make sure it good enough, and forbidden copyright.

LLMs have randomness, but they are not random. LLMs probably won't generate new words, for example, so they could not have written shakespeare based on contemporary writings.

Not really, but it still helps you a lot with breaking your own bias.

In a way you could use these AI systems similar to kintsugi. Or in the same way a good wood sculptor uses the flow of the wood rather than working against it.

Using AI to provide a basic frame work on which folks can iterate, manipulate or redact is not such a dystopian view of these things.

Which would be a dream if it stops there, but it sounds like we're fairly hell bent on taking it further as fast as possible?

Fersher

Absolutely. If reading any of the computer technology pessimists of the last 50 years have taught me anything is that they are usually right to some degree.

Since Bill Gates claims he never said 640K is enough for anyone, and who doesn't trust Bill Gates - I guess I have to agree with that one.

I don’t understand what we have to gain from a system that generates code that needs human verification. It’s like Tesla’s autopilot all over again. Either it can drive itself or it can’t — if it can’t absolute, 100% reliably manage itself, why are we even pursuing it? I bet a more clever person than I could actually mathematically prove that neural network based models could never provide 100% reliable results.

For profits?

I just watched an interview with Max Tegmark from 2 years ago where he talks about "Intelligble Intelligence" and that he feels that's the way forwards, it's an awesome concept. He said what he fears the most is huge black box AI, which shows human like intelligence by throwing more hardware at it, and we keep doing that until we maybe get an AGI.

Well...here we are...

I was trying to get ChatGPT to generate a table for all the products I'm trying to put together for Stripe for this conference I'm helping out with, and with all the permutations there's 24 products at different prices.

I fed it the options for all the columns and it couldn't do it! So in the end I resorted to telling it to give me a JS function to generate a CSV, and it did it perfectly. I think I'll end up telling it to give me code that gets me the thing I want (when it comes to data), rather than get the formatted data directly. It sometimes makes weird mistakes, and it's less reproducible.

ChatGPT 3.5 or 4?

That way you also get the added benefit of not sharing your data

I agree with the overall idea, but we don’t need to limit the AI to human collaborators.

I think the future lies in the Copilot approach of giving the Agent relevant context by building smarter prompt generation pipelines. The AI are really good when given enough context and they are bad at recalling information from their own training corpus. When you combine multiple AI agents together with good context and tools like in the LangChain approach these AI agents suddenly become really good at full automation.

Which imo is the path to AGI.

Do you have examples of combining multiple AI agents together with context and tools? I'd like to learn more about what you're referring to.

This is a basic example of a Reason and Act (React) agent: https://python.langchain.com/en/latest/modules/agents/agents...

Search is a tool that provides more context to the agent in order to answer a human task. By combining many agents together we can solve complex tasks.

Cooperation with people is what I thought of when I heard of the forthcoming Business Chat feature for Microsoft Teams <https://news.ycombinator.com/item?id=35182827>. From what I have read, it is going to produce summaries of conversations. That's not quite what I'd imagined, which is something like this:

----

A: ... and that is why I think we should go with option 1.

B: No, the points you mentioned support my case for option 2.

C: Nothing you guys have said changes my mind about option 3 being best.

D: Business Chat, what do you think?

BC: Based on this discussion, and my research, option 1 seems more realistic but option 2 would be more profitable if possible. My reasons are ...

C: Business Chat and you guys all don't understand point N, which is the main reason why option 3 is best.

B: Higher profit is exactly why I think option 2 is the way to go.

A: No, our rival is going to hit the market next month. We need to get something out there ASAP. Option 1 can do this.

D: You've all given me things to think about. Thank you for coming. Business Chat, email me a summary of the meeting, and set up a followup meeting for Tuesday 3pm.

----

That is, AI used as a colleague/assistant, not necessarily subordinate but not seen as omniscient, either; another viewpoint to consider. When will the above be feasible? This year? Five years?

This type of thing really is science fiction, in the sense of Data or C-3PO sharing their (often ignored) opinion or assessment.

People would probably tend to grow an aversion to Business Chat for its smartypants opinions. Microsoft may want to avoid that. ;)

They are good at telling people what they want to hear. There are lots of people like that too, but yeah they generally dont achieve much on their own.

Right. If you don’t dig deeper, it will give you only generic answers, and even fully incorporate the intention of your questions, strengthening your biases.

Prompt engineering, in my opinion, could be considered "semantic programming."

That's a great thought. Although an overarching generalisation.

Agree - we're taking a hybrid approach - meaning the gen AI is 'prebuilding' the automation and then the user reviews and possibly corrects it before fully executing - check it out here: bardeen.ai/ai

I've just read these models are getting to the point they run out of training data. They pretty much scanned most of the internet already. I wonder if we will reach a disappointing situation in 3-4 years where we're at "peak A.I" abilities. Well disappointing for some, for others it means they will get to keep having a job with income :)

The current problem isn’t a lack of data. Meaningful data can be augmented with GPTs, just iterate over sparsely represented subjects and enrich them with context-aware samples …generated with GPT.

We are actively leveraging GPT4s brainstorming-ability to generate datasets used for finetuning downstream. The fine tuned models downstream can then be used to augment new training data for more complex root language models, just formalize your quality assurance with common sense DSLs. We’ve reached an upward spiral with quite the exponential feel to it, where it will lead, nobody knows.

That’s only going to get you so far. Sparsely represented subjects have have an actual reality you’re trying to model. So while yes, you can generate synthetic data to interpolate between the points that have already been sampled, whether those interpolated points have anything to do with the underlying reality is a matter of chance. If you really try to drill down the existing tools into a specialized problem area, it becomes very clear that they lack a sufficiently informed model of the subject to be useful. That’s why I’ve found that ChatGPT is great at the beginning of a project and increasingly irrelevant as you approach the core technical challenges of the project.

Since the LLMs are probabalistic is it possible for the inference engines to highlight lower probability content. That might at least make them more "human" rather than being confident and inaccurate.

Since LLMs generate words one by one, you can only determine whether the next word has a high probability. Some ideas occurred to me earlier, like applying prompt engineering to evaluate if the model is confident about the contents to be generated.

Like "Please reply T if you are confident about the text that you are going to generate, and F if you are not".

But if this could be done by architecting the neural networks, they would perform much better.

"Since LLMs generate words one by one, you can only determine whether the next word has a high probability"

We don't fully understand how they work or what their limits are.

we do understand that they generate words with probabilities one by one though. They are transformers, the AI bit takes the entire prompt as input and returns a range of probabilities for the next token (token == word, it doesnt understand or see letters). A "dumb" algorithm then selects from these probabilities randomly using the probabilities returned by the ai based on the "temprature" setting (low temprature favors high probability words & high temprature selects from a whole lot of words but leads into really weird territory fast)

It's also worth remembering that this is not just a markov chain as many people seem to think, it doesn't simply remember what words come next in its training set, because statistically if you take a random 10 consecutive words from a piece of text, chances are its never been written before. (also the trained model is much smaller than the size of the dataset, so to simply remember everything it would have to be the worlds best compression algorithm by orders of magnitude) Thats why we need an AI here, to learn the general rules of language so it can respond to chains of words it has never seen before.

The sense that we "dont understand how they work" is that we dont know what the "rules of language" that it has learned are.

"we do understand that they generate words with probabilities one by one though"

This is no more helpful in understanding AI's than is knowing that human brains operate according to the laws of physics is helpful in understanding the human mind.

Yes, there are important mysteries that need to be solved through research [1] but there are still some limits we can deduce from how they are built [2].

[1] https://skybrian.substack.com/p/dont-settle-for-a-superficia...

[2] https://skybrian.substack.com/p/ai-chats-are-turn-based-game...

My guess is that a low-probability token corresponds as much with creativity (it's saying something in an unusual way) as truth, since it's at the word level. Words aren't normally true or false, sentences are. Do LLM's represent their confidence in what they're saying anywhere? I don't think anyone knows. Another mystery.

But another problem is that confidence is a character attribute, not a writer attribute. If you ask an LLM to imitate Richard Feynman it's going to write pretty confidently about physics, despite not knowing as much as him about physics.

This is the equivalent of giving your RPG character high intelligence on the character sheet. Doesn't make you smart!

To make an LLM express confidence consistent with its actual knowledge, it would need to have good self-knowledge and actually use it. So far this doesn't happen automatically. Instead, OpenAI uses reinforcement learning based on what the people at OpenAI think the LLM can do. So that's why it sometimes refuses to answer for some kinds of questions.

That has the most effect on the default character, the "helpful AI assistant". Any other characters you ask for will likely have poorer self-knowledge.

I've read that, mysteriously, more training does make bigger LLM's better calibrated, but the reinforcement learning makes it worse again.

One other thought I had, was did anyone fact-check the training corpus. It is probable it contained factual errors, which would mean that the trained model would propagate errors some how.

I'm amazed at how much nay-saying there is on HN about LLMs. This community has bought into so many fads, like crypto, yet when presented with what's clearly going to be the most disruptive technology of your life, you act like it's no big deal. Is this just nervous denial? Are you suddenly incapable of linear (not exponential) extrapolation? Imagine what ChatGPT, Midjourney, or whatever Apple and Google are cooking up will be capable of if they only improve linearly, at say 1% per year?

People ought to be preparing themselves for this as if it were a COVID-like black swan event, except much, much worse. And do not bank on Sam Altman and his fellow tech bro billionaires holding the reins of these job-annihilating machines delivering with their promise of UBI.

I get why you're being down voted but your overall point is valid imo. We didn't get any good answers about what happens if this thing keeps getting better.

"People will find new jobs/things to do" doesn't really cut it. It's not really the tech titans fault, the governments really need to start stepping in.

There are AI boosters making exaggerated claims in their enthusiasm and therefore people take the other side in reaction to them because they’re wrong. The skeptics then sometimes write blanket dismissals in opposition, and that’s wrong too, and you get a loop. That’s just how it goes on the Internet sometimes.

I totally get the motivation and it’s why I started blogging again. To get more interesting conversations, I think we need curiosity while keeping in mind how little we know and how bad we are at predicting the future.

Often I can get it to fact-check itself by asking for something in two different ways.

What about AI collaborating with AI?

> Generative AI is good at ~~cooperating~~ with people

fooling

FTFY

It follows the "yes, and" rule of improv. This is cooperative if you understand what it's doing. If you want to believe it has any fixed motivations other than that, you fool yourself.

I fully understand what it's doing: it exploits the human brain being wired to see patterns where there is none. It vomits up words in such patterns humans have a high tendency to trust them. Of course , there is zero actual information content in it. It's a zeroday on human cognition, a terrible, a terrible danger to society but there's no stopping it. https://futureoflife.org/open-letter/pause-giant-ai-experime...

You could make the argument that communication between humans, or even human thought itself is nothing more than that.

You could -- you could make any argument -- but it would be utter nonsense.

Care to elaborate?

Good ideas can sometimes be generated via play, without proper justification for them. There is chance involved, but if you test them and they hold up then they become meaningful.

Reply on news.ycombinator.com