For example, I get _unbelievably_ excited with knowledge extraction and question answering demos on PDFs. Why? Because i've built similar systems for over a decade and know how difficult it is to build on top of messy archival data. Now, with very little code, i'm getting SOTA results.
If you didn't have experience with this, you'd probably thing "Huh, that's not impressive, XYZ does this already". But it's the moving of the baseline that's what's really impressive.
===
AI hype-beasts aside of course - the breathless pontificating of "influencers" and former crypto bros is cringe.
https://www.pinecone.io/learn/langchain-retrieval-augmentati...
Langchain / Llama indexes are both toolboxes that abstract away a lot of the plumbing for doing this kind of thing, and Pinecone is one of dozens of vector databases.
Personally, i'd try out langchain and chromadb and go through some of the examples langchain has in their docs, then be prepared to completely abandon langchain and just work with the LLM APIs directly. Start with openai, get on the waitlist for GPT-4 tokens, and also get on Anthropics Claude waitlist for the 100k-1.3. It's _very_ good for knowledge retrieval.
Langchain tries to do too much in extracting away the prompts, and the prompts are really what matter in getting interesting stuff out of your own data. Use langchain, llamaindex pieces but build from scratch for most things as your tinkering.
It's really not hard if you have a background in programming, and it's _so_ much fun. You'll feel like you have superpowers once you get a scraping interface hooked into an LLM. All of a sudden you can automate some really complex pipelines very quickly
Having tangled with natural language processing and transformation in the past (always with dismal results), I can say it's one of the most annoying problems to tackle in computation, because natural languages have the horrible tentency of being very irregular (i.e.: they are a fucking mess).
ChatGPT capabilities to parse and generate fluent language never ceases to amaze me.
At the same time I don't think it's going to take over the world. It's more like a game-changer productivity tool (with all the upheaval that comes along when those appear) than the birth of Skynet.
Just made me more aware of how bulshit is the norm and not the exception.
What exactly is it that makes you so confident?
Furthermore, I would argue there are strong complexity-theoretic bottlenecks to computational processes which limit the expressive power of neural networks, even if they could harness galactic amounts of energy. Physical Turing machines have bottlenecks that upper-bound the power of oracles.
So what? If it kills us, we're still dead.
> I would argue there are strong complexity-theoretic bottlenecks to computational processes which limit the expressive power of neural networks
Of course there are physics/CS limits to how intelligent something can be in a given volume, but those limits are vastly higher than our own so I don't think they are particularly relevant. For instance, a system which could simulate the brains of a thousand scientists as smart as Einstein a billion times faster than realtime would not violate any rules of physics, even though it is far beyond our current capabilities.
We do have thousands of Einsteins today wielding untold resources, but the slowing pace of scientific progress suggests there are limits to scaling intelligence. Even with a hundredfold increase in scientists, I am unconvinced that would lead to a meaningful increase in social progress and have come to believe the bottlenecks we face are not due to a lack of intelligence, but a lack of other virtues (e.g., kindness, curiosity, courage, compassion, perseverance).
Obviously, your arguments are never going to be convincing to someone who does not share your beliefs.
Safety has become a near sacred value. Near my house a new light at a cross walk has a computerized voice that says "stop stop stop" , "walk walk walk". A cross walk to a dead end street with basically no traffic ever.
I can almost hear the city official who had it installed "if it saves just one child's life..." even though it is a near certainty no accident ever occurs at that cross walk either way.
AI safetyist are basically that city official to me. What you posted is besides the point. It is about lowering a probability no matter how remote that probability is and even if the whole exercise is basically pointless. The gods of safety will be pleased by our offering.
A lot of this is because I think it sort of wedges a knife into an area of our brain which makes us think it's similar to us. Look at this output, it's able to write about Shakespeare or summarize Beowulf or talk about these topics in a way I can understand. But ultimately it's an affirmative mirror; it will respond exactly as you prompt it. It cannot disagree with you or ask 'why' or synthesize the world beyond what it's told to do.
And it's incredibly hard to get some people to understand this. Even harder when you have companies pushing this because it's trendy even as we see the issues of private data being leaked or the frays in the data sets appearing.
I like how suddenly the image generation state of the art made unexpected and significant progress. One can witness something like that only time to time.
What I don't get at all is that there are people like you, who are frowning upon and somehow disgusted by people like me.
To be very blunt: If you believe that GPT and the like are magic, then you're not thinking clearly. You're blinded by the results (which are impressive).
It is not magic, it has never been magic.
We aren't, and I haven't said otherwise.
Einstein found wonder in the simplicity of a circle. What do you find magical?
> Magic is that which can not be explained.
You have now redefined what you mean by "magic" as "that which inspires wonder". Changing definitions after a series of comments is a pretty poor way to have a discussion.
There are plenty of explanations of how LLMs work, by their creators, incidentally.
1 - https://www.jasonwei.net/blog/emergence 2 - https://arxiv.org/pdf/2206.07682.pdf 3 - https://www.quantamagazine.org/the-unpredictable-abilities-e...
[1] Is a summary of [2], by one of its authors, not a separate source.
[2] Defines "emergent behaviors" in a way that you're clearly misunderstanding (because "emergent behaviors" is an extraordinarily poor way of communicating this--it's partly the fault of the researchers who chose this ambiguous language). All it's saying is that bigger models can do things that smaller models can't, which should be surprising to no one. It's NOT saying that the capabilities are anything more than the sum of the input data.
[3] Is written by a journalist, not an AI researcher, and so it's limited by the things the journalist is excited about. The journalist, for example, downplays sections like, "The other, less sensational possibility, she said, is that what appears to be emergent may instead be the culmination of an internal, statistics-driven process that works through chain-of-thought-type reasoning. Large LLMs may simply be learning heuristics that are out of reach for those with fewer parameters or lower-quality data." If you're going to try to gather things from journalists rather than subject matter experts, you need to understand how journalists work, and how subject matter experts work, and look for paragraphs like that to understand what's actually happening.
Yes. Your point? I included both because I found them both interesting. The paper is the source, the 137 emergent behaviours page is one of the authors continuing the work, and [3] is a journalist talking about this, so I included it as it's a unique perspective.
I used the word "emergent" because that's what the SME used when describing this. From 5.1 in the paper linked:
> Although there are dozens of examples of emergent abilities, there are currently few compelling explanations for why such abilities emerge in the way they do.
You say this "should be surprising to no one", yet the authors disagree.
Additionally, in the GPT-4 system card - "Emergent" appears 15 times, specificly section 2.9 is interesting https://cdn.openai.com/papers/gpt-4-system-card.pdf So it's not just a word used callously by one group of researchers at Google.
Where I went to school, the teachers kept repeating that they wanted us to think critically about stuff when we were writing essays for homework. They didn't really know how to teach that, so most kids didn't learn it, but it's a valuable life skill: learn to think about what people say, and why, not just take everything everyone says at face value, especially when the person is in a position of some sort of authority (like being the people who released a system, or who wrote a paper, or, hashem yerachem, The Godfathers of AI). Authority is the mind killer.
But I can see from your other replies that your real objection appears to be use of the m word, which seems odd given its expressive power, but you do you.