Things like Stable Diffusion and DALLE are pretty cool, though a bit novel and toyish at this stage.
Meanwhile, even a very critical paper on this from some Scandinavian researchers [3] says GPT-3 cost 190,000 kWh (0.00019 TWh) to train. ChatGPT/GPT-3.5 is allegedly an order of magnitude larger in terms of data and cost to train, so let's say it is 0.0019 TWh.
When The Register reported on that paper, you can see how they tried as hard as they could to make it sound big: the cost to train GPT-3 was the same as driving a car to the moon and back (435k miles). They could have said it cost the same amount of carbon as 25 US drivers emit each year. In the grand scheme of things, that's nothing. That's one long-haul flight per trained model. And you just need to train them once. Querying the models cost far less.
And the electricity generated for US-based server farms is way cleaner than cars, planes, or the coal mines powering Chinese bitcoin mines.
[1] https://ccaf.io/cbeci/index
[2] https://ccaf.io/cbeci/ghg/comparisons
[3] https://arxiv.org/pdf/2007.03051.pdf
[4] https://www.theregister.com/2020/11/04/gpt3_carbon_footprint...
A question as an ignorant layperson, if I may:
They could have said it cost the same amount of carbon as 25 US drivers emit each year. In the grand scheme of things, that's nothing. That's one long-haul flight per trained model. And you just need to train them once. Querying the models cost far less.
There is probably a need to refresh it periodically to account for what the MMAcevedo fictional story [1] calls "context drift" -- the relevant search terms to infer from the query are themselves contextual. Say, if I ask Bing today "is Trump running for president", the right search term today could be "donald trump 2024 election", but ten years from now it might be "eric trump 2036 election".
Sure, and thanks! Some keywords to look up are transfer learning, zero-shot learning, and fine tuning. These approaches focus on exactly this problem: not having to retrain the entire model from scratch to add new information. GPT-3's training data is 100 billion tokens of text, but to extend it by another 1 billion tokens of text is far closer to 1/100 the original cost.
It actually wasn't the energy/carbon cost that motivated early work in this, it was more about adapting to new domains and letting people customize models for specific purposes. Image processing really adopted it first to great success. Orgs with resources trained really big models on all of ImageNet that needed server farms of GPUs, but they released it so that other people can use a single commodity GPU to fine-tune it for whatever their specific image processing task.
Edit: now you can pay "Open"AI to fine tune their models for you, but only Microsoft has access to the raw model itself
Well, if you fine-tune GTP-3 on another billion tokens you get a fine-tuned version of GPT-3, that's perhaps better at modelling those billion tokens. If you want a better GPT-3 you have to pre-train a new model, probably with a few more billion parameters. So transfer learning is not going to save the day here.
Bottom line, the cost of training "GPT-3" varies a lot and is paid multiple times.
________
GPT-3 paper for ref: https://arxiv.org/abs/2005.14165
Two very different things. I could never really see crypto taking off without massive political changes. ChatGPT, on the other hand, is already my daily assistant. The utility is here and now.
It's rarely perfect, but it gets me 90% of the way there, then I tweak the output.
How good it is depends on what you're using it for, as with anything - I don't think it's really a good fit for something like a search engine (yet) as it's terrible with facts.
But scaffolding code, summarizing text, expanding outlines - it's very good at those kinds of tasks, often astoundingly good.
It's terrible with specific facts, and frequently hallucinates dates, math, book titles and citations to sources.
But it's awesome with ideas and concepts.
Today I spent an hour chatting about the competing ideologies, groups, motivations, objectives, and foreign allies and adversaries in Spain leading up to the 1936 coup and civil war.
I cross-checked many of its assertions, and it nailed some pretty esoteric stuff.
Two of the kids were university/college age, talking to their parents about it.
They were using one of the AI models daily for things like: recipe ideas for food they had in the house, scheduling daily activities, bouncing ideas off for essays, and asking for gift ideas.
They understood it wasn't much good for facts, but liked the conversational interface as a way to give them ideas or jumping off points for things.
Amusingly, there's a mildly rude expression that means "I don't care," and ChatGPT's strong aversion to swearing biased it toward completely fabricating a word-for-word translation. It insisted that the verb was a conjugation of something innocuous, and when I pointed out that that verb doesn't conjugate that way, it both agreed with me and continued insisting.
It seems to have a pretty solid grasp of language in general (so long as you don't need any hard logic), and it's a fun tool to come up with ideas you personally never had before.
This was a little experiment the other day; it's delightful how many silly (yet justified) reasons why you can't walk to Hawaii it came up with. But what do you expect from someone who's only learned about the world by reading the Internet?
> By "unsummarizing" we mean to take some text and expand upon it at length, using fancy words, extra facts, anecdotes, and interesting observations. Your goal is to unsummarize the following text as much as possible.
> Text to unsummarize: What do you mean I can't walk to Hawaii? (GPT-3 follows)
> You may be surprised to learn that it is impossible to walk to the Hawaiian Islands from anywhere on the mainland United States. Though the Hawaiian Islands are located in the middle of the Pacific Ocean and are roughly 2,400 miles away from the nearest point on the mainland, it is not possible to walk there due to the vast expanse of ocean between them. Even the most dedicated and determined walker would be unable to traverse the depths of the Pacific. The nearest point of land to the Hawaiian Islands is the Johnston Atoll, which is located approximately 1,400 miles away. That is to say, it is not feasible to traverse the 1,000 miles of open ocean between Johnston Atoll and the Hawaiian Islands by foot.
> The Hawaiian Islands were formed as a result of volcanic activity, and the chain of islands were formed over millions of years. This activity is responsible for creating the immense walls of water that separate the islands from the mainland. The Pacific is one of the deepest, widest oceans on the planet, and its depths are not suitable for walking. The bottom of the ocean is filled with a variety of topographical features, such as trenches, ridges, and seamounts, which make it a difficult environment for walking. Additionally, the ocean is filled with a variety of natural hazards, such as hazardous currents, extreme temperatures, sharks, and other dangerous marine life. All of these factors make walking to the Hawaiian Islands not only difficult, but also impossible.
> Pacific is one of the deepest, widest oceans on the planet, and its depths are not suitable for walking
I love that. Almost a dry humor.
> [The volcanic] activity is responsible for creating the immense walls of water that separate the islands from the mainland.
In some sense, that's a true statement, right? The volcanic activity didn't create the water itself, but the "wall-like" nature of that water does depend on the islands jutting out above the surface. Otherwise there would be no "immense walls" to refer to.
Can you elaborate on that?
He then said he asked it to tweak it in a few ways (e.g. more time on study, a 5 minute break between activities, etc.). Then he'd print it out and pin to his desk.
For 2), he said that after writing an essay, he'd ask the GPT model something like "I've written an essay on X, and talked about A, B, and C. Are there any other important factors I've forgotten to discuss?". Then use that as the basis for more research.
From the sounds of it, he was treating the model more like a colleague he could bounce ideas off, not something to be treated as infallible (much like another student I guess).
Asking it to teach you something and cite sources is a massive stretch. People are just trying to trip it up on a dumb use case. It's pretty fun to trip up an AI, but it's not really surprising.
The primary use case of a search engine is to find sources of information on a particular topic. Embedding an AI chat bot in your search engine is very much encouraging users to treat it as something that can find sources of information on a topic.
However, it gets me an answer to my question in seconds almost always faster than searching. Traditional googling will have me wading through forum posts and bad documentation which almost always takes much longer and sometimes completely fails or takes hours to get the right answer.
Fast occasionally wrong answers are way more helpful. I’m quickly switching to first asking chatgpt for an answer and only switching to google or official docs when the answer doesn’t seem to work.
I absolutely don’t “trust” it, but I don’t have to. I try its suggestions and if they don’t work move on to other sources of information.
That is not journalism. It is the opposite of journalism.
I agree that there are plenty of outlets that might qualify as news-shaped, or perhaps news-flavored. They might use AI-rewritten press releases. E.g., CNET's private-equity slide into untrustworthiness. But these things are basically parasitic on actual journalism. So although this is technically a use case for ChatGPT, it's a societally negative one.
While there is some valid discussion that needs to be had about the short comings and application of AI (LLMs, neural nets, diffusion models, etc), lumping it to the out right Ponzi scheme of crypto seems either disingenuous or ill informed.
Can you elaborate on how they are similar so I can better understand your point of view?
There's a large amount of cost sunk into the idea of conversational AI from energy use (not just during training, but data storage, on-going compute, etc), to the cost associated with businesses employing people and the personal investment that folks have invested via degrees and time. All of this creates a pretty deep trench worth defending by various interests. These things clearly aren't ready to be put into search engines, yet here we are, and there are people defending it. That's the original grift that makes it akin to crypto if you believe the real problem in crypto wasn't the tech itself but people pushing these concepts into mainstream use before they'd really had time to be vetted and fully understood. The societal impact was massive and on-going. There are also grifts that come from the seed grift, which gets into AI plagiarization and how to train these models. When you add it all together and then frame it under the premise that it's now in one of the most trusted and prominent places in search, I feel justified calling the underlying activity a grift.
Silly opinion. This is the first wave of conversational AI, and you are calling it quits. This is like "I own a Model T and I'm 100% convinced we'll never make a better vehicle."
The use case, at minimum, is: customer support, answering phones, taking orders. Trained on specific data sets and told not to veer outside. Within one, maybe two more iterations of AI (maximum) we will be there 110%.
ChatGPT was released widely this year, and there are so many absolute statements about what AI will or won't do. It's frankly silly and maddening.
> customer support, answering phones, taking orders
My secondary fear of this tech is that it will be used for these purposes. Especially support.
ChatGPT on the other hand does a lot of tricks, but trying to fit it into the real world is challenging. Even using it in programming requires someone to double check it's work. The idea that it can handle customer support I think is very dangerous. In an industry that has done the opposite of creating fulfilling customer support experiences we should be wary of filling that void with an LLM that's fraught with factual errors in output.
That's to say, ChatGPT does novel things, but nothing useful beyond fantasy (I do hear people talking about using it for RPG characters - which is fair). Ironically, many of the commenters here responded the way cryptobros did when their tech was regarded as useless, which is telling about where this is going. The problem wasn't the tech, the problem was the inability in everyone around it to sit back and acknowledge that how they described it did not match what people wanted and experiences on the ground.
Nice jab.
I think if you produce a novel technology and make it free then people will use it. If you feel confident that these things are "useful" then have people pay for it. Will they still use it for beyond entertainment then?
CoPilot, after an extensive free preview period, garnered something like 400k subscribers at $10/m. I'm curious to see where that number settles at over time. My hunch is that the things you're seeing people use ChatGPT for are more novel than useful and a lot of the value prop is that it's currently free.
To put this in context, GitHub has $1B ARR and roughly 90M active users.
Now consider that ChatGPT is able to subsume it https://www.reddit.com/r/ChatGPT/comments/112z2j4/i_built_a_...
This is only one application among many. PS people _are_ paying for it, the premium is $20/month.
So yes, you are blind on this, sad to say.
It's easy to tear what I said apart if you fixate on the Model T. But the point of my comment is the potential GROWTH of the technology.
If you looked at a simple engine designed to chop firewood. You might say, there's no way the entire geography of the United States will be altered with highways based on this invention. Look at it, all it can do is cut wood! And who will be there to hold the wood in place? It doesn't know how to align with the grain of the wood, etc etc.
And imagine saying this within the first year of said motor's invention.
That is what people are doing with ChatGPT.
Now the brand mostly just produces traditional electric scooters and the self-balancing model found a small but legitimate niche among people who would otherwise patrol on foot all day.
Not everything that looks exciting meets all its revolutionary ambitions.
Whereas, what are the equivalent improvements to be made to a scooter?
Model T you could obviously compare to a Tesla. Which is how you would compare, say, a LLM to AGI.
But what are you going to do with a scooter? Make it fly? Add a cabin? lol
> ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.
This is definitely not my area of expertise, but intuitively, it looks like increasing the complexity/varying the training techniques can increase the likelihood of correct answers, but I think the need to give the model leeway to let it work means that ultimately, either human or automated fact checking will need to be incorporated when using this kind of model for fact-finding questions.
Personally, I've been able to use them to summarize / rewrite various topics that I am interested in finding out more about but don't want to go to 6-10 different sources to find out more about.
So I think they can at least take away some of the drudgery in doing research / summarization work.
For creative writing (this isn't limited to stories!) or slightly more contextual boilerplate, I love it. I think of it like a slight upgrade to what I already do, to allow me to get to the steps that really matter, not this revolutionary new world people are imagining it to be.
- Looking for suggestions for a birthday, with some constraints that make most recommendations impossible. I didn't go with anything straight from the suggestions, but I've went with a personalized variant of one of those.
- Looking for suggestions for resources and books on niche languages and read languages.
- Looking for documentation on how to do a certain thing using a certain app or framework, several times.
It seems to me that ChatGPT is no worse in its recommendations than what an expert can give you recollecting what they encountered a year+ ago (in addition to the training data cutoff) without the opportunity to double check if they remembered correctly. It's immensely useful in the same sense that the expert solves most of the discoverability, and leaves you the relatively easy task of sanity checking.
I'm not sure about Bing, but I'd expect similar or better, or eventually similar or better.
People point to an oversized vehicle but will defend their oversized TV with their lives. Whilst rationally speaking there are very few "monster truck" vehicles whilst almost everybody has such a TV. The "do good" factor of banning/taxing the TV would be infinitely more impactful.
People hate miners, not for their energy use, instead because they drive up the price of high-end GPUs. High-end GPUs, you know, which gamers intended to use to do high-energy use gaming.
Both cases (insanely large TVs and high-end gaming) would be defended by them delivering value or purpose.
Well, guess what, the planet doesn't fucking care. There's no such thing. And if anything, neither use cases are in the realm of life support systems.
The average TV (55-inch 4K LED) will use 5.7 kilowatt-hours (kWh) of electricity per month.
A traditional car uses around 40+ kWh, an EV about 30kWh per 100 miles.
Personally, I haven't had children. You're welcome.
A natural language interface and the ability to maintain context over a conversation makes it incredibly easy to interact with.
Have you tried to do anything productive with it? If you're just using it as a front end for search then IMO you're missing most of the potential.
I actually play with it a little every day, along with some other models. I'm an end user at the end of the day and a QC at best. The most I've gotten to do that was useful is all what I'd file under "novel". Using it for programming is haphazard, even summaries are a bit fraught because it will feed you wrong information. If you have to verify the critical points of the entire summary then you are only left with a jumping off point, which some random person's blog or "relevant search results" already does for me.
As for "legitimate", I meant in the context of a business using it to make or save money (ie: assisting in search). If it lies, or is incorrect, putting it in front of users with a facade of trust/legitimacy seems awfully dangerous. Even image generation is a bit fraught with respect to Stable Diffusion and DALLE, but folks aren't trying to stuff those models into mainstream use cases, so I consider them less a grift. To me, that presents a lack of legitimate use cases.
There's tons of use cases. One of the use cases is definitely NOT startlingly accurate citations. But you're delusional if you think there isn't any use cases.
There is a large class of AI researchers who have been working on this problem for decades (and I am one of them). What you see right now is the tip of the iceberg. Compared to that, crypto was invented by an anonymous dude in some random forum. Remember that what you see in the surface is not all there is, and thinking so is a very shallow interpretation of things.
This piqued my interest. What more can we expect to see than what's already available for the public to use?
Which problem is that? It's not clear from the context and I don't want to make uninformed assumptions.
It is true that many people have worked on problems like language understanding, language modelling and language generation for many years. It is also true that most people who are today so excited about large language models don't know anything about all those years of research. My experience is that, at some point, there was a switch from computational linguistics (i.e. doing linguistics with the help of computers) to statistical natural language processing (i.e. trying to represent language with statistics). The idea behind the switch was that understanding human language, its mechanisms, its origins, and so on, is too hard, and we can make a lot of progress if we stop trying to do all that, and instead turn to learning statistics from large corpora (cf. "every time I fire a linguist...").
Of course, without good understanding of what language is, and how it works, it is very difficult to evaluate such progress, but the answer to that was obviously to establish new metrics, and then try to maximise performance according to those metrics; metrics such as BLEU or ROUGE and friends, that are suspiciously like the kind of metric that one would choose if one wanted to measure progress as the ability to learn statistics from a large corpus.
So it's true to say that many people have worked on all sorts of problems to do with natural language and AI, but I'm still left wondering exactly what you mean by "this problem". For me the problem is that an entire field of research ("NLP") abandoned the search for answers to interesting scientific questions and turned instead to the production of toys, and to spectacle, presumably because that's where the money is and researchers must also pay them bills.
And so now we have large language models that are better at reproducing language than earlier models, but we don't have the tools to evaluate or understand those models; and so people who aren't experts, but also people who are experts, can't separate the wheat from the chaff, and keep coming up with wild, senseless proclamations that could well have been generated by a language model themselves.
As for the edginess of what I said, if you take it in the context of money-making and society-affecting use cases it is a nothing burger. Maybe you would have preferred if I highlighted the use cases for fantasy RPG or story telling in general. The grift is in the people trying to push this into things like search and customer support before they can figure out how to make it stick to truths, which makes the energy waste and dollars spent become a very deep trench to defend.
But sadly, that may be evidence that there's a market here. If Amazon can cut support costs by 80% and only moderately worsen their already bad quality, some execs would consider it a success, not a failure.
So the flaws of chat AI currently are exactly what make art great. And art came before science, so perhaps scientific thinking is much harder on an evolutionarily timescale than we think.
I used ChatGPT to give me a list of test areas for a certain type of scenario. And it actually pointed out one that I had inadvertently missed. Now if it said something off the rails I would've known that too.
I've probably had 100 conversations with ChatGPT and I can only recall one blatent lie that it told, which I think was P != NP. Which maybe wasn't a lie, but it didn't have sufficient evidence to make that claim.
I've actually so far gotten more accurate information from ChatGPT than I did in Wikipedia's earlier days -- where I discovered that Matt Damon dropped out of Harvard because he was too stupid to make it there.
They can certainly be wrong, though.
The fact that you don't, or have managed to pass that as anthropomorphizing is curious. I'll have to think about that, but on its face seems strange alongside your comment about if someone can't find a legitimate use case for AI that they shouldn't be a person who summarizes it.