> Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues.
> In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.
I would argue that national politics and international affairs as they are covered in mass news media is a different type of coverage. It’s not about learning technical details and explaining them. It’s more about observing actions and broad sentiment and literally reporting those observations. I wouldn’t trust a reporter to get scientific details correct about some chemical process that’s the subject of some debated EPA regulation, but I could trust a reporter to summarize the debate and report on the sentiment of politicians, public sentiment based on opinion polls, etc. It would be weird to say “that newspaper screwed up the difference between a dominant 7th chord and a major 7th chord in a story about a famous songwriter, therefore I can’t trust them to report which politicians gave the most heated arguments in the debate about the EPA bill.”
https://www.valleynewslive.com/2023/03/09/fargo-cbd-store-ow...
"“Semi-synthetic derivatives, or chemically derived cannabinoids, refer to certain types of substances that are produced by converting a cannabis extract into a different substance through chemical reaction,” said forensic scientist with the North Dakota State Crime Laboratory Charlene Rittenbach at the March 3 committee hearing of the bill.
For example, according to the Addiction Prevention Coalition, to make THC-O: you need to extract CBD, extract Delta 8 THC from the CBD and then add acetic anhydride.
The National Library of Medicine says acetic anhydride is corrosive to metals and tissue, and it’s used to make fibers, plastics, dyes and explosives."
Now, instead of authoritative sounding humans who are possibly wrong or with an agenda, we have an "infallible", "impartial" oracle capable of inventing whatever it wants to. Yes, I have seen ChatGPT treated as infallible on this very forum: "you are wrong because ChatGPT says <insert nonsense here>". Can't wait to see LLMs taking "post-truth" to a whole new level. The propaganda potential is immense, to point out just one application.
But underneath all the bullshit is something truly useful, so I wouldn't necessarily say that Confidently Incorrect™ is ChatGPT's primary ability.
Coincidentally, over the past few weeks of this ChatGPT craze, I've needed to create a lot of fake data to seed a database. Normally not a big deal, but these records need to have foreign keys pointing to one another. I wondered whether ChatGPT could help me out, so I briefly described the fields I needed, their format and data type, the type of information they contained, etc etc. It did it almost perfectly, and fixing the little errors was trivial. I was dreading having to do this because it's such a pain in the ass.
To me that kind of thing is going to be the most useful application. Everyone's freaking out (in both the "scared" and "excited" sense) over AI's ability to replace creativity, but I'm focused on it's ability to replace toil.
Meanwhile ChatGPT is already being hailed as the Google killer. People are using it to write essays and form opinions. Plenty of online and offline debates contains phrases from ChatGPT repeated verbatim. It's being adopted in newsrooms around the world. So what it is good for in theory and what people are actually using it for are worlds apart. IMO the technology is going to have its "self driving car kills pedestrian" moment very soon because of over-eager and careless users, and that is going to set the entire field back.
If you approach it with the right mindset, it’s extremely valuable.
The only Google-killing aspect of it is a better understanding of user queries.
Although given how often I have to remind it of the details of my questions, maybe it's not even good at that.
No, lack of ads. Before content marketing, Google was pretty useful
Add in all the automation of customer service via actually good Chatbots and lots of very cheap (and fairly high quality) copywriting and it seems world changing to me.
Reddit tends to be a second best bet, and the most upvoted is not always the best, on a busy thread some ‘real’ experts are usually a bit down with lengthy descriptions of their analysis.
Search is third, it’s quick but I do often need to append stackoverflow and reddit to get meaningful answers. I forgo Google completely more and more since the quality is so low, perhaps 2% of my queries go there tops.
A script which parses jaon for a specific key or transforming a table from y to z.
Now everyone has this available to them.
It's great already!
Oratory prowess is way more important than accuracy. If you ever want to "lose a debate" on reddit, pick something that's counter-intuitive or widely misunderstood and use nuance and citations in your defense. You'll get cyberbullied every time - sometimes even banned from the sub.
It’s hard to be surprised that “eugenics, but the good kind, just hear me out!” and “state’s rights, but not the disingenuous lost cause kind, just hear me out!” are a tough sell with complete strangers who have no prior reason to trust you or be interested in your opinions.
Btw, I removed them from my examples, didn't want to "discuss" that here.
I'm kind of a history nerd and it's like a fishing hook with delicious bait when I see stuff like that. "Oh wait! You think it was just the south that argued states rights?! Actually (brings up fugitive slave clause jurisprudence)"
Or how manner and conduct books of the 1800s, eugenics texts, and jordan peterson's 12 rules for life are basically dots in a straight line. That stuff is just catnip.
I just need to stop being distracted and go write actual books or something.
There were multiple points of interest for the Eugenicists. One of them was for the general public and how to get them to have a proscribed conduct and behavior.
Maybe you've seen those diagrams where they have 2 life paths depicted from the Victorian era. That's from Eugenics. Example: https://ia800301.us.archive.org/BookReader/BookReaderImages....
Here's "The Road to Success" for "The Young Man" https://archive.org/details/naturessecretsre00shan/page/244/...
Again, this is part of Eugenics. We focus on the part that was removed (race science) and leave mostly unexamined the parts that were reinvented or repurposed as say, "The Power of Positive Thinking".
Eugenics books, for the general public, were a different beast. Probably the best succinct description for how they thought of themselves can be summarized in this 1921 diagram: https://images.prismic.io/wellcomecollection/efc97918-8073-4...
It is a dangerous psuedoscientific epistemology - a practice with a bunch of faulty logic and ways of knowing that can lead to disasters and it's alive and well - for instance, assuming a social structure in lobsters applies to humans. That these adherents tend to be revealed as secretly racist and thus subscribe to the part which was removed, should be of no shock.
We've mischaracterized Eugenics[1] and allowed it to fester by other names or as Maurice Bardèche, the french neo-fascist observed about fascism in the 60s, "With another name, another face, and with nothing which betrays the projection from the past, with the form of a child we do not recognize and the head of a young Medusa, the Order of Sparta will be reborn"
The point is that if you don't draw the perimeter right around an idea, you can't reign it in. It'll just escape from your hands and re-emerge as something else. So, for instance, the slave plantation becomes the prison farm (https://en.wikipedia.org/wiki/Prison_farm).
We tend to, for good reason, ignore societal trainwrecks and blights on our past. But trainwrecks are the most important thing to study if we strive to build safer trains and seek to avoid similar disasters in the future.
Things have to be accurately tackled with in order to defeat them.
---
[1] it's worth noting it's a big topic and some "real" science has also come out of it, such as biosocial theory and biostatistics studies. Peer reviewed journals such as https://en.wikipedia.org/wiki/Journal_of_Heredity used to be eugenics journals. This is, however, a very small minority.
There is little accuracy in your description of events. Peterson said we share serotonin with lobsters and it fuels a victory/defeat set of behaviours. If you drop the serotonin from his idea, it's no longer what he was talking about.
Your description falls to my mind as a velvet blanket to hide underneath from scary racist and nazi ideas that have nothing to do with Victorian era manners and metaphors about behaviour, expanded out of natural science.
You can't draw a 'perimeter' around the wind.
This type of literature has been a hobby of mine for years and recently I've noticed that I've given myself some expertise.
Hopefully I'll be motivated to write a more complete volume in the future. It takes many pages to build proper context here.
Thanks. I've read your reply but I'm trying to be judicious with my time these days.
Good luck.
When i was younger the things Jordan Peterson said actually helped me. This whole "Reverse Midas Touch" thing is getting really out of hand. Just because a person has had bad ideas, doesn't make the person bad or require destruction of every idea they've ever held or spoken. Just because a group of racist and ignorant people held bad beliefs, doesn't mean every possible belief they held was bad and must be destroyed. This whole argument is just, "but the bad people thought that so it must be bad too", without ever establishing what's actually bad about it. Tell me what's wrong with the ideas without mentioning a group of bad people, tell me about why the ideas are bad. Tell me about the people who "manners and proper behavior" has hurt, and then explain to me why it can't possibly be different. Speak about the ideas, not the people. Everyone knows why those people were bad. Now tell me about the ideas
This type of literature has been a hobby of mine for years and recently I've noticed that I've given myself some expertise.
Hopefully I'll be motivated to write a more complete volume in the future. It takes many pages to build proper context here.
Thanks. I've read your reply but I'm trying to be judicious with my time these days.
His biggest flaw is that (like the other Daily Wire pundits) he never goes into economics and has absurd ideas that we live in a competence hierarchy.
The whole "intellectual dark web" imploded after the start of the Ukraine war, when they first went full neocon, then realized that their audience does not like that and now start backpedaling massively.
It's still lacking but I should "put up or shut up". They're doing a fine job
YC News people generally seem to prefer to hear the truth, especially from sources that appear knowledgeable. I've had people here comment in reply to me saying I've "changed their mind". That never happens on Reddit.
Reddit has cliques, quasi-religions, and fads. Anything contrary to the group-think is mercilessly voted down. Intelligent, thought-out counter arguments that aren't entertaining are especially likely to garner lots of downvotes. Not only do the readers feel "personally attacked" by contrary opinions, there is also that ever present undercurrent of anti-intellectualism of typical Americans that makes intelligent discourse distasteful to them.
For example, I ask it to simulate a discussion between a flow engineer and a geologist, and it doesn’t seem to understand they work together regularly.
Not like there’a a stack overflow for this sort of thing.
its latent representation is vast, I find you can get much higher quality+interesting responses if the prompts are tuned appropriately
It's sort of like, how would you imagine an expert would respond if all you said was "simulate a discussion between a flow engineer and geologist". Without providing any further context or hints into what you want, I also wouldn't be surprised if I get a generic response. But instead if you sort of know what you want to ask, they'd have more to work with and be able to better answer your query.
Note I know nothing about this field, I just looked at https://en.wikipedia.org/wiki/Engineering_geology#Soil_and_r... and understood this was for oil and gas, so:
> Simulate a discussion between a flow engineer and a geologist in the planning phases of a drill operation. Use field-specific terminology and aim for accuracy, the scenario will be used to teach a petroleum engineering class focused on deformation monitoring.
Flow Engineer: Good morning, I'm the flow engineer for this drill operation. Nice to meet you.
Geologist: Hi, I'm the geologist in charge of the geological analysis for this project.
Flow Engineer: Great, I understand we need to work together to ensure the success of this drill operation. Can you tell me more about the geological features of the area we're drilling in?
Geologist: Sure. We're drilling in an area with significant deformation, caused by tectonic activity. There are a variety of rock formations, including sandstone, shale, and limestone. There's also a possibility of encountering fault zones, which could pose a risk to the drilling operation.
Flow Engineer: Thanks for the information. We'll need to take into account the deformation and rock properties when designing the wellbore trajectory. Have you conducted any geomechanical analysis?
Geologist: Yes, we've conducted a rock mechanics study to determine the stress and strain characteristics of the formations in the area. Based on the results, we recommend using a multilateral well design with a horizontal section to avoid encountering the fault zones and minimize the impact of deformation.
Flow Engineer: That's helpful. We'll need to ensure that our drilling fluid properties are appropriate for the formations we'll be encountering. Can you tell me more about the porosity and permeability of the rock formations?
Geologist: The porosity and permeability of the sandstone formations vary, but in general, they're relatively high, which means we'll need to be careful when selecting our drilling fluid to avoid fluid loss into the formation. The shale formations are less porous and permeable, which means we'll need to use a different type of fluid to avoid damaging the formation.
Flow Engineer: Thanks for the information. We'll take that into account when selecting our drilling fluid. We'll also need to monitor deformation during the drilling process. Do you have any recommendations for deformation monitoring tools we should use?
Geologist: Yes, we recommend using acoustic emission sensors to monitor rock fracturing and deformation, as well as tilt sensors to monitor any changes in the wellbore inclination. We also recommend conducting periodic borehole imaging to ensure that the wellbore is remaining stable.
Flow Engineer: Excellent. We'll ensure that we incorporate those tools into our drilling plan. Thank you for your time and expertise. I'm looking forward to working with you on this project.
"Simulate a discussion between a project manager and a public health officer in the planning phases of a drill operation. Use field-specific terminology and aim for accuracy, the scenario will be used to teach a petroleum engineering class focused on public relations."
GPT: > Project Manager: Good morning, I'm glad to have you here to discuss the planning phases of our upcoming drill operation. As a project manager, my priority is to ensure that all aspects of the drill operation are conducted in a safe and efficient manner.
> Public Health Officer: Good morning. As a public health officer, my role is to ensure that the drill operation does not pose a risk to public health and the environment. I'll be happy to discuss our strategies for achieving that goal.
> Project Manager: Great. One of the first things we need to do is to identify potential health hazards associated with the drill operation. As you know, drilling involves the use of various chemicals, which can pose a health risk if not handled properly.
> Public Health Officer: Absolutely. We'll need to carefully review the chemicals being used and ensure that they are being handled in accordance with proper safety protocols. We'll also need to develop emergency response plans in case of accidental spills or leaks.
> Project Manager: Agreed. Another concern is air quality. The drilling process can generate dust and emissions, which can have adverse effects on air quality.
> Public Health Officer: That's right. We'll need to monitor air quality throughout the operation to ensure that it meets the standards set by the Environmental Protection Agency. We'll also need to take measures to minimize dust and emissions, such as using dust control equipment and properly maintaining all equipment.
> Project Manager: And what about water quality? We'll be drilling in close proximity to bodies of water, and we need to ensure that we don't contaminate them.
> Public Health Officer: Yes, that's a critical concern. We'll need to make sure that all wastewater and other fluids are properly contained and disposed of. We'll also need to monitor water quality regularly to ensure that there are no leaks or spills.
> Project Manager: It sounds like we have a lot of work to do to ensure that this drill operation is safe and environmentally responsible.
> Public Health Officer: Indeed. But by working together and following proper protocols and procedures, we can ensure that the operation is successful without posing a risk to public health or the environment.
I guess there was some confusion about roles/at the existing temperature roles swap (if you rerun it, you get different results, essentially swapping the lines...)
EDIT: and third try actually arrives at really nice hallucinations:
> Project Manager: Good morning, Dr. Smith. Thank you for taking the time to meet with me today. We are in the planning phases of a drill operation and we want to ensure that we are following all necessary public health protocols. Can you provide some guidance on what we should consider?
> Public Health Officer: Absolutely, it's my pleasure to help. The first thing you should consider is ensuring that all workers on-site are up-to-date with their immunizations, particularly for diseases such as tetanus and hepatitis.
> Project Manager: Understood. We require all our workers to have up-to-date immunizations before starting work, but we will double-check to ensure that all workers have the necessary immunizations.
> Public Health Officer: That's great to hear. In addition to immunizations, you should also consider providing training on proper hygiene practices, such as hand-washing and avoiding contact with contaminated materials.
But I feel that if one is inquisitive enough, they'd know when exploring a topic to ask broad questions of the sort:
- what are the common types of problems encountered in X field
- what are the main roles and responsibilities of personnel in X field
- list some open questions in the field of X
After these open-ended questions one can dive into each of them with:
- what are some common solutions to X, compare and contrast X vs Y
The risk of seeing hallucinated info is always there, but just for an example when I was exploring the types of space-time metrics and asking chatgpt which tensors are most suited for which scenario (and also which coordinates are best for which scenario, such as moving from Schwarzchild to Eddington Finkelstein coordinates to deal with the event horizons near black holes), their individual wiki pages said generally the same things.
I was able to ask ChatGPT in very naïve "idk please explain like im 5" ways and it was able to very patiently rephrase dense jargon into surprisingly understandable explanations.
I'm sure if an expert in GR were to drill down deeper into the details they will find inaccuracies in ChatGPT's responses, but for the inquisitive information sponge user, a dive through ChatGPT to get a general feel of a topic is insanely useful. After that, one could dive into the appropriate books/papers/primary sources to start the real learning once they find what they're interested in.
No context about anything in the outside world or even about the physical nature and purpose of commuting is needed to comprehend and answer these questions. There is no external context or sophisticated logic beyond just matching those sorts of lists to each other. Its a wholly closed system of highly standardized tokens.
I ask it what stops the F and A have in common. 125th St is on the list. I know that's wrong. I ask it to list all the services that go to 125th. The F is not on the list (correctly so). I point out the inconsistency between the two outputs. It says sorry and does nothing. I tell it to remake the list of stops the F and A have in common. It is now missing a few stations from before, and has added new incorrect results.
This is just a snippet. I went in circles with it for probably an hour playing wack a mole with its inability to correctly recall more than 10 true details in a row. These were not obtuse, esoteric, or even logically complicated queries. Nor was it ambiguous stuff open to interpretation. Nor was it even something that you'd need a real meatspace body to comprehend, like the feeling of the sun on a summer day. This should have been a language models bread and butter.
Would be interested in how a multimodal LLM such as PaLM-E (trained on maps, etc) fair in these sorts of queries.
https://www.reddit.com/r/MachineLearning/comments/11krgp4/r_...
Here's another one that fails spectacularly. The digits 0-9 drawn as an ASCII 7 segment display. It gets it mostly correct, but it throws in a few erroneous non-numbers and repeated/disordered/forgotten numbers. Asking it for ASCII drawings of simple objects can really go off the rails quickly.
The fault mode is very consistent. When a prompt forces it to be specific and accurate on an unambiguous topic for 10 or more line items, it will virtually always hallucinate at least one or two. Especially if the topic is too simple to hide behind a complex answer. Even if its learned not to hallucinate 90% of the time, and even if that's good enough to pass at first glance, within a list of 10 things it only has a 35% chance of not hallucinating any of them.
For what its worth, it did very well on law questions. Try as I might, it refused to accept that there's a legal category known as "praiseworthy homicide". Though, I suspect this has less to do with the underlying model and more to do with openAI paying special attention to profitable classes of queries.
I'm sorry to say, I think the problem maybe more intrinsic to the current approach to AI. Frankly, LLM works unreasonably beyond my expectations, but its making up for a lack of a real first principles theory of cognition with absurd amounts of parameters and training.
edit: minor typo
I’m finding that it’s fairly good for code snippets. You can test that pretty quickly. But asking it something you don’t know and can’t verify is a huge risk.
I have some expertise and knowledge is one or two specialised fields, and when I read Wikipedia, news articles and blogs about those fields, it's maddening how incorrect many people are, whilst sounding perfectly knowledgeable.
Sounds like the old days to me, where we would meet up , discuss the world and what have you, without any means to fact check anything online.
Changes there was a real expert on the subject in your circle was at best 20% So we just enjoyed / continue our life. No harm done as long as nobody took the discussion as gospel.
I wouldn't rely on ChatGPT or any system that relies on it, except perhaps for summarising/rephrasing a single, known document. It worries me that people are using it for software development - creating stubs, bootstrap code. It's perhaps OK if you know the language/platform well enough to fix issues, but if you then use it on something that your not an expert in, what is the outcome.
For example:
Finding the source of literary quotes - miss attributing them - took 5 or 6 QA responses for it to get close.
Explaining how a p-channel enhancement mode MOSFET works - explained n-channel
Explain regex - often gets the technical parts right but can make very bad suggestions on what the regex aim is.
Name some classic cocktails created in Europe - suggested Manhattan, which it said was acceptable because it was popular in Europe
Suggest some gin based cocktails that don't include citrus, suggested those with lemon juice.
It's especially common on Hacker News.
https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...
>Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
>In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.
Yep. I asked it to summarize a URL. It happily invented something completely fabricated and convincing sounding (until you go to the URL) based of the domain name and article slug.
Setting aside the creativity aspect (ie where do brand new ideas come from), which is a minuscule fraction of human thought, I suspect we are good pattern matching machines with the ability of extrapolation.
I think with sufficient data that computers can understand, this is eminently replicable.
These misanthropist slogans run up to contradictions so easily.
At the time it _seemed_ like it wasn't just doing a mindless string replace, because different parts of the url changed on most of the erroneous links. I figured it wasn't going out and spidering the link I gave, but it led me to believe maybe there was _some_ form of index that it simply hadn't prioritized prior, but was able to do with the additional context.
The training data however can be a few years old.
I have noticed this when trying to get it to generate some code, it was clearly using considerably old versions.
I have noticed the same thing when trying to use junior devs to generate some code, usually correlating with how old the most popular Stack Overflow question on the subject is.
OpenAI handed this tool to people with effectively no guidance or instructions so that they can trade on the glow of misconceptions and harvest user data. The whole point of giving the thing away for free these last few months is to let people make fools of themselves like this while pumping up OpenAI’s value as a technology firm. It’s no accident that they don’t engage to clarify all these persistent misunderstandings.
If I want to know what chatgpt thinks about something, I can ask it myself
When one logs on to Chat GPT, they get the following notices:
> This is a free research preview.
> Our goal is to get external feedback in order to improve our systems and make them safer.
> While we have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content. It is not intended to give advice.
> Conversations may be reviewed by our AI trainers to improve our systems.
> Please don't share any sensitive information in your conversations.
Effectively no guidance or instructions indeed. Perhaps ChatGPT is sometimes confidently incorrect because it learned from online commenters.
None of them say what the product is, what it does, or how to use it — which are the kind of thing one might call guidance and instructions.
Those are just disclaimers to make sure you don’t hold OpenAI responsible for anything you do with this thing, or blame them for anything they do with your attempts.
It is not a product. They open it so that public can contribute to help them.
They do not say the aim of this research.
It's very unlikely the model weights would contain enough detail to let it match content to the URL that the content originally came from.
ChatGPT by itself can't but if you build a product on top of its API, you can fetch the url itself and send the data in the ChatGPT's prompt
Here is an excerpt from my chat:
" can you please modify the current array using the Microsoft listed CPUs in totality from the URL https://learn.microsoft.com/en-us/windows-hardware/design/mi...
Certainly! Here's the updated $compatibleCPUs array based on the list of supported Intel processors for Windows 11 version 21H2 from Microsoft's documentation:
"
Here's an experiment I just did to prove that it hallucinates content based on URLs: https://fedi.simonwillison.net/@simon/109998914078891574
Or try it yourself: Try these two prompts in separate ChatGPT sessions:
> List the processors on this page: https://bit.ly/422t51T0
Then:
> List the processors on this page: https://learn.microsoft.com/en-us/windows-hardware/design/mi...
That Bitly link is one I just created that redirects to the Microsoft page.
Try changing a few characters in that Microsoft URL and see if you still get the same results.
I just built this locally over our RFCs and it's good but a properly integrated thing would be fantastic.
This might not be entirely true. People were able to fetch urls and data from the internet that couldn't possibly be in any training dataset (in eg posted the same day for the first time).
https://medium.com/@neonforge/i-knew-it-chatgpt-has-access-t...
> But ChatGPT has rewritten my article in a different style, what a interesting finding…
The URL conveniently contains the topic of the blog-article, meaning ChatGTP can 'hallucinate' the contents, even without internet connection.
To show this, I tried to follow the same steps as @neonforge, but with a non-existing URL. Unfortunately `lynx` 'didn't work', but curl showed some interesting information.
`curl https://medium.com/@peter/learn-how-to-use-the-magic-of-terr...` returned some html. Instead of a 404 and a <title> element containing just 'Medium' it hallucinated the following title tag: `<title>Learn How to Use the Magic of Terraform Locals (Step-by-Step Guide) | by Peter | Medium</title>`
I'm not saying that ChatGTP definitely has no access to internet, but so far I have not seen any proof or indication that it does.
If anything, this blog post is a perfect example about how people can put in whatever they want as an input and take the output as truth, without any rigorous approach about what would count as a true fact about the model.
If you want to prove this to yourself, tail a server log and paste a URL into ChatGPT and see if you get a hit.
It convinced him of what he wanted to believe. He didn’t do a very good job of testing.
You can still do useful things with it, that much is extremely clear. But it doesn't think.