LLM Fight Club
llmfight.club
llmfight.club
I've thrown about a hundred prompts at this, half of them phrased as neutral questions, the other half as statements comprising the position of one side of the debate. I notice that when doing the latter, LLM A always takes the position I state, and LLM B is always said to "win" the debate, and the form of this is usually the same: B claims that there's insufficient substantiating evidence for A's claim. Neither says much substantial about the topic of the prompt after the first exchange. Usually A's first paragraph will expand a little bit on the topic and B will mostly try to discredit the arguments made, then they'll repeat this three or four times while insulting each other's intelligence without bringing up anything new, expanding further on the topic, or notably, trying to use any examples aside from the ones initially discussed in the first reply. What's interesting about the "neutral question" style is that the debate tends to go about the same, but the assessment at the end only considers B to have won about 90% of the time
As someone who studies AI and has gotten into a lot of internet arguments, this is a fascinating example of how LLMs are great at capturing the average case with essentially zero chance of capturing the best-case of the format, in terms of being informative on the topic at hand to a third-party observer
I kind of wonder if systems of LLMs are going to become its own new entertainment medium
Entertainment could just become a prompt example dive off point for the user to remix to their liking.
Sometimes hosts/guests gloss over topics as they are trying to sell their book as the one way find out more. LLMs can enhance the conversation as if the host drilled in a little further on some sore point similar to if it were a popular post with lots of comments. Not that those will be free of LLM influence for much longer.
Like I'm aware that 80% of econ and business talk is gonna be BS anyway but turbocharging that effect seems unwise
For some reason people trust the error rate of the internet over the single entity LLM, it all depends on correct use and remaining sceptical. Either way the same gambit that got everyone online will get everyone on LLMs: I'll 1000x your knowledge for some low % error rate (reducing every month).
It's like muskets that didn't shoot straight either when they first arrived but you could compensate and you'd be unwise to stick to the broadsword because of tradition. Ask Scottish highlanders.
As I've said before, I am using LLMs in my workflows, primarily in places where either factual accuracy doesn't matter (ideation/elaboration is a great use of any generative model) or it's feasible to both test the output in a tight feedback loop and obtain primary sources from which it's drawing information. They're also exceptionally good at translating fuzzy questions into actionable search terms, a use case that google's been trying to approximate poorly for years and which probably would have made a much bigger splash as the gamechanging practical NLP result it is if they hadn't (it looks like incremental improvement from an end-user perspective but decidedly isn't). Everyone who tells me how "efficient" they're getting from adding yet another captured passive stream to their already oversaturated information diet seems no more competent or knowledgeable than prior to the widespread usage of LLMs, but they are about 1000x more smugly assured I'm a luddite for thinking their unfathomable credulity in their naive use case being the future is overwrought (and are, statistically, also likely the people who buy the smear campaign by industrialists that's resulted in the modern usage of that word)
From my perspective you're the guy swinging the musket like it's a broadsword, and I would very much like for fewer people to be doing that with such enthusiasm, given the hazard to benefit ratio. But at least you're not the guy trying to chop vegetables with the musket, which unfortunately is the level at which a decent chunk of the business world currently functions in this analogy
Lies by omission are worth railing against even if the first guess at those omissions are wildly wrong, the important thing is you've started the refinement process instead of letting the implied falsehood fester and mix with other choices you might make. The hazard of LLMs is only for people who expect to be babied. The people who want no one else but the elite knowledge experts using LLMs are just chuffed at their own trivia recall abilities and perhaps a little afraid of being easily shown up after dedicating so much effort to the legacy way.
Your proposed model for adoption doesn't "show up the elites", it destroys the ability of anyone who doesn't already have elite-level epistemic integrity checking skills to ground themselves in reality, and honestly may well over time subvert even decent epistemics via automation blindness. Using new tools is something that people should absolutely do, but using them stupidly for the sake of using them creates a small-to-modest momentary edge for an infinitessimal fraction of this especially reckless class of early adopters who luck out while most of them just serve as guinea pigs as the world figures out what the things are actually good for and irons out the rough edges. Regulating the way you ingest new information such that you can tell fact from fiction isn't "trivia recall", it's a way to retain a semblance of soundness of mind. It's readily apparent from the increasingly unhinged landscape of conspiracy theories that it is in fact not better to wildly guess at gaps in your knowledge if those guesses have a chance of becoming sticky, and human brains are a lot better at filling in gaps than unfilling them. You're not making a good case for not grounding usage of these tools in a realistic understanding of what they're good at - even setting aside the nominative determinism that's becoming increasingly apparent in your replies
Plato or Hegel this is not. But it is kinda amusing.
Seeing the fight club topics others asked would be an additionally fun and engaging feature. I asked some funny or interesting ones but would like to see others’…
I'll start, "Tabs vs Spaces" ;)
And are LLM A and B different models or just different system prompts, where the system prompt for LLM B includes that it disagrees with the output of LLM A?
- OpenGraph data somehow is for https://claros.so an "AI Shopper"
- It uses nextjs and tailwindcss
- The model says it is GPT-4, but you'll never know if that's true
- LLM B seems to be instructed to 'argue against the topic'
Anyways. The eternal kwestion "Oliko Urho Kekkonen diktaattori?" produced better discussion I have seen in years.