> If LLMs were successfully deployed in a production environment is the day I bite my tongue.
This shows so much faith in management!
This shows so much faith in management!
Feel free to point out any error in my logic:
There are huge financial incentives--tens if not hundreds of billions of dollars--for developing an LLM which can solve novel bugs. So surely there exists AI companies developing an LLM capable of doing so. If an LLM capable of solving novel bugs exists, AI companies would rush to showing it off to capture tonnes of VC money. AI companies could show off their fancy bug-fixing LLM by closing issues on public Github repos using said LLMs.
No such mythical LLM exists. We are thus left with two choices:
1. My logic is flawed or there is an alternative possibility I haven't considered.
2. The LLM capable of doing what OP asserts doesn't exist and can't be made, despite their assertion that it is trivial to fine tune and put into application.
Your original wager was that it would be in production, not that it would work.
What I mean by successful is that the LLM can generate accurate (>80%) incident response reports and propose correct fixes.
I'm fairly certain anyone literate could have read the rest of my comment and parse out what a successful deployment means.
There is a way to disprove that the statement: "There is no god" by simply showing a counterfactual god.
There is however no way to disprove the statement: "There is a god."
Likewise, there is a way to disprove the statement: "LLMs cannot be successfully used for X application." By showing that LLMs have been used in X application.
Again, there is no way to disprove the statement: "LLMs can (eventually) be used in X application."
The meat of my question was meant to demonstrate a failure to apply the scientific method.
that both parties to the argument agree is a god.
>Likewise, there is a way to disprove the statement: "LLMs cannot be successfully used for X application." By showing that LLMs have been used in X application.
Again there the point of argumentation will be the word "successfully", the LLM would have to be such an overwhelming success at what it is trying to do that one cannot weasel out of it with "successfully".
To answer your question more directly, I would go with option 1.
It seems like your assumptions are unfalsifiable. As computer scientists, I believe it's important that our hypotheses are testable. If a hypothesis is unfalsifiable, then the hypothesis is no better than theology and should be discarded.
What's stopping you and other VCs just pouring endless money into an idea that won't work?
Your variant of the "scientific method" would've meant we never discovered electricity, or invented airplanes, or really anything else, because why bother trying? If it worked someone else would've done it.
Are you saying that if someone finetunes a current SOTA LLM with incident response data and demonstrates that it doesn't work that you'll say that LLMs are infeasible for this application? That would invalidate the hypothesis: "X application can be done on current LLMs."
Such a test could never invalidate the the hypothesis: "X application can (eventually) be done on LLMs."
If it's the former hypothesis you were asserting, then yes I agree that it is testable, but I'm fairly confident you were asserting the latter.
Earlier I had asked you: "I think we're at an epistemic impasse here. At what point would/could you be convinced that LLMs are incapable or unsuited here?"
And you have yet to provide a response.
Proving that LLMs can never do this would require extremely rigorous theoretical evaluation that even top ML labs are currently unable to do, given the problem of interpretability. In general proving a negative is typically harder than a positive, since a single experiment succeeding proves a positive, but a single experiment failing does not prove a negative; generally science does not demand that scientists attempt to prove a negative when running experiments, or else nearly every drug trial, for example, would be impossible to perform. Complaining that you have staked out a very difficult to defend position — that it's impossible for LLMs to generate good incident reports — does not mean your ideological opponents, who have simpler positions, must do your proof work for you.
Please refer to this: https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)
Just to make sure we both understand what burden of proof is:
Suppose two people are having a debate over whether or not a teapot exists in the orbit of Jupiter which is impossible to observe via telescope. Where does the burden of proof lie?
Just to reiterate plainly:
Does the burden of proof lie on the person making empirically impossible to falsify claim or the person making the empirically possible to falsify claim?
Which of the following two claims is impossible to empirically falsify?
1. "LLMs can eventually be used to produce good incident reports."
2. "LLMs can never eventually be used to produce good incident reports."
> That does not mean the hypothesis is true, it only means you haven't falsified it.
Correct. You can't prove a negative, you've only just figured this out? After I listed TWO simple examples that could be found in an introductory philosophy class?! In the Wikipedia article consisting of a couple of paragraphs I linked you?
PLEASE JUST READ. PLEASE JUST READ. PLEASE JUST READ. PLEASE JUST READ.
You can't prove negative statements. I want you to admit this so I know you understand, now repeat after me: "You can't prove negative statements."
I know this is likely wasted on you, but here goes:
You can't prove the hypothesis: "There does not exist an unobservable teapot in the orbit of Jupiter."
For the SAME reason I can't prove the hypothesis: "LLMs can never eventually be used for generating good incident reports."
For the SAME reason I can't prove the hypothesis: "God doesn't exist."
For the SAME reason I can't prove the hypothesis: "A unicorn does not exist at the center of the Earth."
I fully admit this in the comment you've supposedly read.
This is why the burden of proof lies on the person making the positive claim (this is you).
1: https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)#P...