What happens when you analyze your favorite college football team like the CIA?
cultivatelabs.com
cultivatelabs.com
Another team's key player couldn't be replaced on short notice, were they to be removed from the board, in a deniable 'gang-related' incident in front of the nightclub they frequent.
The roster of a favored team could be boosted with secret financial incentives to candidates, funded off-the-books by new gambling and prostitution operations.
Nah.
These people don't like to share. They have to pay and/or payoff players, coaches, staff and so on.
But after that?
Yeah.. they're taking every nickel they can right down to the bottom line.
They're not letting anybody else in on that racket.
The Henkel-Sika-3M lab for sustainable adhesives (I'm just making that up, one doesn't exist) would have a different name and a whole lot less funding if it didn't produce research indicating that whatever it's benefactors want to do is in fact sustainable.
You're right that a human analyst - especially a very experienced one - is going to be able to still do a better job of analysis most of the time, but the AI is useful for having loops to re-examine the question, consume and curate new information, etc. We've built the system with that assumption: let humans do what they're best at, let AI do what it's best at...
But, it can simulate the output of a methodology, which is not the same thing.
The industry believes very strongly that this simulation of methodology is basically as good as the real thing (or at least "good enough" in most cases).
This belief is the fault line between the bullish and the bearish; it requires a leap of faith that not everyone is willing or capable of taking.
Ugly girlfriend means no confidence.
Unfortunately nobody will ever make a serious workplace comedy that ridicules federal intelligence because hollywood and the feds are buddy buddy.
It didn’t “calculate probabilities”, it just hallucinated slop. The same kind of slop almost started WW3 when a similar “intelligence analyst” system told the US navy to attack China because they were smuggling nukes into Iran. They weren’t, it was all slop. But their system was full of “calculated probabilities” too!
It doesn’t care whether your system is made of LLMs, decision trees, or bananas.
However, the problem is that unlike something like a neural net, you can’t exactly use backprop to improve.
On the flip side if the quality of the prediction doesn’t matter, I might as well have Claude spin up something shiny that does the same thing faster, cheaper, and with 10x the magic sparkles.
The entire human scientific community has been shown recently on average to be generating "slop" and calling it science.
My bottom line: humans generate a lot of BS--and LLM's are just averaging it and giving it back to us.
I think that was the entire premise of the CIA analyst framing - you don't know which one of several outcomes is going to prevail but you have a model which constantly updates its priors to increase the probability of one outcome over others.
In this case, you still don't know whether some guy's passing percentage stabilizing over a baseline will help you win the big prize or not but it does tell you that he may win some sort of recognition for being a standout player or that the guy who scouted him deserves credit.