Hacker News Reader

> Am I completely missing the idea? I don't think trust in LLMs is all that different from trust in code typically is. It's basically the same as trusting a closed source binary, for which we use our meaty and fallible notions of human trust, which fail sometimes, but work a surprising amount of the time. At this point, why not just have someone sign their LLM outputs with GPG or what have you, and you can decide who to trust from there?

This has been my problem with LLMs from day one. Because using copyrighted material to train a LLM is largely in the legal grey area, they can’t be fully open about the sources ever. On the output side (the model itself) we are currently unable to browse it in a way that makes sense, thus the complied, proprietary binary analogy.

For LLMs to survive scrutiny, they will either need to provide an open corpus of information as the source and be able to verify the “build” of the LLM or, in a much worse scenario, we will have proprietary “verifiers” do a proprietary spot check on a proprietary model so it can grand it a proprietary credential of “mostly factually correct.” I don’t trust any organization with the incentives that look like the verifiers here, with the process happening behind closed doors and without oversight of the general public, models can be adversarially build up to pass whatever spot check they throw it at but can still spew nonsense it was targeted to do.

> Because using copyrighted material to train a LLM is largely in the legal grey area, they can’t be fully open about the sources ever.

I don’t think that’s true, for example some open source LLMs have the training data publicly available, and hiding evidence of something you think could be illegal on purpose sounds too risky for most big companies to do (obviously that happens sometimes but I don’t think it would on that scale)

While there may be some, the most notable ones seem to hide behind the veil of “proprietary training data” but assuming the data is open, the method to generate the model must also be reproducible, thus the toolchain need to be open too. I don’t think there is a lot of incentive to do this.

But GPU-based training of models is inherently non-deterministic

In what way?

If you keep your ordering consistent, and seed any random numbers you need, what's left to be a problem?

"Inherently" might be too strong of a word, but the default implementations of a lot of key operations are nondeterministic on GPU. With the parallel nature of GPU compute, you can often do things faster if you're willing to be a bit loosey-goosey. PyTorch and TF will typically provide deterministic alternatives, but those come at a cost of efficiency, and might be impractical for LLM training runs that are already massively expensive.

https://pytorch.org/docs/stable/notes/randomness.html

I wonder what the actual speed difference is. I couldn't find any benchmarks.

The inherent faults or even just speed differences[] of hardware.

[] In the real world, a lot of resources are oversubscribed and not deterministic. Just think about how scheduling and power management work in a processor. Large model training happens across thousands to millions of processors (think about all the threads in a GPU * the number of GPUs needed, and add the power throttling that modern computing does to fit their power envelopes at all level... and power is just one dimension, memory and network bandwidth are others sources of randomness too).

Making such a training deterministic means going as slow as the slowest link in the chain, or having massive redundancies.

I suppose we might be able to solve this eventually, perhaps with innovations in the area of reversible computing (to cancel out undeterminism post-facto), but the current flavor of deep-learning training algorithms can't.

There's no reason that has to affect the determinism. When you're calculating thousands of nodes in a layer, each one is an independent series of multiplies and additions, and it doesn't matter what order the nodes get scheduled in. And each one will calculate in the order you coded it.

And if you want something finer, with smaller "slowest links", you can deterministically split each node into a couple dozen pieces that you then add together in a fixed order, and that would have negligible overhead.

I am talking about training models on thousands of machines, each with thousands of GPU streaming processors.

For data parallelism, if you want deterministic results, you need to merge weights (AllReduce, in the general case) in a deterministic way. So, either you need a way to wait until they all catch up to the same progress (go as slow as the weakest link), or fix differences due to data skew afterward. AFAIK, no one has developed reversible computation in DL in a way that allow fixing the data skew post-facto in the general case. (1)

For model parallelism, you are bound by the other graph nodes that computation depends on.

This problem can be seen in large-scale reinforcement learning or simulation, or other active learning scenarios, where exploring the unknown environment/data at different speeds can skew the learning. A simple example: imaging a VR world where the pace at which you can generate experiences depends on the amount of objects in the scene, and that there are parts of the world that are computationally expensive but provide few rewards to sustain explorations (deserts) before an agent can reach a reward-rich area; (without "countermeasures") it is less likely that agents will be able to reach the reward-rich area if there are other venues of exploration, even if the global optimum solution lies there.

(1) IMHO, finding a solution to this problem that doesn't depend on storing or recomputing gradients is equivalent to finding a training algorithm that can work in presence of skewed/unhomogeneous datasets for the forward-forward approach https://www.cs.toronto.edu/~hinton/FFA13.pdf that Geoffrey Hinton proposed.

> Hello. Fires are dangerous. Here is how fire burns down a school. Thankfully, we've invented a fire extinguisher.

Heh. Shakedowns are a legitimate way of doing business these days. Invent the threat, sell the solution.

Sidestory: I'm convinced the weird "audio glitch" that hit American Airlines in 2022-09 was the work of a cybersecurity firm trying to drum up business for themselves. Their CEO (hello, David) had just a few months earlier personally submitted to AA's CEO a vaguely-worded and entirely-unverifiable incident report suggesting American's inflight wifi provider's payment portal or something had been compromised by The Chinese-- and blamed an unnamed flight attendant for destroying all evidence by forcing him to immediately shut down his laptop.

So no evidence, no screenshots, no artifacts verifying he was even on that flight, implied involvement of foreign boogeymen, adverse action taken by malicious/anonymous witnesses, and when pressed for technical details, the reporter dodged questions and feigned ignorance (when asked for his MAC address, he returned one for a virtual adapter and stopped responding). A few months later, AA has a public PA system incident that perplexed everyone and gets attributed to vague "mechanical failure." Could be coincidence, but everything about the former incident screamed of a cybersecurity vendor chasing sales by sowing unverifiable FUD in bad faith. I don't put it past them to engage in "harmless" sabotage.

There is still a design decision to be made on whether we go for TPMs for integrity only, or go for more recent solutions like Confidential GPUs with H100s, that have both confidentiality and integrity. The trust chain is also different, that is why we are not committing yet.

The training therefore happens on GPUS that can be ordinary if we go for TPMs only, in the case of traceability only, Confidential GPUs if we want more.

We will make the whole code source open source, which will include the base image of software, and the code to create the proofs using the secure hardware keys to sign that the hash of a specific model comes from a specific training procedure.

Of course it is not a silver bullet. But just like signed and audited closed source, we can have parties / software assess the trustworthiness of a piece of code, and if it passes, sign that it answers some security requirements.

We intend to do the same thing. It is not up to us to do this check, but we will let the ecosystem do it.

Here we focus more on providing tools that actually link the weights to a specific training / audit. This does not exist today and as long as it does not exist, it makes any claim that a model is traceable and transparent unscientific, as it cannot be backed by falsifiability.

What's the point of any of this TPM stuff? Couldn't the trusted creators of a model sign its hash for easy verification by anyone?

I think the point is to get a signed attestation that an output came from a given model, not merely sign the model.

Why does this matter at all?

You go to a jewelry store to buy gold. The salesperson tells you that the piece you want is 18karat gold, and charges you accordingly.

How can you confirm the legitimacy of the 18k claim? Both 18k and 9k look just as shiny and golden to your untrained eye. You need a tool and the expertise to be able to tell, so you bring your jeweler friend along to vouch for it. No jeweler friend? Maybe the salesperson can convince you by showing you a certificate of authenticity from a source you recognize.

Now replace the gold with a LLM.

You go to school and learn US History. The teacher tells you a lot of facts and you memorize them accordingly.

How can you confirm the legitimacy of what you have been taught?

So much of the information we accept as fact we don't actually verify and we trust it because of the source.

In a way, students trust the aggregate of "authority checking" that the school and the professors go through in order to develop the curriculum. The school acts as the jeweller friend that vouches for the stories you're told. What happens when a school is known to tell tall tales? One might assume that the reputation of the school would take a hit. If you simply don't trust the school, then there's no reason to attend it.

A big part of this is what the possible negative outcomes of trusting a source of information are.

An LLM being used for sentencing in criminal cases could go sideways quickly. An LLM used to generate video subtitles if the subtitles aren't provided by someone else would have more limited negative impacts.

If my reading of it is correct this is similar to something like a trusted bootchain where every step is cryptographically verified against the chain and the components.

In plain english the final model you load and all the components used to generate that model can be cryptographically verified back to whomever trained it and if any part of that chain can't be verified alarm bells go off, things fail, etc.

Someone please correct me if my understanding is off.

Edit: typo

How does this differ from challenges around distributing executable binaries? Wouldn't a signed checksums of the weights suffice?

I think this is more a „how did the sausage get made“ situation, rather than an „is it the same sausage that left the factory“ one.

Sausage is a good analogy. It is both (at least with chains of trust) the manufacturer and the buyer that benefits but at different layers of abstraction.

Think of sausage(ML model), made up of constituent parts(weights, datasets, etc) put through various processes(training, tuning), end of the day, all you the consumer cares about is the product won't kill you at a bare minimum(it isn't giving you dodgy outputs). In the US there is the USDA(TPM) which quite literally stations someone(this software, assuming I am grokking it right) from the ranch to the sausage factory(parts and processes) at every step of the way to watch(hash) for any hijinks(someone poisons the well), or just genuine human error(gets trained due to a bug on old weights) in the stages and stops to correct the error and find the cause and allows you traceability.

The consumer enjoys the benefit of the process because they simply have to trust the USDA, the USDA can verify by having someone trusted checking at each stage of the process.

Ironically that system exists in the US because meatpacking plants did all manner of dodgy things like add adulterants so the US congress forced them to be inspected.

Except there’s a quantifiable difference between 18k and 9k gold.

Differences in interpretations of historical and cultural events are far more nuanced.

We’ll likely end up in a place with many trusted sources of attestation, each with their own bias toward particular notions of the truth.

Like schools and media outlets, there will be many LLMs to choose from that will tell you, confidently and authoritatively, what you want to hear.

Why should we trust your certificate more than it looking shiny? What exactly are you certifying and why should we believe you about it?

You shouldn't trust any old certificate more than it looking shiny. But if a third party that you recognise and trust happens to recognise the jewelry or the jeweler themselves, and goes so far as to issue a certificate attesting to that, that becomes another piece of evidence to consider in your decision to purchase.

Art and antiquities are the better analogy.

Anything without an iron-clad chain of provenance should be assumed to be stolen or forged.

Because the end product is unprovably authentic in all cases, unless a forger made a detectable error.

This seems like a classic example of "I have solved the problem by mapping it onto a domain that I do not understand"

Was thinking the same thing - not sure what this accomplishes that couldn't already be done with a GPG signature on a .safetensors file.

> It's also "open source". Which part of it? Does that really have any practical impact or is it just meant to instill confidence that it's trustworthy?

It means two things: 1) the founders are idealistic techies who like the idea of open source and want to make money off it, 2) they're trying to sell it to other idealistic techie founders B2B. You don't mention things in an elevator pitch unless someone's looking to buy it.