Hacker News Reader

Comment by slowmovintarget | Hacker News Reader

Weights are data, not a type of program.

A computer program is a set of instructions that may be executed. Weights are values that may be loaded by a program, but are not a program in and of themselves.

View on HN

It's a very difficult distinction to make.

Would you consider a Python program to be data rather than program just because it is text input to the python interpreter instead of machine code for the CPU?

It is not at all a difficult distinction.

Weights are literally numbers computed as output. They are not instructions. The semantics of those numbers even when emplaced (loaded) in an artificial neural net is such that they do not execute. They are not instructions. LLM engines and diffusers perform searches where the weights are used to calculate additional output.

Is source code, like Python text, data? Yes. All code is data. But not all data are source code.

If I gave you a web request log, you would not assert it is a program. If I gave you a CSV file with time-series values from a sensor, you would not assert it is a program. If I hand you a database of contact information, you would not assert it is a program. Weight files are the equivalent of CSV files. They are are a dump of parameter values computed from training.

They are not a program.

The definition of computer program is well worn. So is the definition of source code, and the definition of parameters. Weights are parameters.

The difference between code and data only exists in our minds. There is no distinction. Both code and data make the computer do things (and, yes, both code and data only make the computer do things if other conditions are permitting, for example if executed with the right interpreter, or loaded with the right type of viewer). Anything that can be expressed as code can be expressed as data, and vice versa.

If a program has to consist of instructions, then source code written in a declarative language is not a program.

> Weights are literally numbers computed as output. They are not instructions.

They are instructions if you consider the LLM system itself to be a kind of weird, indirect virtual machine. Each number can be mapped to a set of instructions that are executed. Even your CPU uses numbers (machine codes) to execute.

Join me in saying: ...code is data is code is data is code is data...

No, declarative programs exist. They are not instructions.

There is no real line between code and data. This is an observation that runs all the way from Turing Machines in computability theory to the Von Neumann architecture and homoiconicity in Lisp.

What we call 'data' is just code that needs a cleverer interpreter.

Less into theory and more into "wait wtf": Some of the older projects I've worked on were written by people who loved database-driven stuff, to the point they did things like put perl code into one table column (with sentinel values you had to find/replace before `eval`ing the code) and sql into another table that retrieved values for those find/replaces, both retrieved and executed by some really generic code.

Code or data: Well... both.

They’re not data though, they’re coefficients. They are the only thing that significantly differentiates one model from another.

If I told you the economy can be accurately modelled by

GDP(x) = Ax + B

But I don’t define A And B for you because it’s proprietary, you haven’t learned anything other than what you can glean from the structure of the model itself (it’s linear, there’s only a single input etc)

If most of these models are similarly structured, I’d say the weights are the program.

The nature of the data as proprietary or not, important or not, is not relevant.

Parameters, or actual arguments, are values; data. Not instructions.

Valuable data is still data. It's significance doesn't magically turn it into source code.

But not "raw" data. They are derived from other data and a program. If this was a collaboration where one collaborator did the processing and one sourced the data, they would likely both claim some amount of ownership of the trained weights.

At a minimum, it would be an active area of negotiation that the attorneys would take notice of. Source: have negotiated these agreements.

A curated data set is still a data set.

I imagine it is not settled law, but there's a clear argument to be made that regardless of the difficulty in curating the data set, it's still a data set.

Can it be licensed and sold. Yes, surely. Is it proper to pretend an open source license is sufficient protection, probably not.

When people talk about weights, they talk about a network of weights that takes an input and computes an output. There is really not much difference between a saved model and a program.

This is a distinction without a difference. Code is data and data is code.

Maybe they've only worked with machines using a Harvard Architecture

All source code is data. Not all data is source code. Data may be encoded, but that doesn't make it source code either.

Weights are data in the same way that instruction codes in memory is data.

Values for the variables do not the function make.

That's the least of it. In Lisp the distinction between code n data is blurred all the time. In F18 assembly i frequently have "double entendres" which are used as code or as literals depending on the entry point. I think at least once there was code and data in the same entry point. Assembly n Lisp are both homoiconic, after all. N verb at the end of the sentence, are you transliterating German, or a two-foot green Jedi master full of wisdom?

Reply on news.ycombinator.com