{Hypothesis, test, loop} is the scientific method, and I can guarantee it is being used when fine tuning an LLM.
Predictability means that if your hypothesis is correct then you'd be able to formulate other improbable (and ideally currently untestable) predictions from it. And falsifiability means that if these predictions fail to occur then your initial hypothesis was also almost certainly wrong. So for instance Newton's hypothesis was that gravity was driven by a mathematical relationship between the mass/distance of two bodies. It was good science because it lead to the shocking ability to be able to dramatically simplify orbital dynamics, and create a complete predictive system of these bodies. It was even used to mathematically discover a completely unknown planet - Neptune.
Incidentally, his theory would also be able to be shown to be false if any of these unexpected predictions ended up being false. And that's actually exactly what happened. Observation of Mercury's orbit about the Sun showed it was off by ~1/3600th of one degree per century, relative to what was expected. And it's from there that people knew there was a mistake, which would only be explained later by Einstein hundreds of years later, who hypothesized a system with far more absurd predictions... and so the story continues.
And sometimes (often?) techniques do fail to transfer! IME, it's only the simplest ideas which do transfer to new contexts well - there's a lot of dark heat in pushing Imagenet another hundredth of a point, which you realize when you try to take incremental 'new SOTA' papers and apply them to audio problems.
But the things which /do/ transfer constitute real advances. And that's (drumroll) science! Sometimes things look good initially and get falsified. Medicine is still science, even if drugs get disqualified in phase iii trials...
What OpenAI is doing doesn't even resemble science at all.
Stop gate-keeping science. Observe, Predict, Test, Repeat. That is science. Kids do science when they watch adults do things. Just because you aren’t sitting in a lab publishing things no one reads doesn’t mean you can’t “do science.”
Words have meaning and when you dilute them down this far the only result is the destruction of communication.
den· tist ˈden-təst : one who is skilled in and licensed to practice the prevention, diagnosis, and treatment of diseases, injuries, and malformations of the teeth, jaws, and mouth and who makes and inserts false teeth
Brushing your teeth does not imply you are a dentist.
sci·ence noun 1. the systematic study of the structure and behavior of the physical and natural world through observation, experimentation, and the testing of theories against the evidence obtained. "the world of science and technology"
A child may not be a good scientist, but they observe, experiment and explore their ideas. That sounds like science to me. Observational learning is a very common way scientist begin to understand things.
However, to discover something new (not just new to you) does require contact with established knowledge. And if you don't publish, you can't be very sure that it really is "new"; and it does not become part of established knowledge - it has not been discovered.
i.e. the concept of discovering new knowledge means it cannot be done on a desert island.
Replication of the experiment, "verification" of the law, can still be within a largely preliminary side - of the steps that describe behavioural details, patterns over "protocols", without the explanatory side (that which cause understanding, e.g. enabling deduction).
Nobody has really explained why it works.
It's the same in ML. Good predictions do come from real ideas about how the systems work - ideas in information theory, entropy, cognitive science, statistical mechanics and so on - formulated in a context of our existing understanding and prior results. That's science.
What are you basing that on?
Most of AI academics have spent their career theorizing complex algorithms or complex explanations of intelligence.
But the engineers have built large enough Neural Networks to give us data points that show intelligence is emergent out of relatively simple components.
Unsurprisingly, the people who believed they were the smartest were the least likely to explore the possibility that human intelligence isn't general, but specialized.
Echos of the academics building heliocentric models of the universe centuries ago.
I believe it's entirely possible that the scientific method breaks down past a certain level of system complexity defined somehow by thermodynamics.
This would in part be due to the infeasibility of running the proper experiments to understand the effects of single variables when tens of thousands of variables might be changing at the micro level.
I don't think you would call it "science" for a bunch of single cellular organisms to cooperatively evolve a multicellular one. Similarly you wouldn't call it science when humans create digital lifeforms that require actual science to be done to understand how they work.
One thing is identifying (age of copper; age of bronze) the best ways of smelting ore to obtain the metal through trial and error, another is to try and understand the nature of materials.
That is to say, the two things you mention are the same process. "Identifying the best ways of smelting ore to obtain the metal through trial and error" is the easy part, when you get to pick low-hanging fruits in a field. But as the easy options get cleared out, continuing improvements requires increasingly complex, sophisticated methods - that's where the process transitions towards "trying and understanding the nature of materials". It's still the same process, but couple layers of abstraction up from the original "how to get better stick than neighbor and beat them instead of getting beaten".
It is not really (fully the same process), because you could (in theory) "early stop" when you have achieved technically sufficient competence - the description of the optimal process -, before the jump to the understanding. Something works - that is the technical side; why it works is the scientific side.
Note that the approach has controversial sides: if you took Newton's "hypoteses non fingo", He refused that jump explicitly:
> hypotheses, whether metaphysical or physical, or based on occult qualities, or mechanical, have no place in experimental philosophy
I.e.: Newton proposed an «experimental philosophy» which stopped short of a "true understanding of the reasons behind phenomena", in order to avoid contexts in which solidity (at his time) could not be expected. "Understanding" as "identifying universal laws" was the best that seemed affordable.
In the process of improving your object-level "how things work" goals, you end up generalizing and stacking increasingly complex theoretical models. Soon enough, you end up with people working high up the stack - not knowing or caring about the initial goals. Those people end up growing the "mound of knowledge" both upwards and sideways. The work is sort of self-justifying, but really, it's also self-similar. Where early materials science may have been driven by, say, desire for better/cheaper weapons, soon enough, you have people doing materials science because they desire to solve a puzzle. Whether the practitioners are smelting different combinations of ores to find one that will win them the war, or they're mixing up different kinds of equations to figure out a clean solution to a theoretical conundrum - it's the same process, same motivation. And it always involves play.
The kind of methodical, boring approach, with hypotheses and control groups and peer-reviewed papers? That's the boring part you have to do after play.
See also (with no implied judgement in this context): software developers that lose sight of (or care little about) business goals, and instead aim for theoretical markers of what "good code" is, and/or solve abstract puzzles of algorithms and architecture. Or the MBAs that view companies as abstract money-printing processes, running them by the rulebook that's entirely independent of whatever it is the business is actually doing or selling. Both are cases of growing complexity creating a new field of work that's independent of what brought it into existence.
It is also a specific purpose, a deliberate endeavour. We are in front of a world, we have to digest its phenomena, we try to understand it - it is a primal process, i.e. mental digestion. In some ways it could be argued that the "scientific drive" is anterior to the technical one: first you assess, then you act. Consistently, science started as "natural philosophy".
It is relevant, in the framework of "scientific drive as (development from the natural process of) mental digestion", that terminologically "science" is not just "the realm where false statements start to exist (and are tried to be avoided)" ("discriminate" - ex scindere), but also, before that, an activity of sorting, of "separating" conceptual elements - mental digestion. The "scient" (as a participle of scire, used as "to know") is a "distinguishing" mind.
We not only «desire to solve a puzzle» - it is our nature (as per the above) as minds dealing with a phenomenical world we need to grasp, as part of the activity of "seeing" it, already before action.
«Play[ing]» is a process that both allows to refine knowledge of the world, physically (in the physical play), and to refine the grasping of the world, mentally (in the mental play).
> growing complexity
Or just abstraction, enabled by a refined framework (following which, other areas become primary, topical - in the synthetic effort which is part of the above said process).
If you happen to conjure thoughts and let them roam wildly, well don't. Make an effort. Critical thinking as a conscious intentional effort must have plenty of literature. The "feedforward" output, the hallucination, is not a dish for service.
Except we consciously (intentionally) work on those matchings.
> are
Opinions should be offered in proper places. Polling is not debating.
The attempt to formulate laws (the "better faring hypoteses") can bring to "universal laws" or to more humble "patterns". It will contribute to the scientific effort - it does not exhaust it.