Hacker News Reader

Comment by mangecoeur | Hacker News Reader

Should we really call it "engineering" if its a case of "try random things until one of them works without really knowing why"?

View on HN

Looks more like science here. Run a bunch of experiments and see which does better.

But literally the first sentence of the readme is "Prompt engineering is kind of like alchemy."

alchemy is kind of like science before a lot of principles were formalized.

This comes to mind: http://lambda-the-ultimate.org/node/5335

> Run a bunch of experiments and see which does better.

That's empiricism. Scientific method implies formulating falsifiable theoretical claims, producing meaningful analysis of experimental conditions, publishing peer-reviewed and reproducible results.

That's if you're trying to become a scientist. You can do science even if your experiment doesn't contain any of those elements besides trying multiple things and observing the outcomes.

Sure, the same way I can practice medicine by putting leeches on people. I'm not trying to become a physician so it's alright to call that medicine.

Well, not you, but other people could because leeches are still used because they work.

You're trying to correct people on a subject you know nothing about.

You could take some classes and learn how and when to apply leeches and become a physician call it medicine too.

https://www.nationalgeographic.com/magazine/article/leeches-...

Yes, because practicing Medicine is about a whole apparatus of agreed upon practices, not pick and choosing and slapping a label on it. Much like the case with doing Science, which is what I was getting at.

At this point you’re just deliberately making the definition of “science” ambiguous to conveniently support your point.

Is calling someone (or a fictional character) a "mad scientist" incorrect just because they don't follow every step of the scientific method?

Does the scientific method include peer review? I thought that was a recent phenomenon.

Peer review has been a thing since, albeit in a much less rigorous way, at least the XVII century. The so-called 'republic of letters' was a world (europe) wide society of philosophers, mathematicians, and early experimenters sharing results and data in writing. Mathematicians have been publishing proofs and challenging each other in Europe since at least the Italian Risorgimento...

Does the fact that it’s old mean it’s part of the scientific method?

That fact that it's part of the scientific method makes it part of the scientific method. I was addressing the "is it recent" bit.

This is more engineering than science. For a given task, try a bunch of approaches and keep the best one.

Science would be replicable, ie demonstrate that this particular approach to prompt writing yields better prompts than some baseline prompt writing approach across an array of different problems.

The difference is that in science the dialectic is between you the observer or a group of observers and a "reality" that hopefully(!) remains relatively stable, because the human organism is the way that it is - just as physical reality.

But these models are changing literally every day, so there's no fixed thing to reveal.

So no one will be able to reproduce anything at all.

This makes all this "engineering" pretty ridiculous in my eyes, it's literally for one models bizarre emergent properties.

That's called machine learning.

Gradient descent type optimization is far from "trying random things until one of them works without really knowing why". You can calculate all partial derivatives and understand the impact.

Sure you can, but you can also throw things in randomly and see what works then build theory around it.

I think it is a freaking miracle it even works. I understand how it works for say 100 parameter linear regression, but that it would work for billions of parameters (billion dimensional space) by nudging the parameters each a little (based on purely it's impact on the loss, assuming everything stays the same), is not obvious to me. It is a kind of magic.

Regarding randomness, the initialization of the weights is random, and if they use dropouts that is random too, plus the order in which to process the text might be random.

What’s even crazier is the fact that the underlying numerical math solution was exposed early 1800’s by Gauss!

To be honest, there’s always been a tension in machine learning between the: “we won’t do it unless the theory is complete and sound” versus the “we don’t understand this but it works much better consistently so we do it” crowd.

In the 90s and even 00s, the theory first crowd was mainstream and the empirical first crowd was considered fringe. Very fringe.

Personally I appreciated it when LeCun was like: “you can’t find the solution if you only search where the lamplight is shining.” Or other early deep learning practitioners note that ML theory is usually so far disconnected from practice in terms of tightness and bounds that you might as well ignore pure theory completely.

Anyway, it wasn’t until deep learning methods really smashed benchmarks across the board did people give in to the black magic / alchemy driven approaches of empiricism based upon intuition and bias developed through long-held experience.

To a layperson like me, the "try random things" part of machine learning doesn't seem to lie in optimizing the parameters, but in designing the model.

"I don't play craps. I'm a dice engineer!" - prompt engineers, satirically

I think the real etymology of it was "social engineering". Which also means clever hacks, not a real engineering, it deliberately subverted the meaning.

Hasn't software "engineering" been like that forever?

Reply on news.ycombinator.com