Hacker News Reader

One of the great things about LLMs: since they aren’t deterministic prompt engineering will be even more susceptible to magical thinking than it has been with Google

View on HN

The big difference is feedback loop. With prompt engineering you can change your prompt slightly and immediately see if benchmarks improve. With SEO you had to wait at least days, and even then it was impossible to rule out external effects like others adding/removing links to you.

Anyone who had to deal with retraining models understands the pain of retraining. What prompts offer as a feature while at the surface may seem superficial, they are groundbreaking, as they completely remove re-training and the entire ML-Ops deployment lifecycle.

You can set temperature to 0 to have deterministic output per model.

Only if the model stays the same? As they’re constantly being tuned this is almost never true.

Can you give an example where it matters?

Yes, I’ve seen people at work calling it directly to generate code to talk to APIs with the temp set to zero. They still frequently receive different or broken code.

Shouldn't it be persisted/cached once working, possibly with some test/refine/retry?

Requerying everything from scratch when using seems like not optimal solution there?

Reply on news.ycombinator.com