Hacker News Reader

If you are working with natural language, it is by definition 'fuzzy' unless you reduce it to simple templates. So to evaluate whether an output is a semantically e.g. a reasonable answer to an input where non-templated natural verbalization is needed, you need something that 'tests' the output, and that is not going to be purely 'logical'.

Will that test be perfect? No. But what is the alternative?

View on HN

Are you referring to the process of requirement engineering? Because although I agree its a fuzzy natural language interface, behind the interface should be (heavy should) a rigorously defined & designed system, where fuzzyness is eliminated. The LLMs need to work primarily with the rigorous definition, not the fuzzyness.

It depends on the use case. e.g. Music generation like Suno. How do you rigorously and logically check the output? Or an automated copy-writing service?

The tests should match the rigidity of the case. A mismatch in modality will lead to bad outcomes.

Aha! Like that. Yes that's interesting, the only other alternative would be manual classification of novel data, so extremely labour intensive. If an LLM is able to do the same classification automatically it opens up use cases that are otherwise indeed impossible.

Reply on news.ycombinator.com