Hacker News Reader

Comment by spdustin | Hacker News Reader

I would've expected so. Rule-based systems in NLP can express parts of speech, coreferences, modifiers…it's almost too easy to do something like "Extract all subject-verb-object clauses where the object is connected to the subject by, say, a possessive pronoun.

View on HN

So you're thinking of expressions like "Winston said he had noticed a continued decrease in his suicidal ideation" (real example). I don't know if I'd agree that it's clear a rule-based system would be able to catch everything like this. Consider:

1a. (original) Winston said he had noticed a continued decrease in his suicidal ideation.

1b. (simplified) Winston noticed a decrease in his suicidal ideation.

2. Winston's suicidal ideation decreased.

3. There was a decrease in the suicidal ideation Winston felt.

4. Suicidal ideation decreased in the patient.

Many more syntactic repackagings of similar propositional content are possible, too. For perfect recall, rule-based systems need to grapple with this, and additionally, in a real setting you're probably getting predicted (rather than gold-standard) POS tags, relations, etc., adding additional noise.

My expectation (as someone who's mildly bearish on LLMs, btw) would be that if an LLM appears to be doing worse than a rule-based system then you probably haven't tried some low-hanging fruit yet such as adjusting prompting strategies, fine-tuning, using different pretraining data, etc.

They used Flan T5-XL, a 3B model from 2022. Very weird choice.

Your 1a and 1b say completely different things, and 2, 3, and 4 are even further away.

In 1a, someone says something. In 4, you seem to have an objective fact about a patient.

Reply on news.ycombinator.com