Is there a score for internal consistency? I dunno, maybe have another LLM extract structure into some kind of a logic language and score how many assertions are incompatible? Or to a probabilistic language and do some Bayesian analysis?
Absolutely no awareness about anything spatial - what is in which place, who is where, body positions.
Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that.
(edit) just asked GPT-4o to make a photo of hands tying two ropes (my favorite way to see how bad actually gen-ai is) - yup, it has an odd number of ropes coming out of the knot. That is state of art supposedly.
These are some things I'd like to at least attempt to quantify - LLMs are better at translation than thinking[0], so the idea was to translate the slop into something that can be then analyzed by a non-LLM tool. E.g. translate the story into a prolog fact database and generate a couple queries for each paragraph/chapter/combination of these. Rough idea, just something I haven't seen done.
[0] don't @ me
I could see this working as comedy, though it'd be better on film than in print. Either way you'd have to establish a very specific tone in the surrounding material.
do you think this tech deals with invariants in software algorithms any better?
I mean, haha but not funny.
Of course mitigation strategies were using many different LLMs and comparing results (voting) or using a highly trained/specialised model for only entity / context extraction. An interesting benchmark would be when those extraction techniques/models would exceed what a human professional is able to do.
I guess I'm fine with a tall error bar - better than nothing.
> An interesting benchmark would be when those extraction techniques/models would exceed what a human professional is able to do.
Slashdot-style +1 Funny, -1 Inconsistent for RLHF maybe...?
This is done per chapter, and the score trendline is what you see in the "degradation" column.
I would say this could be a reasonable proxy for internal consistency since it is measuring more or less the same ability, i.e. how well it's keeping track of details as context window increases.