Hacker News Reader

Comment by spacebanana7 | Hacker News Reader

I’ve found LLMs extremely useful for Bible study.

If I’m struggling with a verse they can quickly show alternative translations and compare the interpretations of historical commentators.

Of course all this was possible with Google, but it could take an unreasonable amount of effort.

View on HN

Doesn't the section where the author goes through the LLM completely fabricating or confounding different parts of scripture give you pause about the truthfulness of the responses?

How do we know the truthfulness of the original text that was written hundreds of years after the words were originally spoken?

It's asked about what a specific version of a text says and gets that wrong. It doesn't particularly matter how correct or carefully copied that scrap is, the machine can't faithfully reproduce or quote a particular version. In fact it made up a quote that doesn't exist in any version of the Bible the author could find at all.

a machine could, this machine cannot

I was more trying to add an interesting philosophical perspective than to comment on this particular instance

As we build and understand them now there's pretty good structural reasons to believe that LLMs cannot be tweaked or tuned to be more than incidentally truthful.

If you use an LLM only solution, sure, but there are many more opportunities if you expand the system to include more.

We could, for example, use the existing tools for copyright detection, on the output and refuse to send it to the client side. It's just another moderation check in that perspective

Copyright protection software is notably bad because it requires human context and decisions to decide infringement from fair use. Say we ignore that part that's the lowest hanging fruit on the improving generative models tree. Deciding if something is true to have the bot not say output it if it isn't is waaaay harder and the thing needed to fix the problem we've actually been talking about.

If you are asking that question seriously check out the recent Rogan podcast with Wesley Huff. Has the usual Rogan conspiracy theory pontificating, but also some good discussion about the history of how we got the Bible we have today.

First off, the claim that the original text was first written 100s of years after the words were spoken is completely false and isn’t taken seriously by anyone in the field.

I saw this post a while ago: https://benkaiser.dev/can-llms-accurately-recall-the-bible/

While LLMs are in generally fairly good at recalling bible verses, they can't do it perfectly. If the Bible truly is the infallible word of God that we believe, then shouldn't we use more caution than just "welp, sometimes it makes mistakes"?

You could counter this by saying a person can't remember Bible verses, and this is true, but a person usually recognizes when they can't remember something instead of making something up. If you asked me to recall any random Bible verses, chances are I wouldn't be able to do so. However, unlike an LLM, I would admit I don't know for sure and I would pull out a Bible or look online for an authoritative source rather than adlibbing something on the spot.

I’ve found verse recall to be a problem, but this can usually by mitigated by adding the verse to the prompt or pushing the LLM to search the web.

The main use case for me is interpretation, which LLMs are excellent at.

There is good venerable software that does exactly this. A lot of options including several open source ones.

Reply on news.ycombinator.com