I think interpretable is a overloaded term. If you use RAG, sure you can look at the retrieved text and understand what is being used, but at that point you're just interpreting the retriever.

LLMs need to be interpreted so that they can be edited have their biases understood in a systematic way. However just as people aren't "interpretable" these algorithms are not going to be able to display their inner workings with 100% confidence. It's going to remain probabilistic, which might be fine for the majority of use cases. I think we're coming from an age where everything was 100% interpretable because we knew what was going on inside the machine (e.g. in a knowledge graph).

There needs to be some definition of what we want to achieve with interpretability for us to understand what standards we need to keep.