Knowledge Graph Reasoning Based on Attention GCN

Current performant (high accuracy) LLMs have a quadratic cost (space and time) in sequence length, they have a finite size typically much less than any KG of note, their connections are all fuzzy, they aren't especially amenable to small updates, training and fine-tuning don't work well with high-entropy data, and most computations physically cannot be done via any single pass through an LLM regardless of how it was trained.

Those constraints together create a landscape where if you have a big knowledge graph there will invariably be important questions the LLM cannot appropriately answer about it, no matter which strategy you use to try to ramrod the KG into an LLM architecture. If you don't train/fine-tune the LLM on the KG, it doesn't have enough context to answer your questions. If you do, your KG doesn't have enough data duplication to allow training to work well. If you manage to train it anyway, you can't ask compounded questions because of the max LLM circuit depth. If you try anyway and just run the results back into the LLM as input you have a compounding error effect because the whole thing is fuzzy. If you try to circumvent that with error-reduction techniques you tend to blow through the current context windows (quadratic costs) and still have unreliable results. And so on.

None of that is necessarily true forever, but suppose you have a problem where a KG is a natural fit but some of the data is a little fuzzy (you have pretty good graphs of how cities and roads and individuals and companies and whatnot are related, but it's not perfect, and some of it is textual or not otherwise appropriately structured). The KG can answer a number of queries very well, limited by the lack of structure in the node/edge representations. An LLM can't do much because it can't compress all those possible edges into its weights, because it can't fit the whole KG in a context window, it can't be appropriately fine-tuned to the data, and even if it could it couldn't recurse well without compounding error. If you instead use the LLM as a pre-processing step on the nodes or as a fuzzy neighbor search (restricted by the KG) or in some other way, you get a data structure that looks a lot more like a better prepared clean KG and can run traditional KG algorithms to ask questions like who might need your tax prep services or whatever. Getting an LLM to do that for the same cost will take a _ton_ of engineering beyond what I've seen poured into the space.