An LLM is not a universal Turing machine, though. It's a specific family of algorithms.
You can't build an LLM that will factorize arbitrarily large numbers, even in infinite time. But a Turing machine can.
You can't build an LLM that will factorize arbitrarily large numbers, even in infinite time. But a Turing machine can.
So, yes, you can.
Once you have a (2,3) Turing machine, you can from that build a model that models any larger Turing machine - it's just a question of allowing it enough computation and enough layers.
It is not guaranteed that any specific architecture can do it efficiently, but that is entirely besides the point.
They can do lookup in a table with 100% reliability, yes, because you can make then 100% deterministic if you wish by using numerically stable inferencing code and setting temperature to 0.
Finite context is irrelevant, because the context can be used as an IO channel.
A Turing machine does not have infinite state within the mechanism itself - it requires access to a potentially infinite tape. A Turing machine can be constructed with down to 1 bit of state (a (2,3) or (3,2) Turing machine are the smalles possible, where one number represents the number of states, and the other represents number of discrete symbols it can handle).
An IO channel is computationally equivalent to an infite tape, and unlike an infinite tape, an IO channel is physically possible.