Anything public you can share yet? I’ve been interested in how much LLMs “understand” symbolism.
This was paid work until recently but the start up pivoted to something else and now I'm wondering what to do with code that takes a few thousand dollars to run every time and is of questionable copyright ownership.
I am attempting to create parameterized "logic" problems (similar to the zebra puzzle) which cannot be solved by LLMs even when they are trained on it, or even when they "reason" on it.
Meanwhile this approach is even simpler, where it is demonstrated that LLMs cannot recognize 3 state DFAs. https://arxiv.org/pdf/2501.02825