If you are interested in this field (exploring the limits of neural models using formal language theory), I help run a weekly seminar on it, Formal Languages and Neural Networks:
We have had many great speakers (most of them are recorded and available on YouTube) and have a welcoming Discord.