Funnily enough syntax errors is the one thing that you can completely eliminate in LLMs simply by masking the output symbol probability vector to just those which are valid upcoming symbols.
One minor footgun I've seen with that approach is that while the model is guaranteed to produce syntactically valid outputs, it can still "trap" the model into outputting something both wrong and low-probability if you design your schema badly in specific ways (contrived example: if you're doing sentiment analysis of reviews and you have the model pick from the enumeration ["very negative", "negative", "slightly negative", "neutral", "positive"], then the model might encounter a glowing review and write "very", intending to follow it up with " positive", but since " positive" isn't a valid continuation it ends up writing "very negative" instead).