With today's LLMs I could see them used as a way to speed up the generation of transformer code, but I'm not sure you could trust them to 'just work' as the layer.