This is kind of what tldraw are doing except they use 4V and instructions are png files. This could be a nice wysiwug for llamaindex or langchain agents in certain cases.
I really enjoy tinkering with LLM outputs that generate code that can be executed directly. Especially the faster models like GPT-3.5 Turbo are a joy to play with.
Working on something similar. Closed source at least for now. See bio for details!