The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format.
You need a whole metric fuckton of these.
After that, you train your model to produce those 3d scene files instead of image bitmaps.
You can do this for a lot of other tasks. These general purpose models can learn anything that you can usefully represent in data.
I can imagine AGI being, at least in part, a large set of these purpose trained models. Heck, maybe our brains work this way. When we learn to throw a ball, we train a model in a subset of our brain to do just this and then this model is called on by our general consciousness when needed.
Sorry, I'm just rambling here but its very exciting stuff.