I spent quite a lot of time at Dynamicland Oakland. It's great fun and the people are just wonderful. I think of Realtalk as adding another layer to programs, the Physical Layer. Program behavior can be determined and controlled by physical layout as well as code. It's not completely unlike a frontend/backend distinction, and in fact you could make a Realtalk simulator using a javascript canvas much more easily than building the camera/projector setup.

Just like building full stack software, there's a large amount of nonobvious skill in dictating your separations of concerns between the physical layer and software layer. Good programs are flexible, remixable, modular, intuitive, and let non-programmers make nontrivial interactions and enhancements via the physical layer. Bad programs require you to have the physical objects in a particular configuration, or break completely if one piece of paper is lost. I found these programming design questions a really interesting part of playing at Dynamicland.

A solid limitation of the system is that the pieces of programs aren't actually modular. You can't take a Cat from one program and a Dog from another program and have the dog interact with the cat. This is obvious in software - that's why we design APIs - but it's frustrating when all your programs exist in the same space (that's the whole selling point) and when bringing part of Program A into Program B is so intuitive and, when you have dozens of these programs lying around the room, inevitable.

I'd love to see them explore (wait for it...) using AI. Incorporating object recognition could remove the need for pasting dots onto every object by defining rules like "when you see a car, color it Red". It could allow for inter-program interoperability via the shared language of object recognition. And it could even determining logical interactions in a fun and surprising way: what _should_ happen when I take the cactus from this program and put it on top of the balloon from that program?