Back in the early 90's I saw this interesting band called D'Cuckoo at Stanford that used a variety of midi triggers to redefine instruments relative to performed movement. https://en.wikipedia.org/wiki/D%27Cuckoo Now this can be done with AR.
Your implementation is very tight. I can see a lot of potential here. I'd throw money at in app purchases of other setups and even better would be the ability to load in my own samples and customize the associated movements, etc.
Is this a side project or are you planning on monetizing?
Anyone know of other AR music generation tools like this? I tried some searches, but didn't come up with anything.
During my research I found a lot of projects that connect body-movement and interactive music which could usually be put in one of these categories: 1. research projects that never left the university 2. art installations or performances - So both were tied to a specific time and place. My goal was to create something mobile, bringing this kind of experience into the home, or wherever.
Right now this is a non-commercial product, like a test balloon. I can imagine to collaborate with musicians and offer their creations as IAP in the future.
About bringing in your own samples etc.: I have started to work on an editor (native Mac app) with which people could create their own interactive compositions. They could then import them into the app and ideally even share them with others. I hope to have a first version out in a couple of months; luckily I got a tiny bit of funding for that project.
>I have started to work on an editor
Godspeed on the editor! I could see joint collabs with musicians and dancers creating an essentially net new art form.
There is a lot of potential here.
A few ideas at random:
- could be neat to have the ability to send OSC (i.e. send the same values that you're using to generate the music i.e. limb position / angle / velocity, etc.) so that computer-based synths, video softwares, Max MSP, etc. could ingest it in real time as well
- it would be fun to have different synth "presets" or "patches" which connect limb movement to different parts of the audio generator -- i.e. connect right arm with LFO rather than the oscillator pitch for instance
- just curious, does it support multiple dancers?
on OSC: I already have implemented this feature in my development version. It would just send the raw body tracking data (joint 3d location). I was not sure though if it would be used by anyone and where to put it, so I did not ship it.
About the presets and general interactive/creative features, please read my answer to wintermutestwin, I am working on it.
It does not support multiple dancers because the "Motion Capture" feature (which is proprietary) does not support it… :/