Have you read about ChucK? It's a real time language for synthesis, composition, and audio programming.
The name of the program comes from their 'chuck' operator, which works similarly to what you've described.
If you look at the wikipedia page, it looks exactly like what you are proposing - I was looking to use it myself for a different project involving music analysis.