One thing that I think could be improved is the layout of your code. Right now you have a single controller that handles everything. For example, selecting a shirt color should be a directive, and so should the quantity picker. Then use the $scope.$emit and $scope.$on to hook events together. This way, if I make a better color picker (although yours is pretty stylish), I can swap out yours for mine so long as I emit the same events as yours.
This would also make it easier to add new directives - for example, maybe I could choose a decal on the front and back of shirts, or jersey numbers or something. So I think that the controllers in this case should be pretty minimal, and just provide a scope to tie the various directives together.
Anyway, just a thought. Thanks for sharing this and the included GitHub repo.