You of course don't have to throw everything into directives but then you risk making your code less portable and your tests more brittle. Wrapping a jQuery plugin in a directive is so trivial it's not even worth mentioning, and it certainly doesn't result in "enormous amounts of glue code to please AngularJS".
> over-complicated testing procedures
And this is where you showed your hand. You can't have written tests for an angular app because if you had you wouldn't have made such a bizarre statement. Angular's dependency injection combined with the fantastic Karma test runner makes testing an absolute breeze.