Compatibility: these addons could be broken very easily because they could depend on almost anything, and with the monthly release cycle, it is very difficult for mod authors to keep up. For instance, some addons would work by taking a core browser function written in JS, convert it to a string, run a regular expression to edit the string, then use eval to create a new function to replace the old one. In some release, the syntax of the "convert a function to a string" output changed slightly and it broke these addons, because it broke the regexp they were using.
Performance: XUL addons could do all sorts of things that are horrible for performance, and there was no real way for a user to tell what was causing it, because the addon wasn't isolated in any way. I ran into somebody who was having severe performance issues because the browser was generating colossal amounts of garbage for no reason. It eventually turned out that on a whim they'd installed a "LaTeX the World" addon, which would look for LaTeX typesetting instructions on pages and replace it with the nice looking output. The problem was, the way it worked was that every 10 seconds or so it would convert the entire contents of every single tab you had open into a zillion strings, search those strings, then throw them out.