Please bear in mind this is second hand information (I worked at several VFX places, and by the nature of the short term contracts people move around internationally a lot. I also worked at a software vendor (not autodesk, so no direct interest in replacing marionette))
There were/are two big things that are against you if you develop in house animation software(or any software for that matter):
1) The training gap. People are most efficient with the software they are used to. Maya has been the defacto standard in animation for pretty much 7 years (if not more).
2) the R&D cost. VFX is constantly moving. (Possibly much more than normal web software) With the increase in CPU/GPU power you are able to do much more. Take hair for example, In monsters inc, Sulley has something like a million hairs. (monsters U its 5million +) You need to control that hair system some how. There are people that groom digital hairs for a living. Its a constant battle between cheating, simulation and feedback.
The hair system alone could swallow a team of Five for a year, but then you'll be negleting the other systems that need bug fixes, improvements and R&D magic.
Having an outside company, who has the resources to have many teams work on each individual part, along with a UX designer to either make it work more fluidly(or just put annoying transitions in the way.) and most importantly of all, a team of QA, bugfixers and Tech support.
You're right, custom tools give you an edge, Take katana from sony: it allowed an artist to work on an entire city scape (Billions of objects) on a single computer, in 2006 (it was developed for one of the spiderman films) Something no other system could do.
Another example is of course renderman.
But these are only the successful ones. Many custom tools are built, atrophy and die. Sometimes taking the company with them.
But like the rest of IT the march of commoditisation is in VFX as well. Most software can be bought off the shelf (maya, modo, nuke, Premier, renderman, arnold) Its also almost possible to render in the cloud (although its crushingly expensive, and slower.)