Your idea has the drawback that browser updates might be few and far between (for some users), so when embedding them in the browser you can never rely on a certain version of a certain library to be available.

A more general caching system would be better. For example, "Cache by digital signature", as some people above mention.