UIKit is noticeably better than AppKit but I doubt this is the reason why Messages is webview based. I don't know the exact reason behind it, but my guess is that they stopped caring about updating it when the people working on it either left or moved over to other projects.
I've taken that same guess after the new fancy effects and animations for iMessage got fully implemented in iOS 10 and barely supported at all in macOS Messages. It seemed like an afterthought on the Mac.