I guess? The fact that the CM team doesn't even generate 10.1 builds for the Passion is a pretty good indicator that it's too weak. Or HTC won't upgrade the graphics drivers.
On my HP Touchpad though, Jellybean runs like a charm. I think the real reason fragmentation was ever an issue is because so many devices shipped without enough space for apps.
It doesn't help of course that Google and their strategic app partners (see Facebook) assume that every device has a tonne of space (which modern devices do of course) and grow ridiculously big. Chrome and Currents are both unusably big on an older device.
Bit of a ramble, but in short you're getting a much much better experience by not upgrading, unless there's a way to hack that tiny storage partition?