Anecdotally, a friend has been using TX2s and got a Xavier to test out. He was blown away by the performance delta. It’s got an octocore ARM for a CPU, and while I don’t recall what the TX2 has... that’s a lot of CPU cores to work with.
I’ve got a Xavier sitting on my desk too, but haven’t played with it much. Running OpenCV on it and doing some light live video processing was really smooth.