Havent tried Xavier yet, but I am using TX2 in my work (which is previous generation of this type of device) and CPU is very weak to allow any serious gaming.
I’ve got a Xavier sitting on my desk too, but haven’t played with it much. Running OpenCV on it and doing some light live video processing was really smooth.
The 8 core Carmel CPU in Xavier is not like the SPEs in Cell.
"Weak CPU cores" does not mean anything, you have to couple it with a metric or comparison -- the Switch has a locked, 1Ghz quad core A57+A53, and it does just fine for a huge amount of games...