Perhaps the Vision Engine is better for computer vision tasks, but having to use TIDL suite when compared to Jetson Nano's JetPack with tools which we use regularly on bigger GPUs is going be a hard compromise to make.
Jetpack includes CUDA 10, TensorRT, OpenCV 3.3.1 etc. by default and PyTorch is available separately for Jetson Nano. Besides the community is very active.