YOLO is a combination of backend encoder and head. Backend (Darknet) is unique to YOLO, that would be the main difference with SSD, if I'm not mistaken.