I've been road riding with people who switched to gravel bikes and they were suddenly noticeably slower. They also seemed to run out of gas earlier which means they were expending more energy to keep up.
My own road bike is nothing particularly fancy, but it's very light. Just because those bikes may be made to take more of a beating than a road-bike, they may be heavier. I'm thinking in particular tougher wheels and front work, possibly the whole frame.