What if you need to generate two server implementations (possibly in different languages) that adhere to the same specification?
You don’t always go server -> spec -> client.
You don’t always go server -> spec -> client.
Even so, a e2e test suite would surely serve far more utility over a spec that simply stubs out endpoints with no functionality.