I've implemented Swagger with several APIs and agree that it's crazy complex and time-consuming to write Swagger files manually.

I believe the best use-case for Swagger is to develop the API (perhaps just defining the routes with payload and response, but without controllers), and then auto-generating the Swagger files. This way the API consumers always have an up-to-date documentation, and there is only one place which represents the current state of the API.