The fundamental issue is that it binds the server to the client, who must understand HTML structure. It also makes it more difficult to reuse the endpoints in different ways or for new applications.
Returning data and letting the client render it decreases coupling and increases flexibility/testability- you can run unit tests on the client for mock data, and run unit tests on the server to test the desire
whatever frontend framework you're using has really nothing to do with it at all.
I daydream about making a framework that lets me use django templates as a SPA that does server side rendering on the first request, or when javascript is unavailable.
This is a drawback unless you truly need an API. YAGNI.
I've found a hybrid approach works well. Your site is mostly server side, but some individual pages have Vue and an API endpoint or two. If that mobile app or that third-party integration becomes a need, add endpoints for that particular use case.
I've gone the other route and prematurely optimized for the "what ifs" first, and it's painful when they don't happen and you're left holding all that complexity.