Yes. 100% this. The API / object modeling is nearly the spec and the main thing to nail down. On past projects we've used Apiary as a pretty good option for getting a nailed down API Spec before any coding is done: https://apiary.io/. I think here more out of the box best practices around APIs will make these API specing tools even more effective.

I know what you mean about changing underlying data structures too. There are ways to fix the issues after the fact. E.g. wrapping existing concepts in other ones, and then gracefully deprecating the old ones. But, it definitely creates all sort of weird tensions with customers that are using the old concepts.