I also find myself wondering about this from time to time. For the last year I have written all of my posts in markdown first, and then converted them to html before publishing. So a markdown oriented workflow is really appealing to me. I don't allow comments so that isn't a concern. My site really could just be static at this point.
Some things that hold me back:
- The migration process. I don't really want to invest a lot of time in migrating everything to a new tool.
- Automatic generation of things that are important to me like sitemaps. I would say RSS but I don't think anyone actually subscribes to my RSS anyway.
- Media management. A lot of my posts include pictures. Something I really appreciate about WordPress is how easy it is to do things like adjust the size of an image for a given post as well as adjust how the text flows around the image. I know you can do all of this with CSS, but then I would have to memorize all of that. I nice compromise would be something like a modern day Front Page (in the sense of being desktop first) with in editor tools for things like linking to other pages, posts, and tags as well as adjusting image position and size.
- Analytics. I guess it isn't terribly important. I don't get a ton of traffic, but knowing how many page views I get is something that keeps me going regardless. I do hate how absurdly bloated jetpack is though.
What I really want:
- Markdown first authoring with some WYSIYG features for managing media, page links, tags, categories, and permalinks.
- Desktop first editor that uses git for storage and tracking changes.
- Automatic generation of sitemaps and rss.
- Some way to track page views.
- Easy migration from WordPress. Ideally it will be something like "install this plugin and download the export of your blog".
- Configurable theming. It'd be nice to have a way to put together different wrappers for your site with the CSS lib of your choosing.
- Export everything as a nice static bundle that is easy to deploy from the desktop tool.
There is probably more that I can't think of right now. I've often thought of building something myself, but that would take away from my writing time. In the end I always end up sticking with WordPress because it mostly does everything I want it to do.