This is my workflow as well. Not ideal, but better than nothing.

Though I will say the biggest pain point with Wordpress is syncing and merging DB changes amongst team members, since so much of WP configuration is stored in the DB this becomes a real point of frustration especially on very large sites. I'd peg that problem as a much higher priority in terms of WP workflow than how to manage git inside WP admin.