1. Thumbs up for making loading times a priority!

2. Using Hugo instead of Octopress won't make your site any faster to users ¯\_(ツ)_/¯

3. You're almost completely forgetting about server-side performance. I'd recommend looking into GitHub Pages for a free and fast (CDN-backed) way to host a static site.

---

Unrelated, but:

4. I'm not a fan at all of the self-upvoting link at the bottom of your post. Not cool.