I ended up switching to Sketch and it's orders of magnitude again better for me. I primarily draw icons, logos and interface mockups. Sketch gives you lots of ways for quickly iterating over lots of mockups that I couldn't find in Affinity Designer so I'm much happier now.
Either way, they're both good for different use cases and both have free trials.
I'm considering Figma now because I don't like being tied to using a Mac and want something that might work on a Chromebook. You can run Inkscape on Chromebooks now but I'm pretty sure sure I can't go back to the interface. Are there any alternatives I haven't considered?