I don't have any professional experience diagnosing problems like this, but I've managed more than a few large-ish home networks (many roommates, large family w/ a ton of devices, etc.) in sprawling houses with thick foundations all the way to signal-dense apartment buildings where 2.4GHz is all but useless.
I personally cannot believe how terrible consumer-grade WiFi has stayed. You'd think that some of these manufacturers would be upping their QC at some point: I've experienced all the random undiagnosable problems you've described where nothing short of rebooting the router fixes things, and only until things break again later. I currently own one of those fancy Netgear Nighthawk routers and after months of going through PCI-Express cards on my desktop machine where random packet loss made gaming impossible (but not on any other devices even in the same room), I finally gave up and bought a 100ft cable off of Monoprice and ran it over some doorframes with tape.
I see a lot of buzz around all this "pro-sumer" mesh network gear[1], but I haven't had the kind of sprawling living situation that justifies it yet. (Though I did just recommend that article to a coworker on a very large property, and he's hoping it fixes his many, many problems.)
> it keeps getting harder and harder to diagnose, since nobody believes in ugly things like log files or serviceable hardware anymore.
Hear hear. To OP's original point, I can't believe how much stuff simply doesn't work like it's supposed to. I always thought it was my high-tech demands for plugging together services/software that weren't necessarily built for each other (e.g. Google Music on a Roku, or Snapchat on an Android phone), but after going home last Thanksgiving, it's unbelievable to me how many mainline iOS scenarios are completely broken. Every conversation I had with relatives was "oh yeah, that's easy to fix...lemme see...wait, what?...that doesn't make any sense...google-fu...huh, I guess there's some random problems with this...let's try deleting/re-adding your account...okay, that didn't do anything...wtf..." Throw in some "you got your iCloud password wrong twice and now we're locked out, time to go through insane account recovery", and I was ready to move to the woods and give up technology forever.
> moving full-time to software development where I can at least close a bug report once in a while.
Hahaha, grass is greener, man. Be prepared to be not fixing A LOT more bugs. ;)
[1] http://thewirecutter.com/reviews/best-wi-fi-mesh-networking-...