Yes, the argument in the article seems to revolve around it is harder user to verify a build that it is to simply build it themselves. If they build it themselves there is no need to verify it of course - so what's the point of verification?
You could apply that same argument to lots of things. It's hard to verify things adhere to standards in general - whether it be flammability of children's clothes, or additives in food. Bu that logic I should be making my own clothes and meals rather than relying on the manufacturers claims.
Obviously, no one does. The point is someone has made a verifiable statement, in public. There is no need to go to the effort of testig that statement a zillion times. In fact needn't be done at all in many cases - just the treat of doing the test with something goes wrong suffices. In the case of food where I live, a recent recall of strawberry's devastated the industry. No one was testing if strawberry's contained needles until someone eat one, no one got seriously hurt when they did. Well no body except the strawberry industry which was on its knees for months until once again people started to take them at their word their product was safe to eat.
Similarly if you care about security (and I do), if there was a choice between several several Signal like programs that all claimed verifiable builds, and then it was shown one was in fact lying about verifiability I'd drop them like a hot potato. And I wouldn't be dropping them for just months like the strawberry's, it would likely be forever.
In one line: the idea that every user has to test a verifiable build does in fact verify in order for it to be useful, is seriously flawed.