If you're providing binaries, you might try to cover the latest couple of versions of a distro, plus maybe a stable/long-term-support version for more conservative sysadmins; that's 3 versions.
Let's say you do this for x86-64, x86 and ARM; plus a slightly different ARM version to handle raspberrypi quirks.
You might do this for popular Linux distros, like Ubuntu, Debian, Fedora and Arch. Then there's FreeBSD, and you might want one of the other more-popular unices (e.g. OpenBSD, (Open)Solaris/Indiana/whatever.)
Assuming you've done all of that, you'll still get complaints from someone running AIX on a PowerPC.
On the other hand, if you provide the source, along with some common build system (e.g. autoconf for C, setuptools for Python, etc.) then it should work out of the box on most of those systems; plus, if there are any issues with a particular setup, you've given the user the resources they need to debug and patch it themselves, if they're that sort of person.
It also means I can disable functions or entire pieces of code that I don't need. Less enabled functions _often_ equate to less attack surface. This also results in smaller binaries and _sometimes_ smaller memory footprint. In a world of VM's and containers, optimization of memory and disk usage become more relevant.
Building from source is also useful if I happen to need the latest feature of a specific application that is not available in the shipped/supported binaries provided by an OS distribution. It also means I don't need to start adding software repositories from other 3rd parties and drag in additional library dependencies that can spiral out of control. I keep all my custom builds confined in /opt. The OS is kept pristine and easy to update.
For example, I use NixOS, and have had to use the "patchelf" utility to alter hard-coded paths in a few programs (either binary blobs, or bootstrapping systems bundled with source).
Sometime certain programs have a particular configuration that you might want, but most of the users won't. In those cases the developer can simply omit those when they build the distribution binaries and then you as a POWER USER make your own changes and configuration. It happened me multiple times (want some experimental features?).
Some time ago I used to rebuild my Linux kernel to fix problems in my distribution, or to gain some edge in performance, but as the development had advanced that edge won't make up for my effort so I don't do it anymore.
I'm a linux user, and most of the time I hate to compile software if I don't have to, so I don't but when I do I'm happy that I can.