I am one of those X11-only, no-toolkit Emacs users. The only times I need gtk support is when I run a variant build with gtk3 support in order to have a native Wayland window which doesn't require Xwayland. But I primarily use X11 so the no-toolkit, X11-only build.