This is on gentoo, so it looks like I have X, acl, and gpm use flags set for vim, which are a weak correspondence to configure options. So, I'm assuming you can remove SM,ICE,Xt,X11...gpm,acl,xcb,Xau,Xdcmp from my list. Quite a reduction! I admit. To be fair, nano can remove magic, ncurses which cuts it down to libc, libz for nano, and libm, libattr, libc, and ncurses for vim. I'm assuming you might be able to remove ncurses for vim(?) by configure? I've never built vim by hand.
So you might be right, the stripped out version makes a difference.