Out-of-band pairing is a better idea, e.g. some key on the bulb has to be entered or scanned or read on the device.
Zigbee etc. tries to get around the whole thing by requiring that the pairing process is done from a 5cm or something distance.
Imho that is good enough but I don't know how far you could extend it with directional antennas etc.
That anyone can reclaim a device they hold in their hand is a feature.