The message dialogue would have looked like Windows 95/98, with a reply option. BO was more extensible; ButtTrumpet would run and announce to the person who installed it that you had come online. Another BO plug-in allowed you to change how the message interface worked. My guess is someone saw your IP, and someone checked out your IP to find you open.
I’ll tell you, when I found someone who had open BO/NetBus, I would assign a password to their configuration so they weren’t open to anyone other than me. I have to imagine that hash was was easy to crack, but at the time, I thought it was a solid way to keep people from getting messed with by anyone other than me.