I feel exactly the same way and have faced similar thoughts. If nothing happens I assume no offense was taken and move on I guess... but sometimes that bothers me more.