I highly highly recommend the book by the way, to anyone on HN. Goes all the way from mechanical components to resistors, nixie tubes, CMOS sensors, processors etc. Excellent photography that reveals the internals, along with operation explanations and history for each component.
When I'm connecting an USB cable horizontally I always think "hol up". So I always remember to plug it with the hole up.
When I'm connecting it vertically I think "hold right". So I know that if I want to hold it right, I have to put the hole right.
Has worked for me so far.
EDIT: Though "left" and "right" are more vague compared to "up" and "down", so connecting vertically assumes a specific "mental point of view". I'm just using one that's intuitive for me, but it might not work for everyone.