I think idioms and cultural references are fine - the rest of the world has worked out what US baseball and football cliches likely mean, people can decode most references with context.

But there are some interesting issues with UK <> US english, things like 'quite' which works in different ways in each locale. I was also very surprised to discover the difference in what we consider a frown - which makes a lot more sense of the US 'turn that frown upside down'. Interestingly my uncle who'd lived in the US ~20 years had never uncovered that difference till I asked him about it.

So it's good to know differences - especially when you want communication to be clear.