If everybody knows it’s the deal, which is apparently the case in NY finance, it’s a non issue.

In fact not having a gardening leave might be more suspicious, as it means you’re so useless and out of the loop your previous employer doesn’t think there’s anything you could be carrying over.