IANAL - Can he sue you? Anyone can sue anybody, for anything. Whether his claim is legit, or whether or not he would win, there's only one way to find out.
Scary lawyer letter is the first step any attorney would take. It's a shot across the bow. But, at this point, you don't know if that's all they got or not. The only thing worse than the feeling you get when you get a nasty-gram from a lawyer, is the feeling you get when you are named in an lawsuit.
You could do some game theory to figure out how much he might be willing to pay you to change the name, but that assumes he's a rational actor. Which could be a big assumption.