That may be how some people use it, but that doesn't mean we should put up with that usage. The Nazis slaughtered millions of people because of their ethnicity/religion/sexual orientation/what have you. To use that word to describe an immature group of friends who say racist things because they think it's edgy waters down the meaning of a very powerful word in a way that severely harms public discourse.
If we're going to use the phrase "literally a neo-Nazi", I want to see swastikas or tattoos or something other than garden-variety hooliganism.