On what basis does he "officially own" it? Presumably there is a plat on file with your deed of title to your house and its lot that gives the boundaries of your lot, and another plat on file with your neighbor's deed of title to his house and its lot that shows the boundaries of his lot. Presumably both of those plats say the fence is on your property.
I know what adverse possession is. But there are specific conditions attached to it, which will depend on the particular statute in effect in the particular jurisdiction. For example, if the poster I responded to were to tell in court the exact story he told here--"I didn't realize the fence was on my property until I had a survey done, now I realize it and I'm bringing a challenge"--would the statute in effect in his jurisdiction bar him from seeking any relief?