It involved a lot of web scraping Yahoo with Perl (specifically LWP) and then lots of analysis by humans with some help from automated tools. For example, if you plotted a histogram of each user's posts, you could clearly see when someone was at work (posted between 9am and 5pm with a drop off around noon) and at home (posts between 6pm and 2am with a peak around 10pm).
The analysts would often find a piece of information from a 2 year old post, e.g. 'Go Cubs!', and a one week old post "I just attended my 20 year UofI reunion" and quickly be able to narrow down on who the person might be. Coupled with Lexis Nexis (which was just coming online at the time), we routinely narrowed down individuals to just one possible person.
Given that this was done back in the early 2000's using ancient servers (by today's standards) and basic statistical analysis with a lot of legwork, I would be surprised if there weren't companies also trying to find the people on Reddit today.