Installing a codec pack was one of my first tasks when setting up a new Windows installation in that era. K-Lite was a popular choice.

Then WMP would play ogg, though you'd probably have switched to MPC and Winamp.