This happened in the blink of an eye: in the area of modern Germany, anatomically modern human hunter gatherers have showed up something like 45-50 000 years ago, but after early European farmers have arrived from the southeast 7500 years ago, they entirely displaced the hunter gatherers in just 2000 years, despite the former having 42 000 years of head start (which is 7 times longer than the entire recorder history).
The reason is simple: using agriculture, you can have much more humans occupy the same area. You quickly get population counts that are completely impossible to sustain by hunting. If you nevertheless allow them to hunt, they will very quickly hunt the animal population to extinction, especially when most of the land is already used for cultivation instead of being dedicated for wildlife. We know this, because this happened on multiple occasions. Human arrivals to Australia have hunted most of its megafauna species to extinction. Same happened when humans arrived to Americas.
Point is, if you allow everyone to hunt, and hunters are numerous and adept, soon nobody is able to hunt, because there is nothing left to hunt. This is also happening today, with overfishing.