The Lotka-Volterra equations are a simple and popular mathematical model of this kind of dynamics:

https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...