I see your point. Switching to an app that isn't in the four most-recently-used apps of the first task switcher screen would likely require a similar effort as using the multiple pages of the home screen.
However, if a user is switching between three or four apps for a given period of time, those apps may very well lie on very different pages of the Springboard. The most frequently used apps are not always the apps that matter now.
Even when you would need to swipe in the task switcher, surely it is easier to find an app among 4 instead of 16?
(Maybe the task switcher really is only useful for a small subset of use cases, in which case your position would seem to be the logical one.)