I don't actually use the AWS HTTP API gateway. I control the front end entirely so my UI just talks direct to lambda. It's very simple - instead of futzing around implementing REST API mapping layers I just connect all my front end functions to a single lambda function and specify in the params the name of the python function I want to run. Super easy.
Even if you wanted to use API gateway you could still map all the end points to one single application and dispatch based on the route/URL.
I can make a solid recommendation to anyone use lambda .... seriously consider using Cognito for your user management.
There's no reason why you couldn't have tens of thousands of functions in the same function, in fact probably a good idea.