Out of curiosity, are you getting slow response times only on the first request, or on all?

I'm also using API Gateway + Lambda + DynamoDB

All of them are very slow on first request, even the DynamoDB queries. After that, they're much quicker, unless there are maybe 5 mins of inactivity.