Sounds like you might have a config issue. I have Lambda's that take 20ms and API Gateway returns the entire response in ~40-50ms.