jonathan-kosgei

Should fasthttp be used in production? It seems to get a lot of flak for not fully implementing HTTP.

https://www.reddit.com/r/golang/comments/5w3ang/switching_fr...

Someone shared this with me https://blog.golang.org/share-memory-by-communicating - Share memory by communicating.

And also pointed me to nats.io, a messaging system that handles 10M messages per second on a $50 server.

See the comment at: https://www.indiehackers.com/forum/how-we-handle-25m-api-cal...

Here's the article mentioned; https://www.rdegges.com/2018/to-30-billion-and-beyond/

Thanks for linking to that. Let me include it in the question.

I think I remember coming across this. Thanks for sharing!

Awesome! I've always thought Django to be terribly bloated. Go seems like the best option for any kind of API.

Totally agree, but I'm looking for more case studies like the one I shared.

Very impressive, do you have a write up on this?

Same thing I'm looking for :)

Not particularly no.

Waf would've required setting up cloudfront in front of our endpoints which would've increased latency.

Basic Shield is supposed to be on for all users but I don't think APIG is covered in the range of services covered by their Advanced Protection plan.

At some point the fixed cost of real servers does become appealing.

We were wary of limiting paid users. Even with lambda's max concurrent function executions limit, when the function completes in a few milliseconds, the number of invocations per second can still be high.

At ipdata.co we use the same APIG+Lambda setup replicated in 11 regions to have the lowest latencies globally. We had to do some extra work to get API keys and rate limiting working but it was worth it. Our setup averages ~44ms response times - https://status.ipdata.co/.

We wrote about our setup in detail on the Highscalability blog [1]

A few things have changed since we wrote that article;

- We implement custom authorizers, which have helped lower our costs and the auth caching means authentication only happens once per x minutes and all subsequent requests are much faster.

- We use redis and a couple of Kinesis consumers running on a real server to sync data across all our regions. This setup has been battle tested and has successfully processed more than a hundred million API calls in a single day in near real time. [Use pipes and mget in redis for speed]

Here are some answers to a few specific things you raise in your question;

1. Use Sentry for lambda for error handling. The logs you get are incredibly detailed and have single handedly given us the greatest visibility into our application, moreso than any other tool we've tried (like AWS Xray).

2. Cloudwatch logs are tough. You might want to consider piping your logs to an Elasticsearch cluster, that might be a bit costly if you use AWS's Elasticache.

3. We use terraform for deploying our lambda functions and other resources. I'd strongly recommend it.

[1] https://highscalability.com/blog/2018/4/2/how-ipdata-serves-...

We actually found this out the hard way. We got DDoS'd and cloudwatch was our second most expensive item after API Gateway.

Lambda and Dynamodb were almost insignificant.

We also found that using lambda to process kinesis logs was incredibly slow and definitely not the best use case for lambda.

We had much better results running a server with a couple of kinesis consumers and redis.

This setup was able to process hundreds of millions of records in near real time.

As of May 14, lambda@edge supports node8

https://aws.amazon.com/about-aws/whats-new/2018/05/lambda-at...

This is brilliant! I hadn't yet found any reason to mess with Raspberry Pis. But the way you've described it makes a lot of sense. They'd definitely be a maintenance overhead, uptime wouldn't be guaranteed but I can imagine the sense of self-sufficiency you'd get would be very satisfying.

There's also https://github.com/rpgreen/apilogs but I haven't yet gotten it to work.

Cloudwatch feels like it was not made for humans. Searching and filtering for specific log events is a huge pain.

I think one of the best things one could do is to pipe your Cloudwatch logs to an ElasticSearch cluster.

One of the best things we did at ipdata was start using Sentry in our lambda functions. We caught an unbelievable number of silent errors that would've been impossible to find with cloudwatch.

We got DoS'd with a few hundred million calls over a few days and the dynamodb cost was about $4.

Any links on how to implement your email routing setup?

>We're starting to ramp up traffic now by orders of magnitude (with many more to come) and it's soooooo awesome knowing the stack is pretty much bombproof.

This does come with additional cost. Serverless pricing doesn't scale, your costs increase linearly with your usage, and there're no discounts for bulk usage or reserved pricing.

We were recently on the receiving end of a massive HTTP GET Flood DDoS and although we did not experience any downtime as a result of it, I ended up finding out about it a few days later when billing alarms started going off.

I've felt the same way. Serverless is most appealing when you're starting out and have low traffic. It enabled us at ipdata.co to have the most global infrastructure possible with the lowest latencies at an insignificant cost.

At some point I believe when we're big enough we might switch to using servers in all the regions where we currently run APIG+Lambda.

No one has mentioned gogs yet https://gogs.io/

Thanks for the feedback, this does help. I'm wary of adding another step before users can get started with the API but it might become necessary to do so.

Thanks again!

I can see all the ip addresses from multiple countries. Via announcements on the website. Some are but others aren't.

Nope, you can call our endpoint without an API key but you'd be limited to 1500 requests a day.

That's a pretty interesting history and put in that perspective I see what you mean and totally agree with you.

It's pretty obvious that hitting your local disk is going to be a lot faster than making a network call. Make the right decision for your use case.