How Ipdata Serves 25M API Calls from 10 Global Endpoints for $150 a Month | Hacker News Reader

How Ipdata Serves 25M API Calls from 10 Global Endpoints for $150 a Month

highscalability.com

Interesting read, thanks for posting, but I think you can do better. I've been playing with Fastly and VCL recently and you could write your whole app in VCL and have it served right from the CDN. 25 million requests would cost about $20 for that. If you include the API keys in the log output and batch-process that each day/hour/whatever to keep usage stats, that would remove the need for Dynamo? You'd then keep a table of banned API keys and update that through the Fastly API dynamically.

Just a thought. Still, $150 p/m is not bad!

Awesome! thanks for sharing! I'm always looking for optimizations. Any idea whether fastly would support running out of specific regions? Or how many endpoints they have?

Not sure if you can run them on specific regions. They have a lot of endpoints though, there is a list on their status page: https://status.fastly.com/. I count 55.

Thanks! I'm excited to try this and benchmark it against my current setup!

Considering it's like 20 simple requests per second, you can do it on a couple of tiny virtual servers (for geo and redundancy), totaling less than $10 per month.

the hard bit is the API key and usage monitoring. you could write the whole thing as Cloudflare workers for $5 per month else.

$20 for 55 'servers' spread worldwide isn't bad, with no maintenance overhead.

Writing Javascript is, in my experience, much more pleasant than writing VCL, and Cloudflare has over 150 points of presence.

It would also take some work to scale

My strategy is to use Cloudflare workers to invoke a Lambda function. This way I have a cheap AWS API Gateway replacement. $3.70 per 1M requests becomes only $0.50. Costing my only $12.5 for 25M requests + the Lambda invocations which are hella cheap.

Interesting strategy mate! Any links on how to do that? I've googled a bit but haven't find anything on this!

I don't know if there are any great blog posts on it yet (write one!) but you can read more about Workers here: https://developers.cloudflare.com/workers/about/

https://blog.cloudflare.com/cloudflare-workers-unleashed/

Yes check this out. What you'll have to do is build the aws sdk into a single javascript file that runs as a Web Worker. These are distributed to over a 100 servers worldwide. Then you would want to invoke the Lambda function from the worker, link below. You could potentially distribute this further by invoking a Lambda function geographically close to that specific worker. But this setup will be very fast already.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lamb...

Awesome! Thanks!

It might be cool to create a tool that automatically turns a serverless.json file into Cloudflare workers that acts as little AWS API Gateway services.

If anyone is up for swapping ideas, I made a GitHub repo:

https://github.com/JrSchild/cloudflare-lambda

That does seem good, but when you consider that it’s basically just doing a DB query for an IP (and there might be a clever way to store CIDRs where the DB finds it within a range) + authentication, authorization, and accounting ( AAA), it becomes a bit less impressive. I’m guessing again that the AAA stuff is actually more work than the actual lookups. I’ve made some services which use geoip before with in-memory lookups. Since it doesn’t change that frequently, aside from AAA, you don’t really need an external DB at all — just have the actual executable contain the lookup info, now you’re down to just lambda. Actually, don’t even build a service — just distribute a binary with the tables inside, so then customers don’t even need to make a call, wait, and pay for each call. Consider how well that would scale (and indeed this is already the business model for some of the MaxMind products (e.g. https://www.maxmind.com/en/geoip2-city ), but it doesn’t make for as interesting of an article.

How I scaled my single-function + single-db-query api!

If anything it's closer to - How I built a multi-regional apigateway API with multiple creative workarounds despite there being no official way to do it and it only becoming possible to do so 5 months ago :)

Hear hear! Clearly a significant amount of effort has went into figuring out this design, and it's interesting to read about it.

EDIT: And you're the author -- thanks!

The article is about building a fast and highly available API distribution architecture. You could just as well apply it to anything you want to build.

I will agree that it's not for everyone. And would probably more likely appeal to indie makers like myself.

Could you not spit out the DB as a JSON file and have it included directly into the lambda as a source file then have any updates as part of your CI? Then it's just an O(1) lookup in a map, and as long as the lambdas stay hot you're flying.

Perhaps a JSON file for 4 billion IPs with all the data would be too large to fit in memory. Who knows. Lambdas can be up to 50MB so maybe this is a dumb idea unless you normalised the data somehow.

super cool article!

I do wonder if you didnt try Apistar (https://github.com/encode/apistar) . Its built by the author of django rest framework .and also is much more frequently updated

Thanks! :)

I had no idea apistar existed, it certainly didn't come up when I was doing my research. But it looks pretty full featured, thank you for this!

What is apistar supposed to be solving in this context?

I haven't worked with it yet, but it looks to me to be a Python framework just like Japronto or Flask but one that is focused heavily on building APIs.

It was a top/root comment that's why I asked. I thought it does something special. Also worth a look: http://python-eve.org/

No prob, Eve looks pretty cool. I'd love to benchmark it alongside japronto and apistar. Thanks!

Out of interest where did you get the regional data for IP addresses from? Did you collect it yourself, if so how? There seem to be very few sources that allow you to resell it.

We aggregate the data from a couple of sources, see https://ipdata.co/about.html

Why not push to Kinesis directly in a background worker?

I understand printing log is simple but you still have to push that log to cloudwatch... it's still being done on the same system. So, you end up paying that cost either way.

You can use MongoDB atlas with multi-region replication for low latency queries instead of DynamoDB.

Reply on news.ycombinator.com