How I Took an API Side Project to 250M Daily Requests
blog.ipinfo.io
blog.ipinfo.io
1) more accurate details
2) fraud protection, if the ip is known for fraud or spamming
3) increased rate limits, etc
Just the top of my head s.. I'm sure you i can think of many more..
Since the OP in this case only has a limited free version, I don't think it's going to be an issue.
https://chrome.google.com/webstore/detail/cookie-inspector/j...
I solely marketed it at Stack Overflow and was getting upvotes and that was all my marketing.
https://superuser.com/questions/244062/how-do-i-view-add-or-...
Also a big factor there are good reviews. When users like your project/product, they will market it for you.
westoque - might want to check your code base, if the adware is unintentional.
You can verify that source, enable external sources in Chrome, ('developer mode' or somthing) and then install it - but that says nothing about what's on the Chrome store.
The .crz file type for Chrome extensions is actually a .zip file, so the code can be inspected. If it's obfuscated this doesn't mean much though.
The only way to safely install extensions is to pack them yourself from versions you trust, then drag + drop them into the extensions window.
Will fix immediately :)
Adding to my answer above: users are more likely to install your product too if you provide good customer support.
What if spending money on marketing had made you grow twice larger? Twice faster?
When people say "I didn't spend money on marketing", the only translation is "I knowingly overlooked massive growth opportunities."
> I built the API in a few hours, posted the answer, and forgot about it — until a few months later I got an email saying my server usage was off the charts. I’d been getting millions of requests per day.
that they basically just set it up as a side-project and answered a couple questions on SO about it only to be confronted with its explosive growth after a few months.
One could then interpret the author in that situation as thinking "don't need marketing, who's gonna use this little thing I made anyway. [a couple months later] Oh shit, this blew up and I didn't even need to spend anything on marketing it besides plugging it on Stackoverflow occasionally".
If the headline simply reads "Taking $foo to 250M API req/day" and then it turns out that all of the traffic came from a $4MM ad spend, it's a lot less interesting to those without $4MM to spend.
I think a more useful way to think about it is "Know your users, and your channel." There are some user populations - and developers are usually one of them - that are virulently anti-advertising. Any paid channel usually earns instant distrust from them. Putting money into marketing spending can have a negative ROI for them, because it has a signaling effect on the brand that says "Our product quality isn't good enough for us to get users without paying for them."
Then there are other user populations - most e-commerce is like this - where there is no such signaling effect, and they are happy to check out new products, regardless of how they hear about the product. For these markets, it's silly to ignore paid channels; you're just leaving money on the table.
When ad campaigns are run, they usually work by targeting programmers' bosses, encouraging the organization to purchase & standardize on one solution and then forcing developers to adopt it via management fiat. That's the route taken by Java, MongoDB, Oracle, .NET, many of the companies in the Hadoop ecosystem, etc. It certainly works - these are big companies - but it requires a consultative enterprise sales team that can work to get the solution deployed across the whole enterprise. The author obviously doesn't have the resources for that.
The middle ground, where you provide a developer tool with $100-200/year CLTV and hope to get it distributed via paid advertising, is a very difficult place to occupy. That's the area occupied by FogBugz and RethinkDB and Sandstorm and many analytics providers. Most of them go the community-building route, and it's still very difficult. Only company I can think of that's thrived here is GitHub, and they went the word-of-mouth, community-building route first. Companies like Jetbrains, Atlassian, Rational, etc. thrive as well, but they've got large enterprise sales teams that standardize a whole organization on their products.
The point is that developers don't just buy software. They're marketed stuff constantly, like everybody else. To suggest they're immune to it seems a little naive to me.
Same reason a good mattress is worth $1k+.
You spend a lot of time in it, and quality product greatly increases your quality of life (back pain etc).
Note: I have no idea if Aeron chairs are good or not.
I first found out about ipinfo from a Google search looking to solve this problem. Haven't pulled the trigger on using it, but it's been on the back of my mind for awhile now.
For me, their marketing happened to be their placement on Google. I don't really know how you'd pursue paid channels on this one (outside of SEM and SEO).
It's a solution to a known problem. Unlike, say, the Apple Watch which has to first convince you have a problem... (This coming from a guy who's very interested in buying a watch as I train for a marathon.)
Also, it leads to added risk to building something people don't want and only engaged because of the marketing drowned out finding what they really wanted.
being successful without marketing probably means you were successful due to word of mouth. Its hard to be successful due to word of mouth when your product sucks.
Someone searching for "IP API" on StackOverflow and having his site rank in the top result(s) is the same as searching for "dresses" on Google, but without having to pay for search placement and getting 100% targeted traffic.
I'd compare it to having a kid. If you have money you can probably buy your kid into a good school and hook them up with a good job when they graduate. Or you can set them up with the right foundation and watch them succeed on their own.
I did the same as the OP with a project of mine. I never spent a penny on ads and it's grown into a very popular charting/trading tool among cryptocurrency traders. I also take pride in the fact that I grew it organically.
What's puzzling is that you would deprive yourself from making your great product known to more people simply because you want to do it the "hard way."
Forgive me, but I'd call it "the dumb way."
That's not the only translation. Another one is "I knowingly avoided pouring lots of money down the drain."
After all, just because you spend money on marketing doesn't mean you will see any returns.
We saw a 500,000% gain in sales by marketing, but to be fair we were small and had no idea what we were doing.
I was as skeptical as they come, but a good marketing team basically launched us from nobodies into LEO. Yeah we had a great core product but still... gotta know how to sell it.
What it does is show where every asset on a web page is loaded from. It allows you to visualize how many different requests go into building just one web page. While it's gotten much better, the Houston Chronicle (https://chron.com) used to make about 500 individual requests to build its home page. It's down to about 125.
It's best to run it across two different monitors, with IP Request Mapper on one monitor and your "normal" browser window on another. Then enter any URL and watch the map start populating based on the geolocating every request made by the page.
But it's projects like ipinfo.io that make these other things possible. Standing on the shoulders of giants and all that...kudos to you, coderholic.
Meanwhile, this is a scan for a particularly noisy German newspaper website (faz.net): https://urlscan.io/result/f23e2e7e-e1eb-4591-9794-92f97957dd...
This website contacted 35 IPs in 7 countries across 24 domains to perform 302 HTTP transactions. Of those, 51 were HTTPS (17 %) and 35% were IPv6. The main IP is 92.123.94.227, located in European Union and belongs to AKAMAI-ASN1. In total, 4 MB of data was transfered, which is 9 MB uncompressed. It took 2.51 seconds to load this page. 16 cookies were set, and 42 messages to the console were logged.
invalid csrf token Error code 403
It might make developers think twice about how they build sites if they could see how overly complex they get. As I mentioned, the Houston Chronicle site used to require about 4x the number of requests as it does now so someone did some optimization.
I mean, you might spend 20 minutes more to set it up, but you are safe from having to rely on 3rd party service.
Anyway, kudos to coderholic for creating this and sharing the story.
[1] https://www.google.com/search?q=geoip+download
[2] https://dev.maxmind.com/geoip/legacy/geolite/
[3] https://dev.maxmind.com/geoip/legacy/downloadable/
- Database needs to be distributed to your servers
- Database can become out of date easily
- Database lookup requires going to local disk and having a relatively fast access path/cache for lookups
- In general, a local database requires a large amount of effort compared to just running a curl in your PHP code.
If you are actually going to use a database, the proper solution does not look like "put it on your webservers" anyway, it looks like "put it on a separate service with a fast caching layer" etc etc. So in other words, the proper solution to decouple yourself from a 3rd party API is to... build a 1st party API.
In other words, not a 20 minute job. For small shops, a quick curl during the page load is a 20 minute job.
docker run -p 8080:8080 -d fiorix/freegeoip curl localhost:8080/json/1.2.3.4
Connected to 35.165.108.15:443
HTTP/1.1 200 OK
Server: nginx/1.8.1
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=utf-8
Date: Sun, 02 Jul 2017 08:54:40 GMT
X-Content-Type-Options: nosniff
Connection: keep-alive
Body discarded
DNS Lookup TCP Connection TLS Handshake Server Processing Content Transfer
[ 4ms | 244ms | 518ms | 525ms | 0ms ]
| | | | |
namelookup:4ms | | | |
connect:248ms | | |
pretransfer:767ms | |
starttransfer:1292ms |
total:1292msIt’s great for checking something quickly, otherwise you have the same thing in browser’s dev tools.
(Not to mention with very minimal effort you can usually avoid the majority of the specific tax of latency you mention, by doing things like parallelizing the request with other work or doing it asynchronously to the user's interface.)
Running a Docker container in production is not a 20 minute job.
Was just an extra step in our build pipeline.
In this case the geoIP database is like a 1MB CSV file, provided for free to the entire world by maxmind (a major network provider).
You can put give that file to many servers like nginx/apache out of the box, they will start adding a header with the country and the city of the client.
What this service is doing is effectively looking up the ip block in that csv file and returning the result as JSON.
having hosers abuse your free geoip service listed off the first hit from google is nice but the data being provided can't just be "hacked together" :P.
1: https://en.wikipedia.org/wiki/Fictitious_entry
2: http://www.cnn.com/2011/TECH/web/02/02/google.bing.sting/ind...
I read that you use Elastic Beanstalk for your server config, but I wanted to ask: 1. What programming language did you use?
2. What, if any, configuration did you have to do to the Elastic Beanstalk config to deal with network spikes and autoscaling?
Thanks!
- I somehow can remember that domain. I don't have to google "my ip" and dig through weird domains that change all the time
- The design is clean and simple. Not too many information, no ads, loads fast.
[1] http://i.imgur.com/gVFAMhz.png
I even think I embedded it in a PoC software I made in a previous company :)
curl curlmyip.org dig +short myip.opendns.com @resolver1.opendns.com
In my case, I have a ip_remote.fish file in my $HOME/.config/fish/functions folder which defines an ip_remote function that executes the line above.As an added bonus, you can get all local IPv4 with:
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'- https://db-ip.com/api - https://ipapi.co - https://freegeoip.net - ipinfodb.com - https://www.iplocation.net - http://neutrinoapi.com - http://www.ip2location.com - https://www.telize.com
and a few dozen more. I wonder if collectively they are serving over a few billion requests per day. Microservices & API culture FTW !
The one cited simply echoes back your IP. That's it. How cheaply could you do that and how many requests per second could you handle on one small VPS?
Example, I recently ran https://www.tactical2017.com/ which is a tactical voting website for the UK general election. The cost for serving that whole website to 2.6 million people over 5 weeks, and 650k people in the last day and a half, was $20.
Just push costs down.
Building for what folks want, even developers, is so obvious that I think we often forget about it. It's also not as glamorous as self driving cars or rockets, so gets discredited easily.
Sound points though
Keep improving this and with the rise of web personalization, the demand will continue to grow.
The answers on the SO question https://stackoverflow.com/q/409999/325521 that OP refers to (in his blog post)
also has another answer using freegeoip.net => https://stackoverflow.com/a/16589641/325521
From the comments on this answer (not OP's answer linking to his API), it seems like freegeoip is not all that reliable (i.e. it's down a lot).
Funnily enough, 1 of the comments on this answers links to another free service, called "freegeoip2" which seems to work just fine as of right now.
I got inspired and start researching and Building. ( btw failing yet)
Essentially, a blocking script in the dom <script src="...api.js" /> that prepopulates the window object. With clever error handling, this could improve perceived performance significantly.
A few questions:
1. What differentiates you from ip-api.com and other providers?
2. Do you use MaxMind?
3. Is there an option for no-throttling? 100s of simultaneous requests?
I aggregate multiple IP databases for my SaaS (https://www.geoscreenshot.com) and I need highly performant / reliable IP look ups.
For example, if IP is in China, local fallback for Google CDN as it will fail.
Minor nit, but with that level of traffic I'd expect you to be bragging about P99.99 latency, not P90.
What exactly does that actually mean though?
Does it mean that processing time at your server is 10ms, or 10ms to time to first byte, or something else?
Giving it a quick test, I generally get the actual JSON result in around 400ms. The lowest I got was 200ms, the highest around 1000ms. It didn't seem to make any difference if I used the HTTP endpoint instead of the HTTPS one.
I wonder if having an IPv6 version would work a lot better?
However, it is important to acknowledge that he did put himself into a position where he was available to become lucky (= he built the API and linked to it).
What percentage of those 250,000,000 is from paid plans? Even if it's only 20% you'd be doing $xx,xxx per day. Is that in the ballpark?
Although I must admit, I'm a bit surprised as to why anyone would pay for this, as several HN readers have listed atleast 5 other free (and some self-hosted) alternatives here...
I am ready to launch a startup and currently trying to figure out what to focus on (so many ideas!).
I posted an "Ask HN" earlier today. Wondering if anyone might have some thoughts or advice on this:
400 * 250M / 320K = $312,500 per month.
Or $3.75M per year.
Not counting the expenses.