Cost of serving billions of images per month
medium.com
medium.com
But simply, this floors me. I looked at their costs and with some developer muscle, you could find savings such as:
- Move Fastly to Cloudflare. They don't expressly say what the cost is for that. But moving to CF would eliminate it.
- Move Heroku to Digital Ocean. It's not difficult to create a fully redundant solution.
- Move from Imgix to having Golang micro services which handle the resizing of images and use something like Belugacdn for the CDN. Beluga is $5k a month for a PB. (Or some other cheaper CDN if you don't like Beluga, but damn... imgix pricing)
I'm pretty sure that a savvy CTO could save at least $50k a month with a well designed project that does this over many months and achieves the same result and keeps the redundancy concerns for the small team.
I do realise however, why the team have done this. In the same position (with very few resources) I would probably have done the same. But damn, when a cost of a service gets up to a years salary ($120k) for a good developer. Time to seek alternatives.
We serve significantly more bandwidth and requests than their 2016 numbers at under $15k compared to their $17.5k, and we're not particularly lean on infra costs being a .Net shop with Win servers and MS SQL on Azure, and I can still bring our monthly a lot closer to $10k if I can find time for some projects.
But the reality is that even thousands of dollars per month saved can take a good while to ROI against the costs of development, or more so the opportunity costs.
Correct. Cloudflare's terms state that you can't serve a disproportionate/substantial amount of non-HTML content, so they would need to sign up for Enterprise with a full contract to get image serving via CF.
Enterprise is generally $5k+/month, but CF contracts can offer a subset of enterprise features from $500-$5k/month, so I would guess ~3k/month just to serve multiple petabytes of images. This doesn't eliminate the bandwidth costs so CF may not be the single solution for the bandwidth costs.
Fastly is used for their actual website, judging by the ASN info. Meaning they wouldn't be serving petabytes of images from CloudFlare, just HTML/CSS/JS, which should be perfectly fine.
To manage infrastructure at that scale themselves they'd need someone who knows DevOps. To write services at that scale they'd need someone who knows backend engineering and micro services. Then they'd need to be able to have 24/7 on call rotations for when things break. And of course, like any rewrite it seems like a "couple month" project until you get into the details and it becomes a two year project. Soon you've got a full 6+ person team and are paying them a lot more than $600k/year.
This doesn't have any SLAs nor is it critical software either. 24/7 oncall is completely unnecessary.
Just because everything is in AWS doesn’t force you to use all of AWS services.
But since images are immutable, this is a perfect use-case for lambda functions fronted by a CDN.
The main scaling challenge here is serving and bandwidth, both of which are handled by their CDN. I expect 5 figure costs for that but the rest of the platform is very expensive.
Do you actually run pay-per-action campaigns, or do you provide a dashboard for the customer data ?
Nope. You're forgetting the cost to the company is the burdened labor cost (https://en.wikipedia.org/wiki/Labor_burden), not just wages. Payroll taxes, benefits, office space and equipment, etc. can add another 30-40% to the cost of wages.
We would never run with such expensive options for this functionality, especially if it was a free/donations powered service. This company raised $7.5M though: https://techcrunch.com/2018/02/15/unsplash-simple-token-seri...
(I say this as a user of and contributor to Unsplash)
I am curious how far they are going to grow before they turn the ad machine full throttle, and whether or not that will have a huge adverse effect.
We all kind of know that Instagram is a giant online store for everything, at least it has been for a few years now. Unsplash contributors seem to think that free will last forever. It'd be very interesting to see if they stay or leave.
I use and love the service but I can't see how it can last for more than a few more years.
I hope Unsplash doesn't end up similar to those.
[0] https://podcasts.apple.com/us/podcast/tales-from-the-crypt/i...
Of course, that was also critical to Twitter's success.
Let’s also not forget that they have a vendetta against third-party clients (so they can push their own client with the bullshit algorithmic timeline and ads) so it wouldn’t surprise me that he says that just as an excuse. In reality their web access was just as unrestricted as their API so it should share some of the blame too.
[1] https://techcrunch.com/2018/02/15/unsplash-simple-token-seri...
I run a service that does ~30 million requests a month, and I only spend $15 on it. It's an apples to orange comparison, but looking at their costs breakdown, I'm sure a couple engineers with a couple weeks can dramatically reduce their costs.
For me, the biggest cost savings were at browser caching. With `immutable` and far future cache expiration dates, webp images, etc, I could pull of the 30m and growing stats without the server load going more than 0.40.
It could be true, but would mean engineering time not dedicated at features and likely more risk and more maintenance. It's often hard to admit but costs are not that easy to assess and trade of are tricky.
Seriously, I know I'm not in their shoes but I know what they do, I know popular options in these areas and I know they are literally throwing cash away by not trying to in-house any of this.
The worst thing is they seem proud about their decisions; that their current situation has justified $100K/month expenses. I'm not the only person in here that thinks they could be delivering the same —even better— service on a fraction of their current budget, even after hiring. I think they should be ashamed.
Could they do both, yes. Do they want to, no. The goal, imho, of a startup is doing what you love on your own terms. They clearly love the product and have optimized accordingly.
What I found interesting, is that just a sliver of the budget is spent on software rather than hardware. I don't think this is uncommon, few teams are willing to pay for software they can get for free or make themselves. It's a bit surprising (but explainable), that commodity hardware that everyone has access to dominates the budget.
This is no different from saying people should be ashamed because they pay so much for food, then it’s much cheaper (almost free!) to grow it yourself in a pot.
They're not in an all-or-nothing situation. There are so many individual elements that they could improve efficiency on and they're just not.
And I think cooking your own food vs going to a restaurant is a better analogy (because it focuses on a service provision, not resource gathering). They're currently eating out, seven days a week.
https://medium.com/unsplash/scaling-unsplash-with-a-small-te...
It's absolutely nothing for them to be ashamed of. We can all disagree with the decision and say that we wouldn't do the same, but we shouldn't call it a bad of shameful decision.
Unsplash is eating out seven days a week. But they don't have anything to show for it. They're burning through cash. The product is basic. There's no magic, nothing clever. It just is.
They're saying everything is great! I think it's a shambles; simultaneously mediocre at both making money and being an excellent example of engineering. Why is anybody here celebrating or defending this result?
I maintain what I said. Unsplash should be ashamed. They shouldn't trot out numbers like this label them a success. It's a record of successive failures.
So money is now going to these CDN hosts and other software suppliers, spent like water with a business plan that is not in the article. I guess the likes of Adobe can bung them a few million a year as the service is worth that to them. Same with Squarespace, they can partner and pay a few million to keep unSplash's lights on.
So this has some interesting negative effects from local communities where people once had their own websites and their own local photography businesses. Although unSplash might not directly be reaming out every small town in the world making it impossible for anyone to make a living from stock photography, Squarespace kind of are. Web pages have made themselves complicated today, once you needed Notepad and an FTP program, nowadays you need a team of developers to spend months in order to put 'hello world' online. So people are nowadays going with Squarespace and other highly marketed services, that, in turn, rely on UnSplash.
The images initially look very impressive with UnSplash but I was once really impressed by a Pret A Manger cafe. Then I realised there were many Pret A Manger cafes in many towns and they all were the same inside. Quality stuff but identikit. At the same time, each outlet was not exactly identical, there could be a different amount of chairs, the counters could be laid out differently. (Readers outside the UK could swap Starbucks for Pret a Manger).
I see UnSplash as a photo version of a Pret a Manger cafe. After a while the images are all very much the same. This is good and it is bad. It is bad in that so much of the internet becomes predictable and generic. It is good in that it becomes quite easy to do better. Much like how you can do better than Pret a Manger at making a sandwich, just by making your own, so it is with UnSplash, you can get far better photos for your project by taking your own photos and doing your own editing.
I would not bet against the success of UnSplash any more than I would bet against the success of Pret A Manger. However, I will be making my own bread or visiting independent cafes, oh, and taking my own photos rather than using 'free' formulaic stock photos.
Enough of the analogies, a critique of UnSplash and what they are not doing. If you look at a Squarespace site that has some UnSplash banner then the image will be several thousand pixels by several thousand pixels. This is good but bad. Good in that native phone camera resolution images are nice to see, bad for bandwidth.
This bandwidth is mitigated by CDNs. The CDNs do the different resolutions but it is quite dated tech being used to do this. I think that for most website owners, e.g. one's small business, there is a performance benefit in using one's own hosting and running Google Pagespeed to optimise and serve the images. This can work with the low bandwidth browser flag and do responsive source sets.
This means putting the original image dimensions in the HTML for Pagespeed to write out the srcset images, putting in a picture element and so forth. The images also get sent as webp when needed.
Now if I did want to show some thousand pixel images then I would want to do that with OpenSeadragon. This takes the deep zoom aspect to a whole new level. It works fine for a small website without a complex CDN.
Now there is no reason why I could not 'steal' a few UnSplash images and serve them my own special way with Google Pagespeed doing all the work on my own server but I am not going to be running generic stock photos anyway.
Google's Pagespeed falls in between the gaps between large compartmentised teams so nobody is using it. Why would you when for the same reasons as nobody ever got fired for buying IBM, nobody gets fired for spending a fortune on a CDN? However, if Squarespace and other UnSplash clients decide they can actually offer a better, faster product due to improved tech for serving images then they can start doing their own stock photography.
Many in-house projects end up costing 10x the cost of an external vendor. Of course, that's not something you read about often on HNews, its super common though.
At the last place I worked I was in charge of a team that ran our image service so I have a very solid understanding of the infrastructure around it and what resources we had in place to develop/monitor/support/etc. It wouldn't be hard to do a calculation and see if they are using money inefficiently.
I will say that most people don't really have any idea of the cost of simply pushing bits. The majority of what I did was video distribution which gets into eye-watering costs pretty quickly. Without knowing a bit more detail on the precise volumes they are storing/moving I don't think it is fair to criticize.
Heroku’s sweet spot (to me at least) is for anyone with less than 3-4 servers. After that, it just becomes so expensive there’s no rationalization in the world that makes sense.
I understand a scrappy startup needing to focus on growth and using the most convenient tools, but once you’ve been around a bit it’s time to think of boring stuff like what if stuff goes wrong. It might be technical, security related, business related or political. Eliminating SPOFs is wise.
Happy for you to downvote this but if you do please drop a sentence to say why I’m wrong. I’m happy to be proved wrong it helps me learn.
"I understand a scrappy startup needing to focus on growth and using the most convenient tools"
Well, there's your answer. As a scrappy startup, you can't afford to plan migrations you aren't planning to execute. It's that simple. Heroku is not going away (they're owned by salesforce) and they have a good reputation for uptime.
The "what if" worst case scenarios have to be weighed against other business concerns. And this company appears to have done a good job of navigating such things.
Your comment is not only bad, but outright dangerous. This is a company that is burning through hundreds of thousands of dollars on hosting and staff with no business model yet. The primary risk to this company is not their host going away; the primary risk to this company is that they won't figure out how to make money. Worry about optimizing your serving infrastructure after you have a business that you know is worth saving.
You seem to have some insight to share here though, could you expound?
*edit typo
If you’re a small startup, the least of your business risks are one of the major cloud providers shutting down.
You don't move. You load balance across the clouds prior to the disaster.
You have your stuff work on both. I mentioned K8s because you could set up a managed cluster of that on a few cloud providers, and most of your TF will be the same in terms of setting up k8s, with some differences on how you set up those clouds.
This might be overkill for many people though so see below...
> If you’re a small startup, the least of your business risks are one of the major cloud providers shutting down.
I agree. a major cloud provider wont shut down....
... but they might shut YOU down.
Why? Billing Issues / TOS / 'Suspicious Activity' [0] / etc.
Now do you mitigate for that? Not necessarily but it is worth considering if you need to.
At the preparedness extreme you have a probe that detects the problem and flicks you over to cloud 2. Or a load balancer as I mention above (which would probe). That's probably too much for a scrappy startup.
But a middle ground is you have tech that is easy to move. Doesn't have to be k8s/TF but maybe a bash script you run on a new debian VM or whatever. Then you phone one of you awesome developers at 3am and tell them to migrate to AWS or whatever, and because it's easy they'll figure it out as they go, most stuff running by 3:30am and everything dandy by 5am.
The other extreme is you are tied in heavily to specific stacks by specific providers, and it take X hours/days to get back online again.
I'm not recommending to anyone what to do here - but I am saying consider the black swan events. You might consider them and say no I want my devs adding feature X so we can sell more. Fine, but I think when you can spend $100k a month on cloud you can probably afford to think about it a bit.
[0] Source: one of the major cloud providers cut all our services for 12 hours due to "suspicious activity". Turned out later it was due to a reused IP we were given from a pool that someone else f'd with. They gave us some credits to be nice afterwards.
Again, this is a small struggling company, do you really think they should be spending resources having a backup plan just in case AWS has a multi AZ or multi region outage? Is that really their largest business risk?
Also do they really want to go from the simplicity of Heroku all the way to k8s?
The only things that a company at this level needs to be concerned about are reducing burn rate, finding a way to better monetize, and getting another round of funding.
But, I seriously doubt that a company spending 100K a month would be on their free support plan and not have a business or Enterprise support plan where they wouldn’t have someone to call at AWS with a much smaller SLA than 12 hours. We are a small company and I can just open a support ticket and get someone on the phone/chat immediately.
And if you are a small startup, you aren’t just using AWS with a few VMs. You’re probably also using a lot of other managed services that aren’t VM based. If you are hosting everything yourself. You might as well be at a colo. If you are using your cloud provider as an overpriced data center hosting VMs, you’re probably doing it wrong.
Otherwise there are far cheaper server providers than can get you monster machines and unmetered bandwidth.
And seriously, not everything has to be hosted on k8s, not everyone has to use Terraform, it's ok running your application on Heroku, DO or some other hosting. If Heroku goes out of business, you'll have a few months to preapre a migration plan. It's ok to handle it then, right now there are probably more important things for them to do.
I think this baby is ready for an IPO!
Then the baby is ready for IPO.
I Do wonder why they don’t charge for API at all as even currency APIs charge and that is effectively public knowledge.
Very interesting though as this is some serious scale indeed...
I just would have liked some more details on what it means technically... :-)
Though it makes sense, buy over build is almost always the best choice early stage. It's hard to slow down product velocity to fix things that work fine when costs start to add up. (Also it's easy to say "do you want to save 3k a month or deliver features x,y, and z" and let costs slide)
"We continue to use Heroku as our main web platform. Despite its premium cost over AWS, Azure, and Google Cloud, Heroku’s built-in deployment and configuration tools allow our team to move faster, more confidently, and more reliably.
As we’ve detailed previously, the alternatives would undoubtably be cheaper on paper. But in reality, the increased simplicity and freedom offered by Heroku for a small, product-focused team is a major cost savings advantage."
I'll believe Heroku is easier, but can it be that much easier? For people spending piles of cash on their webscale?
If later on you need to do something more complex you have all kinds of extension points.
If you want something cleaner and just to use a Github -> CodeBuild -> Code Commit/Lambda deployment and have something closer to traditional deployment pipeline, there are the CodeStar templates.
We are a small company without a Devops team. The leads all know how to setup a from scratch system on AWS.
The point is that costs on Heroku and AWS are two intersecting lines with different slopes and y-axis starting points. The dollar cost is the basic dimension, but once you add team hourly rate and opportunity cost you’ll see that Heroku is cheaper than AWS until you hit X000 req/sec or users. That number is varies from team to team, but it’s higher than you think.
We do a lot more with AWS’s managed services that would go beyond what Heroku could do.
That being said, if we were just a simple database+website. How would AWS maintenance be any more than going through the VPC creation wizard one time and using Elastic Beanstalk for deployments? I’m very much an advocate of managed services so it’s not that I’m anti Heroku - that would be hypocritical if I’m saying using EB - But how is it easier?
With Elastic Beanstalk if later on if you do need to add complexity, you easily can through startup scripts and .ebextensions (cloud formation).
All this literally takes one click with Heroku.
Again, I’m far from a dev ops expert, but I cringe at how much EB (and Heroku) does that’s magic and I would much rather use CodeStar with the templates that create a standard CodeBuild/Code Deploy/CodePipeline/CloudFormation set up, but I am comparing like for like.
I’ve never deployed anything to Azure but I have used VSTS (aka Azure Devops) with various combinations of hosted builds, on site builds with agents on the build servers and deployment servers and that’s even easier than AWS’s offerings. Even if the GUI setup would make a real Devops person cringe.
Yes I realize Heroku for all intents and purposes is just another managed service that sits on top of AWS and that you can even buy Heroku services from AWS Marketplace for consolidated billing (https://aws.amazon.com/marketplace/seller-profile?id=0112b5d...)
I’m also not arguing not to use Heroku just because it cost more. We always choose a managed service over having to manage things ourself even going as far as preferring Fargate even though regular EC2 based ECS would be cheaper.
But, I haven’t seen anything that Heroku gives you that couldn’t be duplicated with EB or if you need a more traditional approach a CodeStar generated template.
Every time we need to re-negotiate our contract with Heroku we ask ourselves whether we should move away from Heroku or not. So far, the answer has always been no. Complexity would go up and the time spent on DevOps would go up. Right now, any idiot can do a deployment, or a rollback. Most of the things we need are taken care off for us by Heroku and it doesn't cost us any effort. We use Heroku Postgres for our database and we get point-in-time restore for free, without doing anything. We can upgrade our database or create a new follower in minutes. It just requires a couple of clicks. On top of that, all the monitoring is taking care of by Heroku. It just gives us a nice dashboard.
As for costs, on an negotiated Enterprise contract, it's not that bad. If we'd move to AWS or DO for example, we might save a little bit, but the extra engineering resources would quickly make those savings useless. Even if it's a bit more expensive to run on Heroku, it's still cheaper than cobbling together all of those features on another platform.
We've made a couple of small engineering investments to get features that Heroku doesn't offer. For example, we built a simple auto-scaler for dyno's that consume from RabbitMQ. This helps a bit to keep our costs down and respond to massive spikes quickly.
We run our web servers, database, caching, data pipeline etc on Heroku. However, we are not _that_ dependent on Heroku. We use Docker for deployments, so we can easily reproduce what's happening and it reduces our vendor lock-in.
I hope this explains a little why some people might choose to stick to Heroku. It works for us, it might not work for you or anyone else.
For the record, you get this with Azure too. Not sure about AWS and GCP, but I would have assumed the same.
I missed this part. If you are okay with using managed services, why are you doing so much “undifferentiated heavy lifting”? Setting up autoscaling based on queues, managed RabbitMQ based messages (Amazon MQ), and without knowing what your data pipeline consists of, probably that too could all be done with managed services.
However, we recently found a managed service to do this for us so we'll be switching to that as soon as we can.
If we'd have to spend more engineering resources on our infrastructure than the occasional simple tool, then we'd consider switching away from Heroku. The benefits still outweigh the potential engineering effort of not running on Heroku.
The service seems pretty cool :)
As someone who appreciate creative marketing i like how they generate millions of backlinks by asking you to link back to their site and credit the author.
In few years they became among the most popular sites in their niche, a lot thanks to their creative growth approach.
This is just bullshit, really. I'd buy this excuse from a service like Snapchat/Facebook in their early years, but unsplash is nothing that changes dramatically all the time.
Every single point in their analysis just shows how they like to waste money. Imgix as the main image CDN? Start serving images using thumbor and a cheap CDN like BunnyCDN or KeyCDN.
They could easily save 60% on their monthly bill. That would enable them to hire 1 or 2 engineers capable of maintaining a few dozen servers on AWS, DO or GCP.
If they reduce their bill and then hire engineers to do the same job they've gained nothing other than an increased headcount for no discernible benefit.
So blow up their currently working infra to save 40k to reallocate that 40k to build a replacement? How does that make sense? This idea seems penny wise, pound foolish - run the numbers yourself.