Hacker News Reader

Comment by liquidise | Hacker News Reader

> Hosting costs per DAU were $0.68 in Q3 2017 as compared to $0.64 in Q3 2016 and $0.61 in Q2 2017.

Those are staggering figures to me and the trend is equally surprising. Maybe it's ignorance about hosting costs at such a scale, but i feel like cost/DAU should be declining over time, right? Is there something i'm missing here?

View on HN

They use Google App Engine. Not known for being cheap at scale.

https://www.recode.net/2017/3/1/14661126/snap-snapchat-ipo-s...

Apparently $SNAP is also not known for trying to keep costs down.

Have heard they are working hard on that, which makes this more surprising to me.

They have to receive the picture, then send it to a number of other users. So it's probably mostly bandwidth charges.

They use Google, right? When was the last time Google (or any of the cloud providers) lowered their bandwidth charges?

That small fluctuation is probably just minor variations in usage.

Yes they have a giant contract with GCP https://www.recode.net/2017/3/1/14661126/snap-snapchat-ipo-s...

Google just recently lowered it's egress rates Through the release of standard networking.

Also don't expect snap to be paying list rate.

https://cloudplatform.googleblog.com/2017/08/introducing-Net...

(Disclaimer I work for Google cloud. I also have no direct knowledge of snaps gcp rate but have worked on other offline contracts)

That cost is definitely not bandwidth. Content delivery is around $0.005-0.01 per GB with an external provider. Less if yiu have the scale to do it in house. My guess is that the bulk of that spend is compute + storage/io.

The cost is definitely bandwidth, Snapchat doesn't have much of a storage problem as snaps have an actual TTL.

Assuming half of it is bandwidth costs, that means that a daily active user is using ~380 MB/day (assuming $0.01/GB). That seems pretty hefty as an average. Especially if you consider the encoding snapchat uses puts each 10s video at about 1MB in size. 380MB is a little over an hour of snapchat quality video, that seems excessive as an average. If I were a major stockholder I'd be pushing to see why these hosting costs were so high.

CDN? Cross-region replication? DB replication/failsafe/backups? No idea, just spitballing.

I think that’s $0.60 per quarter not per day. 380MB for the quarter per daily active user doesn’t seem that unreasonable.

$0.60 for 380MB seems unreasonable, though. I think the GP was calculating based on the quarterly period.

Oh, you are right. I was thinking $0.006 not $0.60 when I did that math in my head.

I bet bandwidth is a large cost, but I bet compute is a big component too. Snapchat uses Google App Engine, an awesome platform but expensive (comparable to Heroku).

Regardless, my understanding is that Snap now gets some sort of flat rate deal from Google, so not sure how important the individual components are.

Could also be incompetence in regards to managing servers and load testing with laziness for not shutting down instances and ignorance of using reserved instances?

Ahem, I mean it could be.

Google Cloud does reserve instances automatically...

Hehe! You will be surprised how often this happens in best of the companies. A large fintech startup I work with had 5 GPU servers lying around idle before the CTO realized that they only need them while training the Machine Learning models and not during inference stage.

Been there. Remember getting a call from a data center I'd never heard of asking to speak to the old sysadmin who'd quit 18 month ago. They asked something about the rack of servers we had in their data center, I replied "what rack? what data center?" Turns out that 2-3 years previous the old sysadmin had set up a dozen servers in their data center for some project (that I'd never heard of) that very shortly afterwards first got pushed forwards and then canceled. Everybody involved in that project promptly moved on to other things and no remembered to cancel the deal with the data center. So for two years we'd been paying quite a lot of money to keep a completely idle rack of pretty expensive servers and apparently no one noticed.

For every user with the app open, they have to constantly calculate new information too send out, not just snaps but ads and news content. It's not mining bitcoin, but not trivial. Probably a few dozen database reads/writes per user every minute.

Maybe the amount of usage per user is increasing (a metric that isn't really reflected by just counting DAU)?

There is a thriving market in amateur adult modeling and porn.

I am thinking of writing an e-book about it.

I would preorder such an ebook.

Not OP, but my insight: Adult entertainers use Snapchat as a distribution channel for their own self-made videos. Some offer their Snapchat stories for free and try to up-sell you, others charge you for giving you access to their stories. Payment is through Patreon or PayPal or some other common payment platforms.

I'm not sure what "hosting costs" encompasses. But maybe this includes analytics costs? I assume they have far more ML and data collection infrastructure than before.

Snap is paying Google $400m a year for hosting costs... https://techcrunch.com/2017/02/02/snap-commits-2-billion-ove...

Seems like at that point it might be cheaper to run your own datacenters?

> Seems like at that point it might be cheaper to run your own datacenters?

Depends on whether they acquire new users in the future. Their hosting costs are fixed, so doubling their user count would halve their per-user costs.

Your own DC, sure, but to build out an entire cloud platform? Even with OpenStack and the like, it's anything but trivial.

I'm puzzled by that - if it's $400m/yr then cost per user should go down as you have more but they seem to be reporting the opposite. ("Hosting costs per DAU were $0.68 in Q3 2017 as compared to $0.64 in Q3 2016")

More features = more resources.

You need a culture of cleaning up tech debt to reduce resource usage.

With so many holes in the ship, the company is going to be focused on features to drive up RPU.

Is this a monthly cost or yearly?

I think it's quarterly. (i.e. the annual hosting cost per user is a little over $2.50)

Probably new features. The new 3D filters look like they could require more resources to serve (filters are not included in the app but rather downloaded dynamically iirc). Maybe they're rolling out more localized filters too.

That stuck out to me as well. It kind of seems to me like they’re saying variable cost of goods sold increased

Maybe those snaps aren't actually getting deleted but are kept for ever...

Reply on news.ycombinator.com