Hacker News Reader

Comment by jonplackett | Hacker News Reader

When I read something like this I realise just how immense of a lead tech companies are building vs 'normal' companies. There's just layer of expertise in how to streamline and organise themselves that anyone without a 1000 strong engineering team and huge engineering budget just are never going to think about.

View on HN

Having worked at some of those companies, along with small startups, the biggest difference isn't really plain old sophistication, but expenses so high as to make the engineering effort worthwhile.

If your production system is 10 instances on some random cloud, a 10% efficiency savings saves me 1 instance, so maybe $2k a year. Taking into account opportunity costs vs doing things that raise revenue, said startup would consider the effort a waste of time unless it took a few hours.

With the same architecture, but instead 20K instances, then suddenly that 10% is saving 2000 instances, and 4 million a year. Unless there's a major engineering shortage, chances are that spending a over a month on 4 million in yearly savings will be completely justified, and would even be a highlight in someone SRE's review.

Chances are that the optimization wasn't even any harder in the big tech company: It's just that small savings on big piles of money are suddenly worth it. It's not possible to find millions of dollars in loose change between the proverbial couch cushions if you didn't have the chance to spend hundreds of millions in the first place. Heck, in a growth company, even at that size, 4 million might not be enough savings, as there might be even bigger things.

Same with dev tooling: Saving a company 5% CI times is not great with 10 developers, but if you are making five thousand developers more productive, suddenly you can hire an entire team of very specialized developers, and it's not a luxury.

This is also why often copying what large companies is foolish when you are small: The tradeoffs are going to be completely different. What would be an unacceptable flaw in a large company is just fine in your startup.

The real trick IMO is the 200-800 range: Large enough that the simple solutions for small companies have probably broken down and are causing pain, but nowhere near the staffing to hire yourself a team to, say, add types to Ruby, get a team to build around the weaknesses of your database, or whichever other problem you have that any member of FAANG would just throw 10 million dollars in staffing costs without batting an eye. I've seen way too many companies that stopped being able to grow their company at those intermediate sizes, and get stuck in technical hell.

The problem is that managers at small companies with some, but insufficient, technical knowledge also read these articles and think they need to do the same. I come across overengineereed stuff at small companies all the time for things they don't need but only do because it's what Google/Facebook is doing.

Engineers do this mostly, not as much managers (even less so for technical managers). Managers typically either want things done yesterday and would see this as wasteful over-engineering, or would be delegating and trusting a senior or staff engineer if they said this was truly necessary.

This is where things are heading, especially for business or corporate 'consumers':

Need web hosting? Certainly you can't deal with a 2Tbps DDoS attack, so let's route that traffic through Cloudflare.

Sending email? Can't send it out with those /23s you have, they're on some blacklist from a shady DNSBL provider. Need to pay for GSuite or Office365.

Can't protect your CEO against a targeted attack? GSuite Advanced Protection or Microsoft ATP has you covered.

Some of those have been true for quite some time. I ran my own mail server back in the 2000s to early 2010s, and incoming mail was especially hellish (effective spam filtering was basically impossible).

These are really just variations on the theme of economies of scale, aren't they?

commercial dns black lists can be very effective at filtering spam and not expensive for personal use.

Ill argue that you don't need it for a small company - just don't overcomplicate your life by using technology that's designed for big companies.

Stick to simple systems like elastic beanstalk or app engine + RDS and something like snowflake and you get powerful, manageable, auditable projects. I'm also finding great benefit in launching each project in its own AWS account. This means you know exactly what project costs how much!

I was thinking more in terms of large but traditional companies. The kind tech companies put out of business regularly!

If it makes you feel better, normal companies have "cost accountants", whose entire job is to worry about this kind of problem. Cost accounting a fairly well-developed field with a long history. There are several major approaches and some less well known ones.

I think the most applicable in our field is "resource cost accounting", which looks superficially like what Netflix described. You keep "resource accounts" in the units of the actual stuff consumed by your business processes, then map backwards from the final product to costs.

Of course I am not an accountant. Just took a course and found it enlightening.

I think the difference is that Netflix sees these problems and has the person-power to solve them and make themselves leaner. I guess it comes down to how far ahead you're looking really and tech companies are incentivised to look further ahead than your average big company. They're all planning for world domination (whether that actually happens or not)

I don't disagree, I'm just trying to convey that in some sense the forward-looking effort would be accelerated by looking sideways as well. Cost accounting is practiced widely in many industries and appears in medium-sized businesses (tens of employees) as well as large businesses (tens of thousands of employees).

Reply on news.ycombinator.com