Build versus Buy
lethain.com
lethain.com
Their profit margin is usually in their efficiency in doing it better than you, and at the same time you get the benefit of all the features they add for other customers.
Oftentimes the decision to build usually came down to the difficulty in integration of a commercial product, not the cost. In most cases the cost is set to be competitive against building yourself.
Except in the case of Splunk. Their pricing is ridiculous.
In the early days Splunk pricing was exorbitant (we evaluated Splunk 7 years ago and dismissed it), but licensing has changed in recent years and it is now priced by volume ingested (the pricing is transparent and listed on their website now). At low volumes, the pricing is similar to Sumologic, and is pretty accessible now to smaller dev shops. Open-source collectors like fluentd also help to intelligently reduce the ingest volume.
At high volumes, the TCO changes quite a bit.
As an example, we have a pipeline of services. I can compute the time spent in each service with multiple levels of percentiles and group that data by high cardinality fields (as in, hundreds of thousands or more values). I just did a search for 4 hours of data across thousands of nodes for half a dozen or so services with multiple eval statements all piped to a timechart doing over a dozen stats operations. Half a billion events. It got done in under a minute.
Splunk charges so much because they are just so dang powerful.
If you can answer that, it's all the justification you need for building it. Anything you build that doesn't differentiate you can be bought by a competitor.
Let's say you build some kind of product based on some Azure/AWS product. The negative reading of it is: you spend money on innovation and have to pay Azure/AWS for infrastructure. Your product will constantly have to improve to keep up with the competitors, Azure/AWS keeps getting your money. And if your product becomes stable enough, they will make it part of their offering.
Another example would be Apps, with Apple later folding their functionality into the OS and kicking your app off the store.
But judging by my own examples, this migth be more of a platform problem.
The other side of that coin is that you use a build horizontal integration strategy to replace elements of the products you buy from Azure/AWS if (and only if) it can be done in a way that increase your profit/value.
It can work both ways. Moving from buy to build also has elements of de-risking and economies of scale if you do it at the point where there is measurable success/product to market fit.
Building something you can buy too early runs the risk of premature optimisation if not done for the right reasons.
There's absolutely nothing wrong with this way of making money; the lower the margins, the more watching your costs is crucially important, and the difference between making a living and not even breaking even!
I just mention it because most people in the world have a job that make them less money than my friends web services.
And I'm assuming most startups, unless heavily founded, may have to think about this trade off as well.
The cost in infra can easily be a factor of ten.
The bigger you are and the more you can think about optimizing spend.
I know for a fact our bill was around 15-20k USD per month for running a webapp that could be ran in digital ocean 40 dollars/month.
For a lifestyle business without VC money to burn it makes all the difference.
Though it sets up an interesting situation for me, where the "special sauce" of _my product_ is the 80% "off the shelf" for my customers.
Their pricing is ridiculous and what is even stranger is that with the ELK stack there is a good and free alternative. Now Splunk is good, no doubt, but I still wonder how they can be that successful with that pricing.
So people start running anything — from dashboards to analytics — from Splunk alone. Their ability to combine and extract data from almost anything in very small amounts of time is unparalleled IMO.
By contrast, whenever I've been on a project where the company went straight to building their own, it ended up being a quagmire of scope creep, and the development team would end up stuck on an endless treadmill of implementing features that stakeholders vehemently insist on, but never actually use.
Even for dev tooling and libraries. Some of the worst piles of technical debt I've encountered happened when some rockstar decided that they knew better than everyone else, and dived straight into building what they thought would be a better mouse trap. It's not that the in-house option isn't any more or less likely to solve the problem well (no comment), so much as that the in-house option is going to end up being more deeply tangled with the rest of the codebase. So you're stuck with it, because, even if it doesn't turn out to be all it's cracked up to be, migrating away from it may be next to impossible. Homegrown ORMs seem to always turn out this way.
The one exception is when you know that a quick-and-dirty DIY solution will be cheaper and easier, because it needs to do only one specific thing while the off-the-shelf option has to take on a bunch of extra complexity in order to be general enough for everyone's needs. But even there, I'm only likely to trust that argument if it's being made by someone who's been burned by the complexities of the off-the-shelf solution in the past, and who has a proven track record of obeying the KISS principle.
Brooks' software system products (9x the work) is usually done this way in practice. In most usage cases, I would also include open source projects.
It's so hard to get interfaces right and it's helpful to have many users. But I considered your point, of it helping to enforce boundaries.
This is a very nice way of framing the question.
I think there may be at least two scenarios to consider. For some products, cost is primarily driven by R&D; for others, cost of operating the service (COGS) dominates. Of course it's a continuum, but I suspect there are more services near the ends than the middle.
For an R&D-dominated service, if your needs are nontrivial and there is an off-the-shelf product that is a decent fit, you probably want to go off-the-shelf. A commercial provider can amortize the R&D over a large customer base.
For a COGS-dominated service, the commercial provider may still be more efficient but it is less of a slam dunk. Log management (e.g. Splunk) involves a substantial operational component, because you're dealing with large volumes of data.
Disclaimer: I am the founder of Scalyr, a Splunk competitor. Early on we realized that, even though log management is COGS-heavy, we could achieve huge economies of scale, because query workloads are highly bursty. By carefully managing a central pool of resources shared by many clients, we're able to run much more efficiently than a single-tenant homegrown solution [1]. This is one of the key points that we highlight when having build vs. buy discussions with potential customers.
[1] https://www.scalyr.com/blog/searching-1tb-sec-systems-engine...
Are there other examples of "we chose to go to market like this (x, y, z reasons) and here's why"?
I am currently using logdna as my splunk alternative, but it's not very good.
Why should I try scalyr next, over the other alternatives ?
My main constraint is the solution should be at least half as good as splunk, and cost less than $50 a month (I generate less than 1G a day).
Your pricing seems to be reasonable. But how do you compare to splunk on features ?
Here are the other tools I am considering trying
You can find a more general overview at https://scalyr.com/product.
Seems like you guys fit the bill. I'll do the 30 day trial, and see how it goes. Thanks for the response.
Steve says elsewhere in the thread that Splunk has the more-flexible query language, and while that may be true I as a product engineer never noticed the delta myself.
TL;DR I think you'd be doing yourself a favor to try Scalyr, and I have no affiliation other than being a happy former customer.
Disclaimer: I'm one of the founders. We built it to be a Splunk alternative and I think it does a great job from a data ingest, scalability, and data querying perspective. We're lacking some of the out-of-the-box capability, but not for long. Kits (our "apps") release is coming this quarter.
Part of this story is here https://netflixtechblog.com/introducing-winston-event-driven...
Disclamer: StackStorm founder here (but not pitching).
What if there are significant costs associated with that? Integration, training, friction to move to a new solution later, etc?
When things are commoditized it works OK but for anything complex and strategic, the cost of implementing a potentially bad solution can be absolutely crippling, especially if dependencies on that solution grow rapidly within the organisation.
"When it comes to build versus buy, the frequently repeated but rarely followed wisdom is good advice: if you're a technology company, vendors usually generate significant value if they're outside your company's core competency; within your core competency, they generally slow you down."
Every startup I've ever worked with has had engineers who advocated "build" for tooling that fell both outside the eng team's workflow and our company's core competencies.
I think part of it is that things like marketing automation or customer support software—things that are vital for most teams past a certain scale—don't seem technically "hard" enough to justify the price-tags, particularly to engineers who've been isolated from other parts of the business.
Without fail, every time I've been part of a team building an internal tool that we/another engineering team won't use, it goes poorly. I've spent way too long learning about the ins and outs of email deliverability so that I could fix a bug in a hacky email platform, when we could have just bought a platform that works.
As someone who's built far too many of those because "Buying is just too expensive" ... good god do I never want to do that ever again. Ever. It's so fucking hard. You will shoot yourself in the foot. You will suffer. And 3 years later you still won't have something half as good as the purpose-built SaaS.
If you do that and enjoy it, just spin it off into its own product. Who knows, you might even get more customers with more expensive problems than the original startup.
They'll learn more by building something in-house than by bringing in a saas and just writing glue for a living. If you want your engineers to make the decisions a founder would make, give them a significant equity stake.
There's a balance.
Definitely not worth it from a business perspective, but has resulted in some interesting toy/hobby projects!
— Joel Spolsky, In Defense of Not-Invented-Here Syndrome: https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
I think you also need to evaluate if it can even be done better than everyone else, and the effort required to do so.
Joel's quote is more narrow, which is a subset of the dangers of being completely reliant on someone else for your business.
However, even more annoying is working at a Never-Invented-Here organization. I've been at a place which hired brilliant people, but then didn't let them innovate on pretty much anything. Write a simple TCP listener in Python to read a stream of data and then reply to it? That's crazy talk! We need to write a full Django app and deploy it behind Nginx behind a load balancer so that we can pretend it's highly available!
Don't get me wrong: the off the shelf approach is usually the right thing, but sometimes it just adds complication where it didn't need to be. It's insulting to be told you were hired for your expertise, but then not be allowed to actually use it.
This has infected my company quite a bit. People get really nervous as soon as you write something that’s little novel. Writing everything from scratch is bad but you also need room for people to innovate.
I’ve found this to be common in places with lower engineering skills. The “engineers” are mortally afraid of building anything novel for fear of failure; the COTS approach can get many clever folks pretty far in their career.
Unless you've spent the last 10+ years working on encryption full time, please don't try it yourself.
I used to think I could write crypto. Then I learned more and realized it needs specialized knowledge I don't have the time to acquire. Then I spent some time trying to acquire it and figured I was OK to try again. Then I watched many other more experienced organizations crash and burn. My current position is that I know enough about crypto to know I never want anything to do with authoring it.
And if it is core functionality, you should have staff qualified to roll their own, and you should all know why you are doing this (i.e. what differentiation you are going after). This really isn't limited to encryption, it's just a hard domain that bumps up against a lot of other domains, so you see more failures than some others.
Basically if you are going to build a startup or new program in a hard domain, you need to have the resources to do it properly, and you need to know what those are before you start. This is where being a team of smart and motivated individuals may just not be enough.
Look at Intercom: now it's a juggernaut, but it started as an API mashup with a UI in front of it. If you can hit the market and start getting revenue faster you should. Many services have pay as you go and/or startup discounts.
That's your launch platform, then you start working from there.
There is also the question of brain drain. The IT department in my company has outsourced a lot of the tech work to the point that there aren’t many people left who are capable of making informed purchasing decisions. They mostly read sales presentations, white papers and Gardner reports but they have no ability to really assess things. This shows in them repeatedly buying expensive systems that never get really used.
I can attest that the sales presentations tend to be overly optimistic and cheery and if the head guy really wants a given solution, from office politics perspective, you might as well just nod your head, point out the issues you think should be addressed just so that you are covered and roll with it. In practical terms, the decision is not up to you.
In one of my previous roles, manager simply said it is happening ( so it did ), spent some time with the system not fully understanding the underlying logic and was somewhat surprised, when it did not work as advertised. IT had to get involved to smooth some edges. The roll out was still painful. But the contract was already signed and the advertised feature shown during presentation was now extra money, so we were forced to use incomplete product. Annoying.
But then, the real answer is that it varies. Everyone needs to do their due diligence.
What I did not notice is that managers see pretty dashboard and fail to consider the use of the rank and file users. This is the one thing I have learned from all this.
Never underestimate the impact of a pretty dashboard.
There is also an undereported gain that is freeing your team to work on higher impact work.
- More senior buyers (CTOs, VPs) are less likely to pit the vendor against a DIY option. I suspect it's because they've been through too many failed DIY projects, or they have great clarity on what is and isn't a good use of their people's time, or they are better at estimating the (usually huge) amount of effort that would go into DIY.
- As the OP points out, a company with huge engineering resources is more likely to consider the DIY option, because, hey, they have all these engineers that need something to do. So if you're a software vendor and are tired of the "build vs buy" conversation, stop trying to sell to other software vendors or VC-backed tech companies.
- If someone chooses the DIY route, just set a calendar reminder to check in two months from now. Chances are they'll be underwhelmed with their progress, and an out-of-the-box solution might sound pretty good by then. (Don't assume they'd reach out on their own in that scenario--they've probably spoken with dozens of other vendors since your last conversation, and forgot about you.)
===
Separately, the part about vendor management is gold:
> To get the full value from vendors, you have to invest in managing vendors well.
I've spent so much time helping evaluate and implement different vendors (usually marketing platforms and similar), only to find out three months later that the software barely gets used.
On the flip side, if you're a software vendor, don't wait until renewal periods to check in on your customers and make sure they're getting the maximum value from your product.
My policy is: no pricing on the website, requirement to call a salesperson to talk price? No buy. I'm even willing to pay more to a company with transparent pricing that doesn't require negotiation. It gives me predictability, rather than have to worry that once they've "locked me in" they're going to jack the price and crank all the gears.
We had the same issue with splunk. We were blowing through our licenses like a champ.
However the thing that managed to reduce our dependency on splunk wasnt ELK or greylog (they are all poor comparisons to splunk in terms of speed or chopping and changing data) it was grafana(with cloudwatch and graphite, later promethius)
What we realised was that the primary way we were consuming splunk was in graphs. Why spend millions of pounds on generating graphs indirectly, when we could make them directly, in near real time!
Not Tangent:
I think like platform choice, it depends on your company. The company that I was with was utterly incompetent, and unable to see a project through to completion. Therefore if we build a logging system inhouse, we'd get 40% of the way there and rebuild it. Leaving two incomplete systems.
Migrating to splunk cloud was done to save costs, and it actually worked. Had we tried to go inhouse, we'd still have expensive local splunk _and_ a shitty splunk replacement used by two team on a legacy critical system with no dev hours.
I just spent an afternoon helping someone to get us to run on-prem. Phone calls! Installs! Vendor management! We didn't even setup TLS b/c gov stuff is complicated! Working with vendor software is so painful!
Except they were originally thinking they'd have to build what we've spent years not only building, but building as GPU / data / etc. specialists, and we've already adopted many best practices for stuff like TLS and navigated other gov systems for them. (It can still be way better, but imagine if you had none of that stuff, and then had to support it!). Even ignore how we're special -- GPUs are magical if you use them right -- just that we have basic stuff built in and with on-going support. That's so key and adds up over the years. We do it for our graph investigation niche, but same deal for almost everything.
As software has matured, and I see more and more orgs dealing with the reality that they're not Google nor Stripe with thousands of the best-paid software engineers, I'm convinced articles should be explaining from the default of "not build".
Besides, building in house is inherently more fun. What would you rather do, write new software, or pore over manuals for software that someone else wrote? For my part, there's a reason I became a programmer and not a tech writer.
Folks don't get recognized as heroes for not doing anything, but when they drive the greater story, everyone wins. Understanding the individuals involved in successful software delivery is so important!
That's what someone pays to save with something like Splunk. Pay cash to make SWEs more effective towards solving the problems that make up the business's value prop. Cash is much easier to get than the opportunity to grow the business with the people already there. That time never comes back.
I'm guessing for most people, it's actually "rent".
It's easiest described by an example: we use open source Gitlab extensively. Numerous times it has been proposed that we migrate to $expensive alternative because they all have advanced functionality that we don't get from Gitlab (in many cases, even the enterprise version). Yet we are sticking with Gitlab, and making extensive customisations (mostly using the API) and workflows adapted to it.
Is our core business developing software for CI/CD/Issue tracking/source code management? Absolutely not. But is that critical to us? Absolutely. Because Gitlab is open source and because it has an extensive API, we can eek out efficiencies all over the place that aren't possible with closed source software.
So I think it's really important that people use the right frame of mind when evaluating "core business" or not. You don't have to be in the business of a selling particular type of software for it to be worth doing "in-house". You do have to be good at it though.
This is addressed in the article as "risk".
The best way to balance this is to prefer buying/renting Open Source software. That way, if your vendor changes direction, or their service level drops unacceptably, you can choose a new vendor without losing your integrations. Or even switch to a self-supported model.
And any improvements you need, you can contribute to the project so that others can take advantage of them and share the maintenance burden.
Also, try to use the product as off-the-shelf as possible. Little or no customisation. If people in your business are unable to work with a product without customisation consider firing them and replacing them with people who can before you get involved in customisations.
For me the dominating factor here is human resource pool. For whatever we build there are no experts on the system except the ones who already work for us. So aside from all the build, maintenance, etc. costs associated with build, I also can't find any employees on the market who are experts or even competent in the software.
If it's not part of your core business/core competency, and you're just reinventing the wheel so you don't have to pay a vendor, you have to weigh against the cost of the off-the-shelf solution NOT just the cost of developing it yourself, but developing it, maintaining it, documenting it, replacing your "irreplaceable engineer" when s/he quits, etc.
Sometimes, you don’t get a choice. In most large enterprises, it is almost a given that you’re going to buy instead of build, and that thing you bought better come with enterprise support.
Many SaaS have 10x'd their prices over the last few years and it is a significant risk for any tool that you are "renting".
- Gives JIRA a report card on them as a vendor to your business.
- Is largely ignored because my business is not Amazon scale with all the power that holds.
tldr; this only really works if you're one of the whales for a vendor.
You start the basic design of how it should work, maybe even based on that early feedback.
If you have no money, and you can build stuff, it doesn't matter if buying is the best advice.
Should you build or should you buy?
Neither. You should survive.
If you understand the role, this doesn't have to be a problem. Hence registry/registrar/delegate separation: the registry is not the delegate. the registrar is not the delegate, and has to meet the demands of systems boundaries implemented by the registry. The delegate has no software limitation and no need to own software because its a role divorced from software. Anything can be out or in source.
Or the CA/RA functional divide in a PKI. Better not to be an amateur implementing crypto, buy off the shelf because your role is to adjudicate what happens, not write systems.