Hacker News Reader

I've lived that life with Datadog. I was the dev who was the most into logging so I got voluntold to be in charge of the Datadog rollout. Told everyone it was not a good idea and presented cost estimates to why, we're better off just keeping our own logs and using one of many open source projects that let you pull in logs and analyze them. But all the cool kids were on Datadog, so onto it we went.

What killed us was the combo of the desire to really aggressive log on the staging-master and release-candidate environments (five envs total) and keep the logs for more than 7 days so the QA team could compare the logs from the current release candidate to the previous release candidates. Between the volume of having all the services set at the info level and the desire to keep it more than 7 days, Datadog was >30k a month.

Which at least finally got me permission to set up an open source log manager and switch everyone over to that. Once the initial panic over the Datadog bill died down.

I haven't used DD logs personally but from what I've been told if you send the majority of your logs straight to archive to an S3 bucket and only rehydrate the logs when you need to access them their pricing is pretty reasonable. Because you're storing the logs in your S3 bucket you can keep them however long you want and rehydrate them when you need them (say 30 days later).

The bill will still be in the low thousands but paying 30-40k per year for a logging solution is probably worth the cost if it means you don't have to maintain it. Having logs and metrics (and the APM if you're using it) all in one place is _really_ nice.

You have to survive as company long enough where paying 30k/month sounds reasonable for logging… esp if your revenue is nowhere near it, and are not VC funded… and you will always have to maintain your integrations at the very least.

If you're using log rehydration with DD it's hard to hit the $30k/mo fee. Before I left my last company we were evaluating DD logging and it came out to ~$5-6k/mo for about 25B logs per month.

When you start you'll likely pay a few hundred dollars per month tops. Your personal time alone is probably worth more than that.

Before I left my last company, we were spending about a couple hundred dollars per month just on media storage/bandwidth alone for serving academic papers/docs/data (as well as our logs)… in top 5k in alexa, top 100 in indonesia with about maybe 60-70 customers paying 150-200 dollars per year coming from mostly EM countries… the customers can barely afford/justify spending what we were offering over using some open source alternatives, and there would be no way in hell we'd spend that on just logging.

I at least helped 10x the customers and cut costs from when I joined to before I left (stayed a little over a year)… but man, talk about constraints…

Whats the point of even having it in data dog? Typically we pay a ton for services like Newrelic for monitoring, the APM, and insights. Like you, we have tons of logs, so I wouldn't want to pay Newrelic or Newrelic-similar prices for a service you can just use ELK for yourself much cheaper. So basically I don't really see why you signed on to data dog at all

I love this comment because it articulates a dynamic that I think is often overlooked much better than I could. I might even go a step farther and suggest that anything that is core to your business flywheel should be built and not bought (in general).

Mostly because you don't want pricing to act as a disincentive to exercising the flywheel.

Then time is the disincentive.

Ah, but if it is core to your business flywheel that is precisely where you should be spending the most time.

Even for built systems the cost of ownership and maintenance will grow as the system grows. Take the logging example above and say you're using the Elastic Stack. Things are going pretty well and usage is increasing. Over time your system grows and logging is now present in all of your applications. You've seen your logging rate go from 100 logs/s, to 1,000 logs/s, to 10,000 logs/s and now are encroaching the 100,000 logs/s rate. Over time the amount it cost to host your ES cluster has surely gone up. You probably went from a single node cluster when developing the solution, to a multi-node cluster, to now a hot-warm-cold cluster with several more nodes than you originally had. You also have a dedicated team making sure your in-house logging solution is working and stable. The cost of ownership and maintenance of this system is real. Not only do you need to pay for the huge clusters you'll likely have a dedicated team supporting it. Most people I've talked to in the industry don't like maintaining their in-house logging solutions and would gladly pay for one if it wasn't for the fact that it was too expensive to switch then. Because you need specialized people that know this logging solution it's also hard to hire for this role when the person currently maintaining this system eventually leaves (because he doesn't want to do this anymore). At the end of the day it's cheaper to pay for a logging solution even if it's in the hundreds of thousands of dollars a year than it is to build your own.

> “anything that is core to your business flywheel”

The mechanisms by which your product is marketed, sold and distributed are core to your business flywheel. For example, if you’re a B2B Saas, distribution is often way more important than the product itself.

So by the “core to our flywheel” standard you need to build your own email marketing software, your own version of salesforce CRM, your own analytics tool for triggering marketing funnels, etc.

That’s just hilarious. As someone who runs a business, I wouldn’t spend a single minute of engineering resources on anything that we don’t sell directly to a customer.

I’d go even further and say if you have less than 5000 employees, ALWAYS buy. Never build.

Hmmm, this is an interesting take, but I think you are expanding the scope of "core to _your_ business" a bit much. Product marketing, sales, and distribution are shared between _all_ businesses so unless you are a marketing, sales, or distribution company I (personally) wouldn't consider them part of the core business flywheel.

For example, let's pretend I have a fictional company that sells a web server. Marketing, sales, and distribution are all important parts of my business, but my "Core" business is the web server software. If I am looking to build reinforcing feedback loops that give me an advantage over time (flywheels) it's my opinion that you shouldn't be looking to build them (at least initially) in marketing, sales, or distribution.

We've often gone down the route of 'modify an open source project' for this reason. For the price of a developer, we get the product in perpetuity and don't have these exact limitations you specified of headcount or worrying about usage. For almost everything out there, you can find a version of it that's been made available with source code. Granted, unless you are essentially a software company this is probably difficult to maintain, given the additional level of expertise required.

Another option we've found works is buy, to just get started, and invest moving forward in our own project. Getting something off the ground quickly tends to help us understand better the requirements.

Honestly, I'm fan of the open source self hosted (especially if only some of the enterprise features are paid). We set it up, configure it, evaluate it and buy the enterprise license (it's usually a per-box license).

Very interesting perspective. Ran into the same issues where Salesforce was charging us by the number of columns we had that ended up forcing the team to overload a column with a JSON object so we could get it to work without overpaying. Integration and maintenance turned out it be a downright nightmare. Would be interested to hear various pricing models to understand the pros/cons for each especially in the logging/analytics space. Does per storage, per event, per user make sense in certain cases?

For things that are directly generating revenue, it makes sense to have per-X pricing because it aligns your vendors' interests with your own. In that case, you want to incentivize the vendor to maximize your usage, because that usage is making you money.

I have lived through this same thing with Salesforce.

A brutal experience.

I hate to break it to ya, but if you do cheats like this when you use reliable off the shelf COTS solutions, imagine the shortcuts they're going to do if/when they try it do this themselves.

Salesforce isn't cheap, but its not exactly expensive either. However, with this short sighted shortcut you did, you certainly cost your company big. That data is useless. If you upgrade, transition, migrate you now have to spend employee time fixing what was a fixed cost.

Human time is expensive. But what is more expensive is putting artificial costs where they don't need to be.

Completely agree. Btw, the abov example wasn’t my company. The CEO just didn’t want to spend. He was also a huge proponent of building everything in-house so we ended up tons of customized software that nobody knew how it ran when people started to leave. The company’s growth eventually stalled and was acquired. Shame because we had some brilliant engineers.

I’ve seen this repeatedly over my career. At the large organizations I’ve worked at (.edu, .gov), unless you have a large general tools budget it’s surprisingly hard to beat the open-source tools - simply running a procurement takes months and might require arguments with non-technical staff who don’t understand why certain options are not truly equivalent even if the salesperson swears they are. This is how you end up spending 7+ figures on enterprise tools while the technical staff are all using Prometheus, Grafana, and ELK because they actually work and are available to the entire group.

On the “clearly crazy to invest in building your own” front I think it’s useful to add the nuance about how much of that time is purely related to the product and how much is the tuning, integration, understanding the data, etc. which every product requires. I think we’re prone to underestimate the latter and have something like Splunk save less time and cost more than expected because the commodity part it optimized for wasn’t as much of the whole as anticipated.

Once we moved a database to cloud, because we were told it would be cheaper, since we would only pay for the space we actually used.

It also meant that whenever the data size increased, however slightly, the invoice also increased, and managers started investigating whether this increase of data size was truly necessary. Sometimes even when then data size didn't change, managers kept asking whether it could be further decreased.

Making the billing proportional to X means that X will become a political topic. Choose your X carefully!

"Making the billing proportional to X means that X will become a political topic. Choose your X carefully!"

That's such a great way to put it.

Memories of developers not logging stuff on datadog not to go over limit.

We ended up running our own ELK stack and it was much better.

ELK is killing it. Especially with their apm server.

> This can have a subtle effect on your culture. Engineers who don't want to get into a budgeting conversation will end up avoiding using key tools, and this can cost you a lot of money in terms of invisible lost productivity.

For exactly this reason I wanted to build a developer screening tool from scratch [1]. In all the companies I worked for, I never got the budget to pay for one of the available tools in the market (most have "contact us pricing"). The result is that most teams I've met spend a lot of time manually evaluating coding challenges. This is a waste of time both for the company and the candidate. A custom screening tool allows the company to calibrate it according to the perceived average candidate level, and avoid those algorithm questions that are useless for most small to medium companies.

In my previous company it was used in a few hiring processes and it worked fine - both for back-end and front-end development. It is not open source yet because your need some basic Docker and Rails skills to use it - even though the screening test can be in any language(s) you want (that runs on Docker). If it matches your experience and you want to try it out, please contact me.

[1] https://github.com/skilltest/skilltest

I posted an expanded version of this comment on my blog, with a suggestion that maybe there are some internal process improvements that could help address the problem: https://simonwillison.net/2020/Dec/13/build-vs-buy/

"Thinking about this more, I realize that this isn’t a technology problem: it’s a process and culture problem. So there should be a process and cultural solution.

One thing that might work would be to explicitly consider this issue in the vendor selection conversations, then document it once the new tool has been implemented.

A company-wide document listing these tools, with clear guidance as to when it’s appropriate to increase capacity/spend and a documented owner (individual or team) plus contact details could really help overcome the standard engineer’s resistance to having conversations about budget."

> I don't have a solution to this.

You basically just articulated the solution. Spend the money to give more of your team access to the analytics tool, using the argument made above.

It will still be infinitely cheaper (by like 100X) than building anything.

And the more you spend with a vendor, the more they’ll be willing to completely customize their product for you.

That's an easy answer if you control the budget.

If you're an engineer in a larger software organization, you probably don't have that spending authority. Which means you have to spend time convincing the people with that authority to spend that money.

That's enough of a friction point that many engineers won't bother - which is why I talk about the invisible cultural damage that this problem causes.

You and I know that it will be cheaper. The challenge is getting organizational buy-in.

Certainly a problem, but the time you’d need to spend convincing higher ups is still trivial compared to the time you’d need to spend to build, maintain, and update the tool.

>the time you’d need to spend convincing higher ups is still trivial

From this sentence alone I'm not sure how much time you spent working for a large company.

It's not just that you have to convince your manager, depending on how much is to be spent they have to convince theirs. Often creating very long decision paths.

And it's not only the time you spent but sometimes also the delay incurred that messes with this approach. Often enough even with buy in you get the answer that it can be planned for next year's budget.

One year delay later, you might get the approval, but it may be too late.

And if the company is large enough often they've already built a subpar internal solution that everyone is convinced you should use. Since it's "already working".

> Certainly a problem, but the time you’d need to spend convincing higher ups is still trivial compared to the time you’d need to spend to build, maintain, and update the tool.

The amount of bureaucracy at large companies cannot be overstated, especially when technology is a cost or auxiliary to the core business.

I’ve run into this twice in my 24 year (!! good god, am I that old??) career.

The worst was during a several month consulting gig for an investment bank, and it was extremely painful.

Every change to the dev and uat environments required sign-off and approval, from sys admins who were in a different timezone.

If the changes had cost implications, it would have been far worse.

And what if the higher ups say no?

In larger organizations budgets are complicated things. Your higher up with spending authority may agree with you, but they've hit the budget for their group and so your request for more logging is competing with a request to pay for a new compliance auditing tools or upgraded server capacity for machine learning models.

I'm certainly not arguing build over buy here - but this challenge is a genuine issue which I don't think gets enough consideration.

But building can be a lot more interesting and at the end of the day you'll get the same salary (unless you get enough equity to care about it).

So true! If the tool can’t be used by everyone on your team, then it is unlikely to be used.

I’ve experienced this many times.

You can still run into that budgeting conversation when you build, but this time it's with an internal team.