Hacker News Reader

> Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs).

I loved Datadog 10 years ago when I joined a company that already used it where I never once had to think about pricing. It was at the top of my list when evaluating monitoring tools for my company last year, until I got to the costs. The pricing page itself made my head swim. I just couldn’t get behind subscribing to something with pricing that felt designed to be impossible to reason about, even if the software is best in class.

I’m a big fan of Datadog from multiple angles.

Their pricing setup is evil. Breaking out by SKUs and having 10+ SKUs is fine, trialing services with “spot” prices before committing to reserved capacity is also fine.

But (for some SKUs, at least) they make it really difficult to be confident that the reserved capacity you’re purchasing will cover your spot use cases. Then, they make you contact a sales rep to lower your reserved capacity.

It all feels designed to get you to pay the “spot” rate for as long as possible, and it’s not a good look.

I understand the pressures on their billing and sales teams that lead to these patterns, but they don’t align with their customers in the long term. I hope they clean up their act, because I agree they’re losing some set of customers over it.

Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs.

Then they have things that I wanted to try for a long time, but... support doesn't care? Repeated "would you like to use this? / very likely, can we try it out? / (silence)". I love their product, but they are so annoying to deal with at the billing level.

> Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs.

I, quite literally, was griping to my Datadog CSM about this exact thing last week. They'll email me and be, "Oh, you know you're logging volume this month put you into on-demand indexing rates, right?" and my answer is always, "No, because your monitoring platform makes it nearly impossible for me to monitor it correctly."

You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.

Maddening.

And why do you continue to deal with scum like this? You're ultimately going to pay it and business will carry on as usual for them.

> You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.

Are you referring to the `datadog.estimated_usage.logs.ingested_events` metric? It includes excluded events by default but you can get to your indexed volume by excluding excluded logs. `sum:datadog.estimated_usage.logs.ingested_events{datadog_index:*,datadog_is_excluded:false}.as_count()`

> Datadog makes it far too easy to misconfigure things and blow up costs

I'll give you a fun example. It's fresh in my mind because i just got reamed out about it this week.

In our last contract with DataDog, they convinced us to try out the CloudSIEM product, we put in a small $600/mo committment to it to try it out. Well, we never really set it up and it sat on autopilot for many months. We fell under our contract rate for it for almost a year.

Then last month we had some crazy stuff happen and we were spamming logs into DataDog for a variety of reasons. I knew I didn't want to pay for these billions of logs to be indexed, so I made an exclusion filter to keep them out of our log indexes so we didn't have a crazy bill for log indexing.

So our rep emailed me last week and said "Hey just a heads up you have $6,500 in on-demand costs for CloudSIEM, I hope that was expected". No, it was NOT expected. Turns out excluding logs from indexing does not exclude them from CloudSIEM. Fun fact, you will not find any documented way to exclude logs from CloudSIEM ingestion. It is technically possible, but only through their API and it isn't documented. Anyway, I didn't do or know this, so now i had $6,500 of on-demand costs plus $400-500 misc on-demand costs that I had to explain to the CTO.

I should mention my annual review/pay raise is also next week (I report to the CTO), so this will now be fresh in their mind for that experience.

That’s just the sort of hypothetical scenario that kept running through my head as I tried to find a way for us to use Datadog. I even particularly wanted to use the CloudSIEM product. Bummer.

Linear has a lot going for it. It doesn't support custom fields, so if that's a critical feature for you, I can see it falling short. In my experience though, custom fields just end up being a mess anytime a manager changes and decides to do things differently, things get moved around teams, etc.

- It's fast. It's wild that this is a selling point, but it's actually a huge deal. JIRA and so many other tools like it are as slow as molasses. Speed is honestly the biggest feature.

- It looks pretty. If your team is going to spend time there, this will end up affecting productivity.

- It has a decent degree of customization and an API. We've automated tickets moving across columns whenever something gets started, a PR is up for review, when a change is merged, when it's deployed to beta, and when it's deployed to prod. We've even built our own CLI tools for being able to action on Linear without leaving your shell.

- It has a lot of keyboard shortcuts for power users.

- It's well featured. You get teams, triaging, sprints (cycles), backlog, project management, custom views that are shareable, roadmaps, etc...

PagerDuty’s cheapest plan is $21 per user month

OpsGenie’s cheapest is $9 per user month but arbitrarily crippled, the plan anybody would want to use is $19 per user month

So instead of a factor of ten it’s ten percent cheaper. And i just kind of expect Atlassian to suck.

Datadog is ridiculously expensive and on several occasions I’ve run into problems where an obvious cause for an incident was hidden by bad behavior of datadog.

Heii On-Call is $32 per month total for your team — not per user. https://heiioncall.com/ (Full disclosure: part of the team building it)

How do you pronounce that?

“Hey”.

Looks super interesting, and that $3/month for hobbyists is just low enough to meet my budget for hobby services, but please, for on-call stuff, you gotta have alerts that make phone calls. Nothing else is going to wake me in the middle of the night. This is the #1 feature I expect from an on-call service - you're on-call because you will be called.

Thanks for the feedback!

We use iOS “Critical Alerts” and similar on Android that breaks through any Do-Not-Disturb settings. https://heiioncall.com/blog/better-alerting-for-heii-on-call... Would you be willing to give that a shot? It wakes me every time :)

(It’s configurable too; we have vibrate-only or silenced modes. Think old-school beeper.)

In the rare case that it doesn’t wake you, we have configurable escalation strategies to alert someone else on your team after a configurable number of minutes.

One of the things about phone calls is that they will continue to ring and ring until you actively decide to answer or ignore; it's not just a half-second "ping!" sound delivered once. This is also useful in meetings when everything else is closed up or silenced, to jar you out.

I usually do not respond immediately to phone notifications, which I can handle async. Phone calls are by definition sync.

Ah yeah, these are no ordinary phone notifications. :)

Heii On-Call will keep alerting you with these “Critical Alerts” until you’ve manually acknowledged. (Or until it escalates to a teammate and they acknowledge…)

And at least on my phone they sound nothing like normal phone notifications, which I personally always have on vibrate and/or DND anyway.

Give it a try and I think you’ll like it.

We are building a great and affordable incident escalation tool as well:

https://allquiet.app

With SMS, Phone Calls and Critical Alerts / DnD override.

We're 5 USD/user.

We try to build as close to our users as possible. Happy for any new try outs! :)

(I am co founder)

I just started building out on-call rotation scheduling to fit teams that already have an alerting solution and need simple automated scheduling. I’d love to get some feedback: https://majorpager.com

Grafana OnCall can be self hosted for free or you can pay $20 a month, and still always have the option to migrate to self hosting if you want to save money

We moved from Trello to Linear and it's been fantastic. I hope to never work at an organisation large enough for JIRA to be a good idea.

To be fair Linear does strike me as everything everyone always hoped Trello would be.

So if that's the upgrade path you're going down I'd expect it to be fantastic.

Newer (aka next gen aka Team-managed) Jira projects are pretty solid.

Do jira pages still take 30 seconds to load, and have all the interaction speed of cold molasses? Does it have nice keyboard shortcuts yet? Do I still need to perform an arcane ritual of setup to get the ticket statuses to be what I want?

Linear has been such a breath of fresh air, with such a solid desktop app (on Mac OS) that I don’t ever want to go back. Stuff happens instantly, the layout and semantics are an excellent “90% good enough” that I would happily relegate jira to only the most enterprise of enterprise projects.

At one of the bigger companies I was at we had an on-prem JIRA in the same office building and it was still so slow that I would often forget why I was loading that page

trigger warning please on the Jira stuff

Linear is making (fairly) good on the promises of local-first software. As opposed to "every click is a round trip to the server" software.

No, Jira loading is relatively OK and on par with other SPAs. It's got a CTRL+SHIFT+P style actions menu for tickets which helps cut down on point and click pain (especially for linking issues etc). Setting up statues and workflows and how they map to a board is relatively straightforward.

There are lots of things where Jira falls short, but the pain points on an under-resourced self hosted instance of ten years ago are nothing like the ones you'll find on Jira cloud today.

Does Jira still have multiple flavours of markdown for different fields and editors? Last I used it, it used a different flavour for creating and editing a ticket. Also another flavour for bitbucket. None of these were compatible and it would convert between them in the backend but I was left confused every time when I would have to switch formatting styles

I remember that from a while back, and getting annoyed - it doesn't appear to be something that annoys me at the moment so it might have been fixed, but on reflection I tend to just use the default rich text editor now.

It takes markdownish input but converts it to rich text as you type - so asterisk-space starts a bullet point list, etc.

I actually can't remember if it has a dedicated markdown mode anymore; the rich text editing supports the usual shortcuts that mean I tend to stick with it.

DatDog is a freaking beast. NY wife works in workday (a huge employee management system) and they have a very large number of tutorials, videos, "working hours" and other tools to ensure their customers are making the best use of it.

Datadog on the other side... their "DD University" is a shame and we as paying customers are overwhelmed and with no real guidance. DD should assign some time for integration for new customers, even if it is proportional to what you pay annually. (I think I pay around 6000 usd annually.

In terms of Datadog - the per host pricing on infrastructure in a k8/microservices world is perhaps the most egregious of pricing models across all datadog services. Triply true if you use spot instances for short lived workloads.

For folks running k8 at any sort of scale, I generally recommend aggregating metrics BEFORE sending them to datadog, either on a per deployment or per cluster level. Individual host metrics tend to also matter less once you have a large fleet.

You can use opensource tools like veneur (https://github.com/stripe/veneur) to do this. And if you don't want to set this up yourself, third party services like Nimbus (https://nimbus.dev/) can do this for you automatically (note that this is currently a preview feature). Disclaimer also that I'm the founder of Nimbus (we help companies cut datadog costs by over 60%) and have a dog in this fight.

I mostly agreed with OP's article, but you basically nailed all of the points of disagreement I did have.

Jira: Its overhyped and overpriced. Most HATE jira. I guess I don't care enough. I've never met a ticket system that I loved. Jira is fine. Its overly complex sure. But once you set it up, you don't need to change it very often. I don't love it, I don't hate it. No one ever got fired for choosing Jira, so it gets chosen. Welcome to the tech industry.

Terraform Cloud: The gains for Terraform Cloud are minimal. We just use Gitlab for running Terraform pipelines and have a super nice custom solution that we enjoy. It wasn't that hard to do either. We maintain state files remotely in S3 with versioning for the rare cases when we need to restore a foobar'd statefile. Honestly I like having Terraform pipelines in the same place as the code and pipelines for other things.

GitHub Actions: Yeah switch to GitLab. I used to like Github Actions until I moved to a company with Gitlab and it is best in class, full stop. I could rave about Gitlab for hours. I will evangelize for Gitlab anywhere I go that is using anything else.

DataDog: As mentioned, DataDog is the best monitoring and observability solution out there. The only reason NOT to use it is the cost. It is absurdly expensive. Yes, truly expensive. I really hate how expensive it is. But luckily I work somewhere that lets us have it and its amazing.

Pagerduty: Agree, switch to OpsGenie. Opsgenie is considerably cheaper and does all the pager stuff of Pager duty. All the stuff that PagerDuty tries to tack on top to justify its cost is stuff you don't need. OpsGenie does all the stuff you need. Its fine. Similar to Jira, its not something anyone wants anyway. No ones going to love it, no one loves being on call. So just save money with OpsGenie. If you're going to fight for the "brand name" of something, fight for DataDog instead, not a cooler pager system.

I'm right there with you on Jira. The haters are wrong - it's a decent enough ticket system, no worse than anything else I've used. You can definitely torture Jira into something horrible, but that's not Jira's fault. Bad managers will ruin any ticket system if they have the customization tools to do so.

Using Jira feels like using IBM enterprise web software from 2005, and I am simply not going to make my teams put up with that amount of inanity.

We switched to JIRA around 2005 away from IBM enterprise web software, because it was a breath of fresh air.

So on the standard tech hype cycle, that sounds about right.

Found the person who never used Lotus Notes haha.

I was blown away when I found out a couple years ago that there were major corporations still using that as their primary communication platform.

Surely has improved in the last 20+ years? :hope:

Yeah, usually Jira hate is really convoluted company process hate. Of course the Jira software isn't perfect, but it's fine. Jira's strength and weakness is it's flexibility.

> I generally recommend Terraform Cloud

I'll be dead in the ground before I use TFC. 10 cents per resource per month my ass. We have around 100k~ resources at an early-stage startup I'm at, our AWS bill is $50~/mo and TFC wants to charge me $10k/mo for that? We can hire a senior dev to maintain an in-house tool full time for that much.

Agreed on PagerDuty It doesn't really do a lot, administrating it is fairly finicky, and most shops barely use half the functionality it has anyway.

To me its whole schedule interface is atrocious for its price, given from an SRE/dev perspective, that's literally its purpose - scheduled escalations.

Why gitlab? GitHub actions are a mess but gitlab online's ci cd is not much better at all, and for self hosted it opens a whole different can of worms. At least with GitHub actions you have a plugin ecosystem that makes the super janky underlying platform a bit more bearable.

I've found GitLab CI's "DAG of jobs" model has made maintenance and, crucially for us, optimisation relatively easy. Then I look into GitHub Actions and... where are the abstraction tools? How do I cache just part of my "workflow"? Plugins be damned. GitLab CI is so good that I'm willing to overlook vendor lock-in and YAML, and use it for our GitHub project even without proper integration. (Frankly the rest of GitLab seems to always be a couple features ahead, but no-one's willing to migrate.)

Mhmm that's actually a good point!! I didn't realize that I couldn't do that with GitHub, I never really used partial caching. I just had a lot (a looot) of issues with our kubernetes runner (which I even made sure to be as close to the vanilla docs example as possible). I guess the grass is always greener on the other side :)

> Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs).

Datadog's cheapest pricing is $15/host/month. I believe that is based on the largest sustained peak usage you have.

We run spot instances on AWS for machine learning workflows. A lot of them if we're training and none otherwise. Usually we're using zero. Using DataDog at it's lowest price would basically double the cost of those instances.

After their ridiculous outage, I wouldn’t touch OpsGenie with a 10ft pole.

This may be a noob question - but why not use Github Projects instead of Linear or Jita?

You're staying within an ecosystem you know and it seems to offer almost all of the necessary functionality

That would totally be my preference if business users didn't want access.

Getting them to use Github/Gitlab is an argument I've never won. Typically it goes the other way and I end up needing to maintain a Monday or Airtable instance in addition to my ticketing system.

Interesting. Atlassian also just launched an integration with OpsGenie. I have the same opinion of JIRA. I've tried many competitors (not Linear so far) and regretted it every time.

> Atlassian also just launched an integration with OpsGenie.

Given Atlassian bought OpsGenie in 2018, this either somewhere between quite late and unsurprising.

Two different measurements (time and Atlassian development processes) that are orthogonal.

Anything Atlassian does is mostly quite late and its integration story is so pathetic that it's unsurprising.

Try to have a bitbucket pipeline that pushes to confluence. Seems like a basic integration to have, after all, Confluence has an API (well, actually it has 3 different ones) so surely Atlassian would make a basic thing like "publish a wiki page" a thing you get out of the box.

Nope.

Oh, I am no great fan. Plus I have a nascent blog post on the subject of 'can you believe ...?' items around this subject.

I suppose it comes back to the comparative priorities (as evaluated by recurrent revenue) of ticking rfq boxes vs solving actual problems.

I'm not sure they just launched anything. OpsGenie has been an Atlassian product for 5 or more years now. I've been using it for 3-4 myself and its been integrated with Jira the whole time.

In fact, OpsGenie has mostly been on Auto-pilot for a few years now.