Hacker News Reader

Comment by dudeinjapan | Hacker News Reader

Hi there, at TableCheck (www.tablecheck.com) we recently adopted Lightstep.

In a nutshell, running all these various components (Grafana, etc) is a royal pain in the neck. Even if `opstrace create` spawns them easily, the problem is running/maintaining them. We want someone to run these for us as a SaaS/PaaS and we're happy to pay them.

Re: your principles:

(1) The customer should always own their data --> we agree. However, we are happy for you to be a custodian of that data.

(2) We don’t want to be a storage vendor --> neither do we. We want storage to be someone else's problem. We're happy for you to use a cloud platform like AWS/GCP and charge us a 50% markup.

(3/4) Transparency, predictability of costs, open source --> all excellent.

View on HN

Jan-Philip from Opstrace here. This is lovely feedback!

> is a royal pain in the neck.

It's fun to see how different people put the same unpleasant experience into words in this thread. Thanks for adding your personal touch. Every time we hear something like that, we're re-assured that we're on the right track.

> Even if `opstrace create` spawns them easily, the problem is running/maintaining them

Yes. You're right. While we can be proud of our setup/installation process already, we know that there's so much more to it. We don't underestimate that. Maybe also see https://news.ycombinator.com/item?id=25998587, where I just commented on the robustness topic.

> However, we are happy for you to be a custodian of that data.

Great.

> We want storage to be someone else's problem.

I share that perspective. We, of course, are happy to let S3/GCS do the actual job.

> We're happy for you to use a cloud platform like AWS/GCP and charge us a 50% markup.

That's great to hear, and I hope you can be enthusiastic about the fact that our markup is _not_ going to be relative to storage volume. It's going to be independent of that.

> Transparency, predictability of costs, open source --> all excellent.

Thanks for sharing. That's incredibly motivating.

Keep an eye on us, and we'd love to hear from you!

This is an interesting perspective. Our current monitoring system charges by the amount of data ingested / stored so there is a perverse incentive to observe less men if we have detailed debug level logs available.

I'm glad I don't have to pay per-man to observe men :P.

Will definitely keep an eye on your service. Again, don't underestimate users willingness to pay for a PaaS product you make, even if you also do a dual PaaS/self-hosted option like Gitlab or MongoDB does, for example. We'd definitely prefer the PaaS, and that's where the big $$$ is made these days.

By the way, we are requiring that any vendor we choose in this area support OpenTelemetry as we've already instrumented our apps with it. Lightstep, Datadog, and others already supporting.

Makes sense! Thanks for the feedback.

And yeah OpenTelemetry is where lots of the industry is going and so are we. You can already do cool things with us and the collector but we plan on automating a lot and expand on that.

A little bit about that in this commment: https://news.ycombinator.com/item?id=25995424

Lightstep was bananas expensive and had several limitations that lead to us moving away from it. Hopefully it's easier to scrub PII from it these days.

Lightstep has recently changed their pricing, it's now the most cost effective option out there. You pay $100 per service with basically unlimited data ingestion, it's a fabulous deal. https://lightstep.com/pricing/

As for scrubbing PII, they are now supporting the OpenTelemetry tracing API which does this as standard. For query endpoints you will see something like "QUERY Users where name=? email=?", i.e. masking with "?" chars as you only care about the keys since those what determine your indexing or lack thereof. (This is handled in the OpenTelemetry application library/plugin level.)

As an aside, PII scrubbing should be done that whether or not you own the cluster, because even if you own the cluster you generally don't want your support staff seeing PII esp. as organization grows larger.

Heh, yes, the PII scrubbing need is very real. This is definitely a contributing factor to our data ownership commitment--just keep your data in your own account.

On the cost topic, last week we published a blog post analyzing the cost of running Opstrace on AWS (https://opstrace.com/blog/pulling-cost-curtain-back). (In fact, feel free to do a local repro to confirm our results.) As mentioned elsewhere here on HN, we are incentivized to provide total transparency in terms of what you spend on your cloud infrastructure. We haven’t compared ourselves to everyone, but feel confident that letting our customers pay S3 directly is the best deal possible.

@dudeinjapan, check out https://grafana.com/products/cloud/ The first line on the page says, "Your observability, managed as a service"

It might not fit your use case...but it might.

We used self-hosted Grafana until late 2020. We looked at Grafana cloud but ultimate chose a combination of NewRelic One for metric monitoring and Lightstep for request tracing. Those two were the "best-of-breed" for their respective areas with the best pricing for our use case.

Reply on news.ycombinator.com