Hacker News Reader

Comment by Klathmon | Hacker News Reader

Has anyone here used their self hosted version?

I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?

View on HN

I am using it, but it is not fun to use. It spawns 28 Docker containers (yes, that's the actual number), including things like nginx, postgres, memcached, redis, kafka, zookeeper, clickhouse. I've tried upgrading it once and it failed miserably, after which I simply started from scratch.

I would like to use their hosted version, or a more stable paid on-prem version. Pricing is not the issue for me, but I don't want to introduce another third-party service and have to include that in compliance documentation for our customers. That is the reason I still self-host many things to be honest.

Would you mind sharing which other saas products you are so posting?

Well the bare minimum that you have to tell your customers is your cloud provider / hosting provider, then you sometimes have a separate provider for Object Storage (S3 or Backblaze), maybe another one for email delivery (Mailgun). You might have an invoicing/CRM portal (Stripe), SCM (Github), Chat (Slack), your own email setup (Gmail). And that is not even talking about any client-side tracking, things like Google Analytics etc. So it really makes sense to cut down the number of third parties, especially if there is any scenario in which they would be handling PII, and especially if you're selling to customers in heavily regulated industries.

Their self-hosted version works well with the caveat that the upgrade process can be a pain and can be buggy. I used their docker-compose open source version which involved downtime to spin down the stack, then spin up the new stack. Considering that there's no enterprise support for on-prem, its nice that it works so well. If you're using their apm/observability you will need to keep an eye out on disk space and memory/cpu after turning it on, we ran into resources issues due to the amount of data we started sending. Otherwise it's rock solid and cost-efficient, we were paying 500ish bucks a month for infra and 1-2 days of dev time for something sentry.io quoted 50,000K/month for. I imagine the kubernetes installer is much less hands on in terms of upgrades. Would use again without any hesitation.

We used to run it self hosted, but the architecture is rather complex, so we ended up switching to their SaaS offering. I tried advocating [0] for a leaner architecture for simpler setups + to run integration tests against... but it wasn't met with much enthusiasm.

I imagine 99% of installations (and certainly CI pipelines) would be fine with their wsgi webserver and sqlite instead of Clickhouse, Relay, memcached, nginx, Postgres, Kafka and a host of other services. I wanted to take a stab at this myself, but given the complexity of the system, uncertainty of being able to merge it back, and, last but not least, their license, I decided against it.

[0]: https://github.com/getsentry/sentry/issues/32794

This has been the experience at my company as well. The previous version of Sentry Server was okay-ish to self-host, but the newest version requires setting up some more services that our infra unwilling to setup/maintain as they are different from the tech stack that our devs use in the company. We ended up with SaaS too

I've being running their self-hosted version, and it's kind of OK.

Nowhere near as nice as, say, GitLab's Ubuntu packages which you can simply apt-get install, unattended automatic (security) updates are available and work well, and it's been mostly a set-up-and-forget type of thing. For GitLab, backups and most ops procedures are well documented.

Sentry, on the other hand, requires you to run a script to upgrade with their docker-compose thingies. You must remember to upgrade.

Self-hosted Sentry runs on docker, which on Ubuntu is a bit of a security shitshow what with docker side-stepping UFW and all that.

Recently Sentry tripled (seems like) the number of containers they spin up, and it seems much heavier than a couple of years ago. You can still get away with 8G RAM, so it's not really that bad, to be honest.

Backing up Sentry was simple enough. However, a lot of questions are answered in the style "if you want to know more, buy the enterprise version". But, I've backed up and restored a Sentry system and it worked out OK, so I guess we're fine. Unless they change something, and then we're hosed without realizing it.

So, Sentry's self-hosted version is OK, but clearly not their number one priority. I have no idea what Sentry would look like with a self-hosted distributed/HA setup. Our data protection policies would absolutely prohibit us from using their cloud services, so maybe I'll have find out, but I'm not looking forward to that.

I've been running it for almost a year, using docker-compose. It seems rather unstable for me. Every week or so, Sentry will just stop handling incoming events, and the request queues just keep growing. And if you try to upload symbols in this state, it will pause forever when trying to process them. So I've got a script[0] I can run that will unbreak it, but I don't fully understand what it actually does (I cobbled that script together based on a GitHub issue[1] that described the same problem). The Sentry architecture is complicated and not trivial to debug.

When it works, it's pretty good. I'm using it with sentry-native on the application side, which uses Crashpad to capture stack traces of native binaries (x86/ARM, Windows/Mac/Linux, whatever). It often doesn't deduplicate events properly, and the stack trace qualities vary dramatically by platform. Sometimes the stack traces it provides are total nonsense, but it does allow downloading the minidump files, so I can dig at them in Visual Studio and see what's really going on. I have discovered and solved many real bugs using it, so I've put up with the frustrating stability issues.

[0] https://gist.github.com/tycho/4279ce2ca47b293a85696695968263...

[1] https://github.com/getsentry/self-hosted/issues/478

I do, it works great with their docker-compose install - processing a few billion events per day here. Installation and subsequent minor/major upgrades without any flaw so far, plus they have extensive documentation for the self hosted version which is to note.

I'm using it for a small project and it works fine. Self hosting is documented here: https://develop.sentry.dev/self-hosted/

Upgrading is a little bit tricky: in my (limited) experience the upgrading procedure for the self-hosted version sometimes fails and requires to take the application offline to perform the upgrade (but it is very possible that I may be doing something wrong).

It is really an invaluable tool.

It works if you can live with 28 containers (messing up "docker container ls" output that I always have to "grep -v sentry") literally eating 2GB+ memory just to have it launched.

Ideally, things run as a single binary with bring-your-own-database but that's not the case here.

I wish I could use my already installed ClickHouse to save some memory but the version constraint and some very complicated docker compose setup on their end wasn't worth trying.

But I still love the product itself, so I'll let it chew my memory.

Upgrade has been fine as the manual says, you pull their git repo, run the installer and worked fine on all my past several occasions.

I run it on k8s for a medium sized company, as running via compose didn't seem to scale well for us. I even upgraded from 9.11 which ran a handful of containers to the most recent version which runs 15+ containers.

You get used to the quirks of running it and then it just works, but it did take some time polishing the health checks etc to get it HA and scalable.

None of the helm charts I tried actually worked or provided HA.

Pardon my ignorance but how on earth does it need 15+ containers?

Is that just a scaling thing for your company? Or are their 15 discrete services under the hood to drive it?

It really does run a ton of different services under the hood. It's designed for their type of scale, a small-scale deployment would probably work OK without Zookeeper and similar.

https://develop.sentry.dev/architecture/

We run the on-prem version, but looking to transition to SaaS soon. While one can point at the current outage and say "good thing we run it ourselves!" it is a chore to keep updated and running smoothly.

That said, it runs untouched for long stretches of time no problem.

The SaaS offering came later. I used to run it on-prem a few years ago before the SaaS showed up and it was very low maintenance. More of a fire and forget situation once it was set up.

A previous company did. Reliability seemed fine; it was a way to ensure no PHI/PII was accidentally sent to a 3rd party as part of an error report.

Not just PII, but access tokens to 3rd party systems in case of integrations and whatnot.

A Sentry data leak could include valid access tokens to, say, all data on a customer's Docusign account. We try to make sure data like that is scrubbed before sending to Sentry… but mistakes can happen. This is simply not a risk we are willing to take.

Thus, we self-host Sentry in our own secure infrastructure (we are a SaaS provider ourselves), and accept all the maintenance burden that it entails.

I looked into it at one point and found it required a lot of underlying services and seemed really complex. Primarily I was worried about the maintenance costs moving forward with something like that.

There is an open-sourced option called GlitchTip (https://glitchtip.com/) that is much simpler to self-host. I believe they forked the Sentry repo before the license change. It's not quite as feature rich but is a pretty great alternative.

Reply on news.ycombinator.com