Sentry 8 is here
blog.getsentry.com
blog.getsentry.com
We're hoping to do a sort-of AMA in the future about rebuilding Sentry (and the open source approach), but if anyone has any questions about the new version I'm happy to answer them.
The big problem I have with all these services is that you need set up each application individually: Each one is a silo, with its own notification settings, API key, and so on. We have dozens of apps (several products + lots of microservices), so this is just unacceptable. I looked into whether I could use a single API key with Rollbar or Airbrake, and simply tag each event with the name of the sender, but it turns out this is not feasible. They don't even have APIs that let you programmatically configure apps remotely.
How does Sentry compare?
(Many other SaaS services suffer from the same issue; for example, Semaphore, which we use for CI, requires a build script for each project. We worked around this by letting the build script be a single curl command that pipes a remote script to bash, shared across all projects. But hardly ideal.)
Even with the limitations, we have many people with hundreds of projects, and the general problems we see are more around notification settings than anything else. The usual suspect is that people are getting too many emails when they either want none, or they want digests.
We are giving this a lot of thought as part of our upcoming infrastructure pass. From my point of view it'd make a lot of sense if you could just have a massive stream of events and sort through them however you want. That causes a lot of challenges so it may not be feasible.
Either way its a problem we're aware of and looking to address. The simplest solution from our point of view is to allow you to configure some kind of organization-wide defaults. That won't resolve the API key constraints, but we don't feel that specifically is a pain point.
For example, our company uses Slack. All notifications go to Slack. There's no need for per-project settings because it's all just one room. At some point we will probably want some projects to go to other rooms, but that's a coarse-grained exception, not something that warrants a per-project setting.
Also, if you need per-person notification settings, then surely that's part of the user's account/profile, not the project. That would be just a single dashboard page with (say) on/off sliders for each project.
In my mind, the data model used by exception trackers (including, it seems, Sentry) is wrong. There needs to be just one stream of events, tagged by things like application name that you can filter on. How those events map to things like notifications and aggregations and rules and source files and teams and so on is completely orthogonal to the stream. Those things then apply to a subset that you can modify dynamically without having to reconfigure the sender.
Hands down the best designed operations product I've used.
I'm not sure when you last looked at our APIs, but we do now have methods for creating and managing projects (though not yet notification settings): https://rollbar.com/docs/api/provisioning/
Your logging service seems overpriced, by the way. I'd love to have a central, cloud-based log viewer, but our system is producing many, many gigs worth per day. At the moment our expenditures are about $80/mo for a small rsyslog box, but with your system we'd be paying thousands. For that price I can whip up a bunch of Graylog boxes and get approximately what you're offering, at the expense of having to admin ElasticSearch myself, which we're already doing anyway.
We do some unique things like when looking at error show you all the log statements related to the same transaction.
Our pricing is similar to our competitors. But there are self hosted open source options as well, like you mentioned.
A lot of our customers like our product because we combine monitoring, errors, logs, metrics and even apm on one easy to use platform. It's very unique.
A few months ago the top-right dashboard lost its total count and I had to rely on hovering over the graph bars to see what the numbers looked like. I generate a hundred exceptions in a day (on a good day), and I like to use that count to gauge if anything is wrong. A bad day can generate hundreds of exceptions and I like to have an easy way to spot those days.
The dashboard view kind of gives that, but there is no single number of exceptions that I can use. And I greatly miss being able to have that count and graph on the same list of exceptions.
Can you bring back that 1 day graph, complete with a total exception count?
https://github.com/getsentry/sentry/issues/2131
(There's a few other related issues to this, but I think this addresses that core problem)
We're also fortunate to call many of the biggest names in Silicon Valley (as well as outside) our customers.
What you'll quickly find is anything that looks like Sentry, acts a lot like Sentry, and usually was inspired by Sentry. What you'll also discover is that even with this in mind, they are far from the same level of maturity. For example, we've spent a lot of time tuning our grouping algorithms, expanding support for power users, and generally solving problems that aren't obvious until you've invested a lot of time into using the tool.
This is a long winded way of me saying that Sentry is the best in the business.
Several of the new features in Sentry (like release management / deploy tracking) actually look a lot like things that have been in Rollbar for a long while, though I expect that is the result of two smart teams attacking the same problem, rather than one inspiring the other.
I'm wondering what the reasoning is behind suggesting I upload source artifacts to you as part of a release. The blog post even specifically calls out Go, which is what the majority of our application is written in. So from my understanding, uploading artifacts would allow the Sentry UI to show me exactly where we did a Sentry Dump?
On the same topic, our current deploy flow makes it a bit hard to upload specific code changes per release. It would be great if Sentry could get the code from Github directly (assuming that I login and give you access to the repo)
Thanks for Sentry, we use it daily and would have a much harder time tracing errors and monitoring overall application health without it.
For Go, you ultimately compile your application into a binary. This binary doesn't have the source code, but it does retain references to the filenames and line number in the binary's metadata. What this means is, if you send along a stack trace to us, we can tell you that information, but we can't show you what the actual line of code is, or what the surrounding code is. By uploading the sources to us explicitly, then baking in the release version into your compiled binary, we can map those up and show you a stack trace that includes source code. This is relatively the same idea with any modern JavaScript. Typically a project is compiled down into a minified version, or run through a transpiler, etc, and the original sources aren't accessible to us. So it's then necessary to upload to us so we can map things up.
> On the same topic, our current deploy flow makes it a bit hard to upload specific code changes per release. It would be great if Sentry could get the code from Github directly (assuming that I login and give you access to the repo)
We have plans for VCS integrations in the future (GitHub included) that would likely work with releases so we can just use the working directory at a current sha.
So right now when we have a need to send a stacktrace (error or panic) we use Raven-Go's NewStacktrace to create a stacktrace and attach it to the packet we send up. In the UI I see the exception with surrounding lines.
What makes releases different from what we already do? Or, do they do the same thing, but the difference is crafting the stack trace in our code vs just upload the code and letting you figure it out?
Yeah, so raven-go will check for the source files on disk first. You must keep your source files along with the binary. :)
There really should be a requirement that SaaS services supported some kind of load/dump configuration API, where you can deal with raw config. The admin UI should be completely optional.
Most SaaS vendors don't have APIs that cover all parts of the service. Some do have decent APIs -- for example, we store our DNS records in files and use Gandi's APIs to update our zones, using a custom script (with some nice diffing capabilities), and we do some automatic configuring of Mailgun -- but APIs are always stateful and never declarative, which means that a client will always have to issue half a dozen calls to diff and converge its state. It's painful.
We'll give some thought into this, as it's not too far from being a possibility.
If take only our data (ignoring anyone who isnt actively reporting to us), It's a fairly close 50% split of on-premise companies to hosted customers.
That said, our estimate is its more around 30% of users run off of getsentry.com, and the other chunk runs on-premise.
For other things we're using select2. At one point we had tried to switch to selectize but we found limitations in its API and frankly didn't want to spend the time to find ways to work around them.
That said feel free to throw your hat into the ring:
We do have some support for Java, but its unfortunately heavily tied into Logging right now. Our short term goal is to at the very least ensure we have proper documentation (and support) for getting things up and running in Clojure and Scala.
Happy to have been using Sentry for a long time now. Excited for the new release.
Will Sentry ever be useful for activities such as tracking occurrences of an event?
https://github.com/getsentry/sentry/milestones/8.0%20Release
Regarding future capabilities of Sentry, we're not quite ready to talk about most of it publicly yet, but we'll making some drastic improvements in the very near future.
"Soon", but you can install it from the github master already. Since we generally deploy right out of master the upgrades are fine to and from there as well.
and the website has two red bars for these warning, but the reporting mechanism seems work well, and there are no warning/error records in logs.
Do these warnings have serious problem? how can I fix these warnings? (I've totally removed old sentry files in /usr/local/lib/python2.7/site-packages/sentry* and made a fresh installation)
Thanks!
That said we also support a lot of advanced scenarios:
- Embedded/shipped code (i.e. PhoneGap, React-Native)
- Minified code via sourcemaps (use webpack? it "just works")
- Private source code (sourcemaps uploadable via artifacts, authentication tokens)
One thing that trackjs does and we don't is provide "breadcrumbs". In reality we haven't seen it be that useful, but it's something we're exploring.
It helps to think of Sentry more like classic crash reporting more so than modern logging. We want (and need) to know precisely what a stacktrace is, which piece is the function name, the line number, etc.
You can't install via standard channels yet (effectively you need to install from GitHub), but otherwise there's no major concerns.
* Can't filter by additional data sent by you.
* Can't filter by Browser names instead of browser versions (i.e all Firefox instead of Firefox 35,36 etc)
* Can't ignore errors created by bots and old browsers.