Hacker News Reader

Comment by tyingq | Hacker News Reader

20B/month is interesting in the sense that Sentry should only be getting a transaction when there's an error in some third party application.

Makes you wonder if a couple of their clients paid for this error monitoring service, then quit watching it. That's a lot of errors.

View on HN

Suppose you track all client-side errors. Some portion of your users have a blocker that bans an analytics tool you use. Every hit generates an error. You could disable the error if you wanted in that case, but why not keep it online and use the graph on sentry to track how often it's happening?

Ah, okay. Just seems wasteful to send the whole stack trace, local vars, etc, for that. But, yes, I can see people doing what you're describing.

One really big thing I learned from doing this on a public website: there are a ton of ISPs, browser extensions, antivirus and malware which inject really horrible JavaScript into every page. If you run Sentry as a front-end error collector you will get a ton of bizarre-seeming errors for other peoples’ code and, if you're really lucky, find that some of them used the same variable or function names as your code.

Is there a good way of preventing unauthorized javascript injections? Some type of onReady cleanup script that can identify what is supposed to be there and then strip out everything else?

HTTPS

facepalm

Not really. That stops an ISP, but local malware, antivirus, etc can be browser extensions.

Sadly, not completely. A surprising nonzero number of users have isps that insist on using their own root cert to inject this nonsense.

Or just local antivirus. There is a shocking amount of user level firewalls that break the web really badly.

Why would a standard browser trust that certificate?

It's installed by the ISP's setup program which they tell everyone is mandatory (otherwise they won't get the adware kickbacks) and the techs are discouraged from skipping.

How often that does even happen? Many people just get a router (w/modem) in the mail which they just connect to with their devices. No software is being installed.

100% of Comcast and Verizon installs in my experience. RCN offered but wasn't pushy and they seem to have stopped.

You can configure Sentry's JavaScript client library to only collect errors that originate from your script files (even rejecting inlined code).

I wrote more about this and other techniques for battling noise here: https://blog.sentry.io/2017/03/27/tips-for-reducing-javascri...

Yep — thank you for the hard work on that over the years. For me HTTPS + CSP was a good denoising traffic but it was still memorable for just how broken the web environment is, especially internationally (not just the Great Firewall, either – I should have saved what looked like a tracker being injected into requests from an Iranian university).

You can set it up so that Sentry only tracks errors on a subset of pages. One approach I've seen is to have 100% tracking on your canary deployments or for a few minutes/hours after a deployment, and then use the tried and true `if (rand() < .2) { sentry.log(...); }` approach to limit the errors you see. It does mean that you may not receive certain critical errors, but if you are trying to stay under their billing tier thresholds or just trying to rate limit yourself, it's a reasonable trade-off (especially if you are logging 100% on Canaries or for a period after rollouts). You can also customize the error rate based on page so that, for example, you get 100% of errors on checkout but only 10% of errors on the homepage.

I was self-hosting so it was a question of seeing traffic rather than quotas. I'm loathe to sample errors since it often seems like the outliers are the most interesting but at some point it's expensive to avoid.

I believe sentry has support for reversible minification mapping. You could use that along with a PRNG (or maybe just magic number-based) minification to drive 'hard to conflict' js delivery. Unfortunately, I bet any script delivered that way looks a lot like malware. ;_;

If you deploy some non-edge-case bug then you can easily generate thousands of events before you even receive the notification - assuming your service is any popular, but even if it's not it can be e.g. some stuck background job.

Btw, it's on average 7.5k events per second. Not that drastic but of course peak, not average is what matters.

I get the idea, but I would think the right use of the product is to fix the root causes, or suppress errors that aren't actionable. I was basing this off of somewhere on the site that mentioned roughly 200 companies using it. Perhaps it's grown. Would be interesting to know the total per month after they are deduplicated.

Not sure where you'd see 200 companies (we've had more than that for most of Sentry's lifetime). Not every team lines up to a company, but napkin math I'd say we have 10k - 20k companies using our cloud service. That's also excluding all of the open source installations, which is a pretty large number.

disclaimer: Armin from Sentry here

I'm not super happy that this number is in the title because I find this number myself very irrelevant to our operations (not even sure this number is correct, it sounds too small). What's more challenging to us is the complexity of individual events and the bursting of it. A lot of interesting engineering went into that.

Yeah we use sentry, and quite often generate an insane amount of errors on occaison (sorry). And also management not prioritising fixing low priority errors, means we have quite a high error count even then. That 20b does sound quite low based on just our usage of it.

Just a few weeks ago, I hooked up Sentry to our co.'s client-side app, which is a widget/plugin of sorts that's installed on at least a few hundred thousand sites.

Errors on even a tiny fraction of our users can be significant -- we immediately hit the free tier cap. 20B/month does not surprise me at all, especially considering some users are going to be overzealous in their event reporting.

If you have that kind of traffic, we recommend enabling sampling. All client libraries should support sampling out of the box.

I'm sure that happens, but 20 billion a month is around 7700 requests per second, for presumably quite a few companies. It seems high, but not exaggeratedly so.

Reply on news.ycombinator.com