Hacker News Reader

DmitryOlshanskyon Proposal – let’s backport Go := to C

As if that ever was the problem with C.

Multiple return values could have been way more handy but even that would be a though sell.

DmitryOlshanskyon Unit Testing Doesn’t Affect Codebases the Way You Would Think

While an interesting read, I can’t help but wonder about two things:

1. This is obviously .NET projects only 2. 100 is a drop in the bucket even if we take C#-only projects. How good are statistical results, is it representative?

DmitryOlshanskyon Amazon, Facebook and Google can all be beaten

The biggest issue with a project under big tech giant is that they could afford to trash it at the moment notice.

There is super low commitment on new projects (say <1% of work force), compared to a startup that is betting all money and talent it could get on a chance that there is an overlooked market for product X. 100% focus, staying far from a comfort zone and taking the risk is what (in a very few cases though) makes it work.

Basically tech gaint doesn’t have to “burn the ships” and therefore has lower commitment, poor focus, has no sense of (get big or go home) urgency and eventually drops the idea before it bears fruit.

It can offord to say “pass”, but doesn’t need to “bluf” and it doesn’t have the same risk.

DmitryOlshanskyon Ask HN: I have an interview with Google in a week. What should I prepare?

Just relax and sleep well.

Always worked better for me compared to times when I spent the last days furiously trying to learn all the stuff.

There are very few things you can learn thorougly at the last moment and it’s not worth it usually.

DmitryOlshanskyon The long goodbye to C

You should absolutely checkout D language (dlang.org).

It’s less stringent compared to Rust’s borrow checker, but also has meta programming capabilities of similar power to macroses. The default is GC manged memory like in Go.

However it’s super flexible in that you can go from 100% simple GC-based programming with dynamic arrays and maps to hardcore low-level stuff with manual memory or anything in between (e.g. Ref-Counting as in C++).

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

> Correct, which is why as part of making processing of iOS symbols faster we wrote our own file format that we can efficiently mmap. It's 10% of the size of the average DWARF, gives us all the information we want without parsing or complex processing

Exactly, cool that you do it.

> enormous amount of javascript and sourcemaps we are dealing with are one hit wonders and/or need fetching from external sources.

Hm, so you don’t know what code your customers deploy? At least in JS you seem to imply that.

Anyhow pardon me for beeing rude in my early posts.

It’s exciting things you do and I’d love discuss on some other medium. How can I reach you?

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

The problem is the title. You guys are doing great, actually!

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

> All event processing (currently) takes place on three 32 core Xeon machines. But it has seen much worse days when we had many more before we optimized critical paths.

Finally I could understand what kind of efficiency you guys have. Not bad at all I’d say though not exactly “high load miracle” the article title would imply.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

> The approach we're using here is parsing the minified JavaScript source from the error location backwards in WTF-16 units, then compare that to the token we see in the unminified source and doing a reverse lookup

What if instead of that you’d actually compile all JS files once to some sensible format that would allow you to do lookup of variable in any scope?

Then you only need token to token map to find the proper location in your precimpiled data.

> However we're different. We're not handling a single app, we're handling tens of thousands of customers and their apps and websites. We can't just have all the debug info files resident at all times, they are too many. Likewise the events we get constantly are from vastly different versions. I don't have numbers right now but I guess we store terrabytes of debug data we use for processing.

So how much do you have of it? All in all, precomputing space-efficient index and storing it in some memcached-style solution should deal with that. I bet you don’t need full debug info as in complete DWARF data.

> Ops can better attest to that but we're not huge fans of depending too much on pinning customers to single workers.

Shards can be rebalanced. Also a memcached or whatever DHTs can used to keep shards. This way workers stay uniform but lookups are still fast.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

Let’s cut down the slack and focus on processing itself.

It wouldn’t hurt to know about the databases if they are heavily used as part of processing

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

Yes, but they don’t run the whole thing on a single laptop do they?

Roughly speaking you take what a laptop can do and a multiply it by 1000x to estimate what a cloud service could easily do. (It’s just an estimate, the reality is not working like that, I KNOW)

Do not take it as Nginx and webpages example, it’s indeed a poor analogy.

I can’t remember the right number for e.g. ElasticSearch fuzzy search across 10m documents of out hand so I’d pick the simpler less relevant number.

Actually fuzzy full text search across 10m documents is at least in the range of 100s/sec on a laptop. Meaning that they have resources to do at least e.g. 100k-500k/s full text searches across gigabytes of symbols.

I bet their queries are simpler then that though (not fuzzy nor full text). It’s debug symbols after all not a natural text.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

Our events are a mess of text that is processed with some convoluted parser to build several cross-reference indexes. At least it allows to piece together events from the same request, etc. But the point is that it’s at best 1% of all the work being done, and it’s not the key problem solved by the system.

Parsing a stack frame would be actually in the same ballpark. Yes, processing it would involve maybe a ~100 index lookups to match stack frames to the source etc, but see below.

Gigabytes of debug symbols? Such as fitting all in a 1/10 of RAM of a typical server you mean? As a hash table, or heck even a trie so you could do range query as well.

Same about a source map. It’s not THAT hard to build a largish lookup table entirely in RAM. Especially if its the MAIN POINT of your business.

I’m curious about that non-linear problem with JSON source map. Could you elaborate?

Also you don’t need to cross-reference symbols/events across your customers. The lookup is thus limited to a shard of that client which I’d dare say won’t contain gigabytes of debug symbols.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

I’d be curious to see how hardware stacks up against events/second end2end as well.

In fact that would be the single most interesting point.

At least a ballpark of say around 20 machines with recent Xeons with ~ X gb ram to process say those 8k/sec.

P.S. In fact calculation were fairly lax and it should closer to 10k/s.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

At 8k/s given a bunch of hardware a typical cloud service will throw at it it can be down to ~100ev/s per node.

Yes I’d like at least 1k/s per node. That’s something interesting.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

I hear you, at the same time you have embarassingly parallel problem that can be easily sharded per customer. (Do I understand your service correctly?)

Processing 8k * 40k is a feat but I’d think you have more then a few dozens of nodes.

Then it gets into ~100s ev/s per node territory. Quite manageable even for e.g. complex search engines to ingress.

Of course, it all depends on where your bottlenecks are.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

I just gave some examples of what can be done at similar speed on dirt cheap hardware.

I my view it helps to understand scale.

More real world example an API at my work logs routinely more then 20M+ events per day, all indexed for NRT search. It’s just a byproduct of the actual work though.

It’s considered business as usual, nobody counts how it adds up half a billion per month.

I assume they index it in some search engine as well. Yeah the scale is x20 bigger, but it’s their _primary_ task.

DmitryOlshanskyon How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

Hate these X millions / billions per month or year or a centuary.

Let’s do the math.

So it’s 20B/30 = 660 m / day

660 / 24 ~= 25 m / hour

25000k / 3.6k sec = 8000 events/sec

Now peaks got to be larger then these measly 8000 events.

Even say peak is 50k/second.

Dang an Nginx can serve a web page faster then that on a single modern notebook. CPU won’t even saturate to do that.

Or if you were to write them to a consumer grade spinning drive disk at 100mb/s you’d get 50k/s if each event is 2kbytes in size.

As usual I’m not impressed.

DmitryOlshanskyon Caching D compiler – preview version

This is one good description of what D feels like. Something that C++ is slowly adopting from except that it can’t get rid of the old cruft.

D at the moment is a the most flexible and powerful tool to build something from scratch. If you care about performance ;)

It’s not a ready as a “platform” though, you are still expected to build quite a few things that are usually taken for granted on your own.