DmitryOlshansky

Rather reliability of sum is a product of reliabilty of constituents (unless redundancy was applied) where reliability is in [0,1.0] range.

Yes, a simple consequence of e.g. 3-digit number being:

100x + 10y + z modulo 3

Removing 9y and 99x gives equivalence: x + y + z modulo 3

Now what’s left is induction with proper base (1-digit).

Why not every hour or even half-hour?

Provided that you can run multiple builds in parallel but each of them has somehow limited parallelization.

It’s a great digest, I wouldn’t watch the talk but now I’m certainly more interested.

Added talk to the watch later list.

Or rather the old adege of:

“The luck favors the prepared”

Even broader look on this is which run-time properties the resulting _typical_ code has.

Most languages tend to produce a distinct behavior patterns and properties, such as (in no particular order):

- multi-threaded / single-threaded / multi-process

- frequency of memory safety bugs

- broken locale / unicode handling

- broken or non-standard network settings handling

- only synchronous I/O calls (or in contrast only async I/O)

- releasing memory back to OS (or not)

- fragmenting heap over time (or not)

- fast / slow text processing

- fast / slow binary codecs

Or how to elegantly waste a perfectly fine piece of hardware in a neatly over-engineered way ;)

The problem with most great projects written by small group of brilliant people in any language:

- it introduces its own vocabulary, it looks like gibberish until you build up the context

- since the group is small and gifted they have no problem creating and using the power tools of their own

- there is no documentation so your only way is through experimentation and direct communication

- the folks who have built it had to move on to solve another (bigger) problem, leaving no traces of what the vocabulary is, where the design was headed, what should be changed with the new requirements and again next to zero documentation

E - extended, meaning from 16 to 32 bits.

A - accumulator

X - wildcard for both upper and lower 8-bit parts, this becomes redundant b/c of E prefix

Needs honorable mentions to:

- Avro

- CBOR

- SMILE

And BSON anyone? I think not much people besides MongoDB using it though.

Yes, compressing JSON with gzip-style compressor usually yields 0.5-1% better results then equally compressed binary format (in my limited testing). Still the serialization speed and savings on compression are great to have.

Have been writing in D that has conservative GC for years.

Yes, on 32-bit platform false pointers are a problem esp as you get closer to 1g for instance.

On the other hand on 64-bit I’ve never seen it at all. The chance of arbitrary long/double to point to some object in the heap area is 1/2^32 smaller and that turns the problem from common nuisance to purely theoretical problem.

> important distinction is that typically GC iterates over something (incurring a cost of O(N))

During _collection_ and N is size of live set, so divided by allocation the cost is amortized and can safely considered as O(1) much like appending a dynamic array where costly O(N) resize is amortized over N appends.

Typically GCs allocate via bump a pointer allocation which is actually faster then what pretty much all of libc malloc implementations would do.

> C++ case iterates over nothing (incurring a cost of O(1)).

Not quite, take a look at jemalloc paper for instance:

https://www.facebook.com/notes/facebook-engineering/scalable...

A lookup to get the metadata for this specific allocation could be anything from O(1) to O(lgN) depending on size and malloc strategy.

The point is that tracing GC searches for things to keep and assumes the rest is free.

Which I believe is the important distinction, and quite often it doesn’t know or care “which allocations to free” because it does it in bulk. That allows it to be efficient.

GC doesn’t search for chunks that can be freed. Also “iteratively” - what do you mean by this?

There is no known algorithm for GC that searches for garbage to delete, the way GCs work is exactly opposite - find all live objects (reachable from roots - registers,stack, statics/globals) and the rest of the heap by definition is garbage.

> Trie structure sucks if you have pretty random words.

Classic uncompressed trie sucks pretty much in all cases.

Now if we go for half-decent implementation of packed variation, it does get significantly better:

https://en.wikipedia.org/wiki/Radix_tree

I actually did that once - splitting one NUMA machine into 2 ES instances each isolated to its own socket (numactl etc).

Sadly at the time there was a big perf problem with indexingdocuments with 20kb+ binary blobs (non-indexed field) and so the change to 2 nodes vs one did have little noticeable effect on the throughput benchmarks I did with our data back then (~2015 ES 1.4) In the end I dropped this split and focused on the other bigger problems.

Would love to revisit this exercise with more of free time and better tools.

Not to worry, I’ve been on the other side of open source - being one of maintainers for some parts of D language. It takes a lot of time and effort.

Hope to work with you guys on OpenDistro more closely in the future.

I stand corrected then. I’d certainly need to check my measurements on this one.

I’m basing it on my limited experience of rewriting things from optimized but messy C++/D to JVM. Yes, the end result is much simpler but is memory hungry and is ~2x slower (after optimizations and tuning). Sometimes you can fit Java in less then ~2x of original footprint but at the cost of burning CPU on frequent GC cycles.

Not every application is the same but the moment it involves manipulating a lot of data in Java you fight the platform to get back the control you require for those things. And by the end of day there is a limit at which reasonable people stop and just give up dodging memory allocations, creatively reusing objects, wrangling off-heap memory without the help of type system.

The total number of shards per node, across all indices.

As far as number of shards per index goes there are many considerations some good ones are outlined in the article.

Thanks, I appreciate that.

I tried reaching out on forums and was assured they’d get to it. I’m used to things going slow in OpenSource so not too worried at this point.

Would be lovely to upstream it though, as I’ve patched 2 versions now and not looking forward to continue doing so ;) Plus I’d be glad to contribute more things based on our experience running ODFE, I’m very intrigued by other upcoming plugins in ODFE repos as well.

See my other reply on big iron vs small instances (as in 31g of heap “small”).

In short there are many variables at play, but without any context generally I’d recommend trying many small instances first. Now depending on your hardware, scale and application things may easily change in favor of big iron.

Seen that and starred probably half a year ago. Never had the time to dig around and see what it’s like in perf, operations and scalability.

(Setting aside the fact that Logstash is JRuby/Java app easily eating said gigabytes of heap)

Do JSON logs take gigabytes?

If they do for you then yes, gigabytes of disk and memory are pretty much guaranteed. Also things tend to pile up with time (even on a few weeks horizon).

In all honesty, I believe a finely crafted native code solution for this problem could achieve x3 less ram usage and x2-3 indexing/search performance. Going beyond that is also possible but will take remarkable engineering.

Update: to expand on the last point, c++ solutions are typically closed sourced. Rust and Go both have interesting open-source full text engines: https://github.com/tantivy-search/tantivy https://github.com/blevesearch/bleve

In near future I totally see someone producing a great open-source distributed search project that is at least on par with today’s ES core feature set.

If the dataset it fits on one node - good. You might still be missing out on reliability (should that node fail or just plain restart).

Simple catalogs and/or web pages search usually fit in RAM, so the advice of disk:ram becomes less relevant, also the type of drive would be mostly irrelevant.

Indeed OpenDistro is a great option to have. Currently using ODFE Security plugin with X-Pack basic (minus X-Pack security module).

Still waiting on these guys to review my perf patch though:

https://github.com/opendistro-for-elasticsearch/security/pul...

Last time I ran with ZGC the main problem was that ES OOM circuit-breakers are going nuts since they are not prepared to have memory mmaped 3 times. So their calculations are off by a factor of 3 blocking most of requests for no good reason.

Running with oom circuit-breakers disabled is something I’m currently considering.

I highly recommend trying out Java 11. G1 got quite a few improvements since 8.

In particular full GC is parallel since 10: https://openjdk.java.net/jeps/307

Certainly a problem with 100-s of nodes. One thing is that GC pauses in big clusters become more problematic as the chance to hit stop-the-world pause on at least one node for every request is getting higher.

Going beyond dozens takes a fair amount of tuning and trouble-shooting.

Currently we are running ES on big iron in production.

There are both advantages and disadvantages. On plus side:

- easier to manage several big machines than shitload of small instances potentially on top of virtualization solution (debugging overlay network etc. at night is not fun)

- nodes are more resilient to big requests (both big bulk indexing and resource-heavy searches). Your risk of hitting sudden OOM is practically zero (although ES does have circuit-breakers that try to prevent processing requests that would cause OOM)

- while not having compressed oops is sad, not having multiple copies of JIT and compiled code cache etc. is a plus

- using a modern concurrent GC is more sensible on bigger heaps. G1 is actually fine on 64g. Some of us are running Shenandoah on Java 13 in production, I’m looking forward to apply it on my clusters (or get back to testing ZGC).

Disadvantages:

- NUMA. Try to pick more recent Java as it’s tuned to run better on NUMA machines, still not all of JVM is NUMA-aware

- fault domain is larger and getting hot node up to speed (in-sync) after restart can easily take about an hour. Getting it up from empty storage going to take a bloody while

- elastic.co folks seem to have recommended settings for lots of small nodes not big ones, so you are on your own to discover proper limits for every setting