DmitryOlshansky

If your drive tolerates parallel writes well (=SSD) having multiple primary shards per node helps scale indexing.

Shipilev (JVM performance engineer) explains it in great detail here: https://shipilev.net/jvm/anatomy-quarks/23-compressed-refere...

But the short of it: Java uses 32-bit numbers for object references if it can address whole heap with it. Given default 8-byte alignment of object we have 2^32 * 8 = 32g of addressable heap.

Once we are beyond that number 64-bit references are used and suddenly we are wasting a lot of heap space (Java is very much objects everywhere). Usually up to around 40% of heap is references.

- article doesn’t clarify if it’s on hardware or VMs

- 140 shards per node is certainly on the low side, one can easily scale to 500+ per node (if most shards are small, typically power law distribution)

- more RAM is better, and there is a ratio of disk:ram that you need to keep in mind (30-40 for hot data, 200-300 for warm data)

- heaps beyond 32g can be beneficial but you’d have to go for 64g+, 32-48g is a dead zone

- not a single line about GC tuning (I find default CMS to be quite horrible even in recommended ~31g sizes)

- CPUs are often a bottleneck when using SSD drives

I think one of big problems in software development is that nobody measures the half-life of our assumptions. That is the amount of time it takes for half of the original assumptions to no longer hold.

In my limited experience assumptions half-life in software could be easily as low as around one year. Meaning that in 5 years only 1/32 of original architecture would make sense if we do not evolve it.

And that’s how we got to 2 pizza teams I guess.

Mostly same observations, but I actually made that plugin work. In retrospect writing complete generator from scratch would take far less time and effort.

I believe original authors have never written a compiler nor considered a code-generator to be a compiler (which it is). Hence the funky architectural style - whatever was the conceptual model behind it fits poorly to the problem at hand and likely was patched multiple times to cover niche corner cases.

But more subtle when you have: a.size() - 1 < len

Damned age of velocity.

The amount of time wasted zeroing out memory pages in a typical OS is quite significant, also take into account that such an operation will trash perfectly good cache space for no good reason.

In fact all ones is not hard either, there were some experiments to give RAM ability to perform simple page-level computations with promising results.

The correlation of bare-metal cores vs cloud vCPUs is very much nonlinear.

The analogy holds as long as we don’t cross layers, which is quite often true.

Therefore counting total number of layers is performance-wise quite similar to running this many layers of virtual machines. Of course, you need code for all of these translation layers.

Usually works decently unless you tinker with std library / runtime where you are mostly on your own.

Any chance for efficient pure functional string search algorithm?

With that it’s just 2 points in space which gives us little information to deduce 58% faster at X or whatever.

Then there is Processing which is a DSL of sorts and this Mita look like them trying to simplify the wrong part of task.

It is not complex to split your program into particular blocks - how you connect blocks and if these blocks are the “right size”, what blocks you actually need etc. THAT needs simpler means to develop faster.

And this id about IoT - hopefully measurements and units are handled well and that last “thing”.. security, yeees that one needs simpler solutions!

At the very least it’s a pity that even good people make _mistakes_ like that.

Untill that day, whatever I do that is at least robust, diesn’t require mainatance each 2-3 months etc. not even fast, just decent.

In short while our idustry features stuff like that I will be angry, sad etc. But boy I will always have money, I might have a lot fun at least for some definition of fun!

It could take years to trully cover all ISPs. Like it was with rutracker.

And then LinkedIn is blocked for a long time now but I get new job opportunities through it quite often. Sometimes w/o turning VPN on.

As the proverb goes: “Undue strictness of Russian laws was always compensated by it being optional to follow them.”

The irony, now I can’t read about subscription addiction.

First thing a good university will tell a student is the boundaries of applicability of a law or formula.

Newtonian laws work fine except when speeds get close to that of light, for instance (simplified).

It’s even more obvious for empirical “laws”.