xakahnx

My company provides me a work phone with a separate phone number. This is where I get pages and urgent or intentionally disruptive calls/messages

Have you tried calling people on the phone? This is the remote-work equivalent of tapping someone on the shoulder. It's equally helpful for companies that have multiple offices where you need to collaborate with someone not physical near you

HN gets comment threads every month or so with "FAANG engineers do in fact make >500k annually". I see commenters doubt this every time, but it's true!

Adding to the speculation here, I'm willing to bet some component of their issue is not entirely technical. Regardless of the underlying cause (PKI was mentioned), for downtime to last this long it almost definitely means some persistent data was lost or corrupted. Of course they can recover from a backup (I'm confident they have clean backups) but what does that mean for the business? "We irrecoverably lost 12 hours of data" could have severe implications, for example legal or compliance risks.

I find the same failed incentive scheme in industry. When a company gets big enough (tens of thousands of employees), taking risks to achieve something new and interesting doesn't pay off. It's more reliable to just be present and focus on writing about yourself during performance review time.

The general attitude here reminds a bit of the following post from the architect of the Varnish proxy. I think the attitude comes down to the fact that modern kernels and in general the foundations of network programming are pretty strong. We should trust them more.

https://varnish-cache.org/docs/5.2/phk/notes.html

So much to relate to in this article. If I'm cycling through stackoverflow for answers, it's because I've convinced myself I'm in over my head. Depending on the day this is either laziness or imposter syndrome. There's always a better resource out there, and better to find out what that is now because very rarely am I just one answer away from completing anything.

It brings interesting trade-offs for program design. You can write the code one way which may be 10x faster but harder to reason about, or another way which is more straightforward but takes an extra 5 days go execute. How confident are you in your code or debugging ability? How many iterations will you need? I'm assuming this was written in CUDA based on the block/thread ID mix-up.

The bulk of this article is more about creating your own programming language to solve a particular problem or set of problems. Less about choosing between general purpose programming languages. Does your code often require the same boilerplate to get going? Do you have common tasks which are error-prone? There are syntactic or programming design reasons to use a particular language, or maybe the reason could be performance. Think of tensorflow for example, maybe it's familiar for some R programmers. Code written in a language designed for a particular purpose can be very elegant. I like [1] as an example.

[1] https://docs.racket-lang.org/scribble-pp/html.html

One of the consequences of TCP being implemented in the Linux kernel is that companies were implicitly motivated to open source any performance or efficiency improvements to their TCP stack. Are any companies collaborating to improve QUIC efficiency issues in open source? The issues are well understood among at least the big cloud providers.

Good point. I was picturing some gather/scatter over strings which are not in adjacent memory (maybe a generous interpretation for my use-case). Concatenating small strings into ndjson may still come out ahead performance-wise.

I like string processing libraries that implement a multi-document feature like the one mentioned here. There's always some efficiency to be gained- maybe the public API has a lot of branching or initialization, maybe it acquires a lock, etc. Batching will amortize that cost, or open up new opportunities for SIMD processing. Letting the user reduce overhead through batching isn't something I see supported in other libraries, even ones that advertise high performance.

Cool! I looked up more info on this and ended up here, http://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2....

But I only skimmed this and I don't see that it mentions about migration across hosts. Only that the hypervisor is able to expose the MSRs or PMCs.

Any idea why this is? I'm even more curious since you've singled out "cloud VMs" from all VMs.

This is still a disaster with hyperthreading where you can have two threads contending for a lock on the same physical core.

Even if your thread won't be scheduled off the core I don't know if you might still have irqs stealing time during the critical section.

Most performance problems are measured as some function of the hardware resources (CPU usage, throughput, disk iops, network latency, ...). How do you find out what application or what part of your application is causing problems or consuming resources? eBPF helps with observability from the kernel up through the application to connect what your machine is doing (disk throughput too slow, maybe?) with what your applications are doing (frequency of reads/writes, object sizes, prefetching, buffer cache usage?).

For some use-cases, the fact that you can insert probes and eBPF code to a running program is a huge win. This is more obvious to kernel developers who can't always recompile a kernel, deploy it, and recreate a particular state to debug a problem. Application developers may think they can just change the code and add printf to get better observability, or maybe use gdb? eBPF has its advantages.

More than just broadcasts as we think of them in networked systems, it requires serialized/transactional broadcasts. Network distributed systems also have problems with ordering which is something made easier when you have a single bus with all transactions in sequence.

Keeping the directory coherent is the difficult part when translating directory-based cache coherence protocols to other distributed systems problems. The directory is like an oracle that sees every transaction in order. This is hard in most network distributed systems problems where you have to worry about availability, network partitioning, or durability of this node.

Also interested in this protection. How does it detect this situation and what does it do when something is detected? Did you notice when it was accidentally removed that your monitoring of this condition went to zeros (or was it never happening during normal operations)?