brancz

We've "fixed"[1] all the tooling that we work on at Polar Signals, and even then I think this is the right thing to do. It enables so much more debuggability aside from profiling.

[1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

With the combination of eBPF and DWARF-based it's not quite that simple, there's a pretty elaborate dance between user-space and kernel-space that needs to happen for this to work. With frame-pointers it's walking a linked list in kernel-space.

I would need to double check with the team on this detail, but if I recall correctly the architecture as it is is specifically designed to make the bpf verfier happy and we didn’t think it was going to be possible with existing formats. But happy to reconsider, we’d of course much rather use a standardized format if possible!

A lot of profilers have, it’s literally what unwind information is for, but we did it in kernel therefore not the entire stack needs to be copied to user space so it’s way less overhead.

Ubuntu hosts a debuginfod server that you can automatically discover debuginfo from any binary from using the binary’s build id.

https://sourceware.org/elfutils/Debuginfod.html

We’ve also figured out an alternative format to use from within eBPF to unwind stacks (we happen to only support dwarf at the moment but theoretically any source information could work): https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

Correct and there are already known exceptions such as the python interpreter in which the “interpret function” function actually falls into this case and so for the foreseeable future python is going to continue to be compiled omitting frame pointers. But by default this destructive micro optimization is off by default until proven a performance bottleneck, just like any performance issue should be!

Hyperscalers have long been doing infrastructure-wide profiling (or "Google-Wide Profiling" as the first whitepaper on the topic calls it [1]). This tech allows Google to reduce infra-resource usage by multiple percentage points per quarter.

[1] https://research.google/pubs/google-wide-profiling-a-continu...

Profiling tools have already solved the ability to reliably unwind in the absence of frame pointers[1], but there are plenty of tools that this kind of investment is simply too much that it won't ever happen, like bpftrace or bcc-tools.

[1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

Founder of Polar Signals here, the profiling product that's mentioned in the blog post. We've already made it possible to unwind without frame pointers relatively cheaply (<1% overhead), and have written about this extensively [1].

That said, no matter how we spin it, frame pointer unwinding is always going to be cheaper, and while profiling is getting better, I think I'm almost more excited about the other aspects of debuggability this is gaining: out-of-the-box working bpftrace, bcc-tool and anything else that needs to deal with unwinding with just about anything that's running on the box. I think we'll see a huge gain in capabilities over the next few years with frame pointers more prevalent in fedora and ubuntu and I'm sure more will now follow.

[1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

I would argue that LTS releases are going to be deployed millions of times and stay around effectively forever with lots of very critical software being deployed on it. Having all processes/binaries be debuggable cheaply and easily in stressful situations is a major improvement that's now here to stay.

While profiling benefits immensely, imagine all the other ramifications. Cheap tracing of MySQL, postgres, or anything else using bpftrace or any other bcc tools. Getting stack traces of processes that are already core dumping...

Profiling is one part, but the debuggability this enables is going to be huge in the long term I predict.

We wrote pretty extensively about what was needed to be able to profile things without frame pointers [1]. It's still possible at less than 1% overhead with the right set of technologies, but frame pointers unwinding is virtually free.

[1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

You're not wrong, but isn't it exciting that it's gaining more traction? Ubuntu is also very widely deployed, so I'll celebrate every distro that makes this the default, be it Ubuntu in this case, or Debian, or Arch, or Suse, or or or...

100% the way I see it! On 32-bit the performance benefits is very major on just about everything, but not so on 64-bit.

Frame pointers are such a destructive micro-optimization to omit by default, I am beyond excited about this collaboration with the folks at Canonical to make Ubuntu debuggable by default!

Same with profilers. We had to go through a lot of trouble to get unwinding without frame pointers to work on the eBPF-based profiler we work on [1], and even crazier for interpreted languages [2].

[1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

[2] https://www.polarsignals.com/blog/posts/2023/10/04/profiling...

I happen to work on a system-wide profiler[1] that is eBPF-based, and what we do is load our bpf programs on various kernels[2] using qemu. This is important as the jit and verifier change significantly across kernel versions and we want to make sure we maintain compatibility.

[1] https://github.com/parca-dev/parca-agent

[2] https://github.com/parca-dev/parca-agent/blob/54434c02773a8a...

I just spent ~1.5 years of my life on various aspects of debuginfos and that even just for profiling where I only need line/column information. So I feel this comment through and through.

You're 100% correct, but for our users it sure seems like a bug.

I happen to work on profiling tools where we rely on compilers to output metadata about the machine executable code correctly. It turns out this is surprisingly hard and goes wrong quite often, and sometimes it's even impossible to maintain correctness across multiple rounds of compiler optimizations. For us the way compilers solve this is basically the worst possible scenario, when the compiler can't correctly track it across optimizations, it just starts emitting less or sometimes even no information. These bugs are just awful to understand.

At the end of the day compilers are also just software and it's just a matter of how many features you end up being exposed to.

DWARF is touring complete and not the accidental kind, it has instructions and all. GDB implements the whole set so you can basically make GDB do arbitrary code execution if you control the DWARF data.

I was architect leading all things Observability at Red Hat until 3 years ago there was an absurd amount of support for this project internally I never understood it either. But there were huge amounts of customer support, sales and engineers who adore this thing, I genuinely don't understand the appeal when we had next level cluster-wide Observability supported on and off OpenShift.

Even being in a leadership position and basically competing within Red Hat against this, I found no answer to your question.

Thanks for the shout out! The team has done a phenomenal job.

If you enjoyed that post, I think you’ll also like the one we wrote about native unwinding without frame pointers: https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

Yeah that’s basically how this system is built as well. We buffer a certain amount of data in the ingestion nodes and then offload it to object storage. In object storage it’s actually laid out in the parquet format so we could allow users to export/download this. Or like I said, the object storage cost is really the smallest piece in the system we might even just decide not to have any retention at all.

Oh we’ll make this more clear, actually only Linux 5.4 is required, Kubernetes is only to add more metadata. It still works just fine and we have customers running it outside of Kubernetes provisioned by ansible.

Great question! That’s correct, for the moment we’re focusing on everything that happens in the backend. That said, the system will accept anything in the pprof format so I’d love to see something like “1% of users get profiled and data sent” from the frontend, that’d be awesome!

We’ll think about building something like this but if there is someone with a use case I’d love to chat and figure out how we can make it work together!

Haskell support should land sooner rather than later actually! We are very very close it’s just that the Haskell compiler unfortunately creates binaries that don’t comply to the x86 ABI. It’s not too dramatic though, it just shuffles around a few registers’ purposes.

Great question!

The big one that existing columnar databases can’t do (or not well), is the ability to search and aggregate on user defined dimensions (think prometheus-style labels). Influx 3.0 is the only other columnar database that is (now) available that was engineered specifically to be able to do this. The good folks at honeycomb came to the same conclusion that this type of columnar database (wide column) is necessary to build Observability with exceptional user experience. Plus we now own our future and can do any kind of optimization while other companies (and I’ve spoken to them) constantly fear ClickHouse relicensing or otherwise destroying their business.

//edit: we call this feature dynamic columns: https://github.com/polarsignals/frostdb#dynamic-columns

Pricing is never final, so that feedback is valuable, thank you! If you want to schedule a chat I'd love to learn more about your use case and happy to figure out what works for you!

https://calendly.com/frederic-branczyk/getting-to-know-polar...