Using eBPF and predefined inspections to minimize “observability tax” | Hacker News Reader

Using eBPF and predefined inspections to minimize “observability tax”

coroot.com

112 pointsapetruhin14 comments

Thanks for the great article!

At my current employer we have a company-wide service for aggregating error logs in particular (WARN, ERORR level log rows and stacktraces, if it was an exception) so developers can analyze them for debugging purposes. Also it automatically gathers information about incoming http request (geo, ip address, user agent, etc) and you can easily see a particular segment of errors, and what kind of users getting them.

As I can see you have logs quantitative metric https://community-demo.coroot.com/p/oc1vhnmq/app/default:Dep... but without any detalization (maybe it works this way only for the demo app). I mean, it would be great to be able to inspect each ERROR event separately or to define custom SLO with alert for particular type of errors.

Another great feature we use a lot is historical data, so you can find patterns of error spikes on months scale and when it has gone after fix.

FYI this error-service I'm talking about is built on top of the ClickHouse, so it's quite responsive regardless of the large volumes of data.

Another thing I want to mention is cron-like workload (or batch jobs, you name it). Is there any support or useful metrics for it?

Its too bad they had to build a whole separate tool because Grafana wasn't capable. The Node graph panel for Grafana really needs some love.

Edit: Also they need a SaaS to sell, so of course.

An update to the node graph is currently in the works, some love is being shown (I work at Grafana Labs).

It's also too bad that they built their own eBPF instrumentation as the Cloudflare eBPF exporter also exists and is very good https://github.com/cloudflare/ebpf_exporter

Alternatively if what you want specifically are the integrations mentioned on the coroot page then my money would be on isovalent and cilium

With ebpf-exporter it is not possible to implement complex logic, such as converting the PID of each TCP connection into a container name and the destination IP into a real IP according to the conntrack table.

This seems like a limitation that could be lifted instead of introducing a separate product (disclaimer: familiar with ebpf exporter but haven’t dug into OP).

Iirc ebpf exporter had some limitations, but they weren’t fundamental. However it was also fairly light, so maybe another tool is just the right solve.

Coroot's agent collects data from various sources to cover all aspects of container behavior. Ebpf-exporter perfectly solves the problem of running custom ebpf programs and turning their output into metrics, but using it as a foundation for more specific solutions doesn't seem reasonable

Are there more data sources or ways to overlay node graph information over time series data?

I’ve wanted to use it, but haven’t had time to write a custom day source.

There's not much to the eBPF profiler they've built, and not very much overlap with ebpf_exporter; ebpf_exporter also seems to require CO-RE kernels.

>An update to the node graph is currently in the works

Some more info there would be fantastic.

Grafana offers one of the best solutions for storing and navigating through telemetry data, but there remains a challenge in using this data to generate insights. Our goal is to address this issue, even if it means occasionally reinventing the wheel. This is a necessary step at this stage.

this looks really useful! For my business I want 1) high-res data about local CPU-memory-IO (e.g. - how can I speed up tests), and 2) summary sampling data from production, to detect weird bugs or attacks. eBPF might be able to solve both cases!

Does anyone know how this compares to Cilium? https://grafana.com/blog/2022/10/24/grafana-and-cilium-deep-...

eBPF is great! Fantastic to see it used more for observability!

Looks nice, but AFAICT only covers observability of kernel calls, not e.g. calls to other libraries.

Reply on news.ycombinator.com