brendangregg

I'm hoping for more Troughton too some day. And speaking of Jamie: https://www.brendangregg.com/blog/images/2025/brendan_hines....

Thanks, I never saw the Tech Crunch parodies before. I also didn't use the words "make/making the world a better place".

I actually said "it's about saving the planet" based on all the environmentally worried news articles I've seen.

Given the widely reported issue of AI environmental cost, and being among the top people in the world who can fix it, I don't feel comfortable with _inaction_. I can reduce datacenter sizes, so I'll do that.

As a performance engineer I'm familiar with Jevons paradox, but it does not discourage improving efficiency.

No, that's not what they say.

When did I say I don't think comp is important?

As an engineer I can't comment on future company predictions. It sounds like a question for Sam Altman, as he has discussed risks in the past.

My wife was paying for ChatGPT before I joined. I didn't ask Mia. I probably have three months of hair growth before my next chance to ask.

Again, many comments here saying I only care about the money, and while comp is an important factor I think it characterizes me as someone I'm not, and forgets what I've been doing for the past two decades. I've spent thousands of hours of my life writing textbooks for roughly minimum wage, as I want to help others like me (I came from nothing, with no access tech meetups or conferences, and books were the gateway to a better job). I've published technologies as open source that have allowed others to make millions and are the basis for many startups. I'm also helping pioneer remote work and hoping to set a good example for others to follow (as I've published about before). So I think I'm well known for caring about a lot of things during the past couple of decades.

To answer a few people at once: I did mention compensation as a factor in the post, but I didn't elaborate details, so easy to miss. Comp is important of course, but so are the other factors. It feels like I can't go for a day without reading about the cost of AI datacenters in the news, and I can do something about it.

Funny to see this fixed in 2023 and the side effects. Back in 2004, before I focused on performance, I did some security work including inter-keystroke latency analysis of captured SSH sessions to estimate the commands typed:

https://www.brendangregg.com/sshanalysis.html

The 2023 patch should finally fix that 2004 issue.

Yeah, different numbers, 110 customer meetings, the other post tracked 1-6am meetings. I'm glad I tracked 1-6am meetings since I've shared that number when people think that remote workers aren't making an effort.

Right, laser 300 was called the VZ300 here. I'm out of desk space so I had to put the VZ300 on a stand above my C64C. Maybe AI can finally help me code some C64 and VZ games. :-)

I like to measure things. In real life and on computers. But I also have a couple of work reasons for it:

As a remote worker, I'm under extra pressure to prove that remoteness works.

As a senior employee, I'm also under pressure to regularly report where my time is spent.

I actually only just got the C64C and was surprised I could get it to fit on my desk. C64Cs are massive. Now to connect it to some display...

Right, this kind of job can improve, and should improve. But right now there's talk of ending remote work entirely, without considering that people have been making the effort for years, hence sharing an anecdote. The bigger point is that remote workers are more accommodating than is assumed, and my anecdote is an example of that (even if it is too much).

One other factor I'm curious what people think: If you were offered 7-figures USD to do these kind of hours from anywhere in the world, would that make a difference?

Thanks, glad it's useful!

To everyone building these things: Please add a disclaimer to say something like:

"This is not a vulnerability: eBPF currently requires root access to do this. Also, eBPF makes this easy but does not make it possible, as debuggers, interposers/shims, and other tools can also attach to pre-encryption points, and therefore banning eBPF (as some people want to do after seeing projects like this) would not actually improve security, but it would instead _reduce_ security as it would prevent eBPF-based security solutions from being used."

I could just regenerate these heat maps with 60 rows instead of 50. I'm limited by the sampling rate that was captured in the profile data file. To provide even more resolution (so you had many samples within a game frame) I'd need to re-profile the target with a higher frequency.

When Martin, my colleague at Netflix at the time, built a d3 version of FlameScope, he put a row selector in the UI: https://github.com/Netflix/flamescope

The origin problem for flame graphs was MySQL server performance involving dozens of threads: as a timeline view you need dozens of timelines, one for each thread, since if you render it on one (I know this is probably obvious) then you have samples from different threads from one moment to the next turning the visualization into hair. Flame graphs scale forever and always show the aggregate: any number of threads, servers, microservices, etc.

I think great UI should do both: have a toggle for switching between flame graphs (the summary) and timelines (aka "flame charts") for analyzing time-based patterns. I've encouraged this before and now some do provide that toggle, like Firefox's profiler (Flame Graphs and Stack Charts for timeline view).

As for 20ms, yes, we do want to take it down. A previous HN comment from years ago, when I first published FlameScope, was to put a game frame on the y-axis instead of 1 second, so now each column shows the rendering of a game frame, and you can see time-offset patterns across the frames (better than a time-series timeline). We started work on it and I was hoping to include it in this post. Maybe next one.

Nice tool. Looks like it uses what we now call a "flame chart" (not a flame graph). I don't know if it ever had an original name, but I've seen these in tools from at least the 1990's.

The answer has always been in the source of flamegraph.pl:

  # This was inspired by Neelakanth Nadgir's excellent function_call_graph.rb
  # program, which visualized function entry and return trace events.  As Neel
  # wrote: "The output displayed is inspired by Roch's CallStackAnalyzer which
  # was in turn inspired by the work on vftrace by Jan Boerhout". 

Neel's did upward "flames" and I think CallStackAnalyzer did as well from memory, so based on the ones I was familiar with this was the precedent. We were also dealing with stacks that were commonly less than 30 frames deep, so it tended to fit on the screen. Also in my original flamegraph.pl implementation was an option to invert the flame graph (--inverted), so I've always let people choose. But I guess I had to pick something as the default.

Neel's was the biggest direct inspiration. I changed it to put the alphabet on the x-axis and reduced the color hues, but they look similar.

Microsoft have been driving the work to make eBPF an IETF industry standard.

Yes, we know eBPF must attach to equivalent events to Linux, but given there are already many event sources and consumers in Windows, the work is to make eBPF another consumer -- not to invent instrumentation frameworks from scratch.

Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."

Right, and we wanted to talk about all security solutions and not make this about one company. We also wanted to avoid shaming since they have been seriously working on eBPF adoption, so in that regard they are at the forefront of doing the right thing.

Right; disabling eBPF doesn't solve this. And the bigger point is that this kind of eBPF is still super-user only.

Apart from the more exotic facilities, the critical facilities that would be hard to disable include LD_PRELOAD for interposers/shims (as you mentioned), and gdb for just setting breakpoints on crypto functions. And if neither of those existed, then I may have to edit openssl code and recompile my own edited version. And if that wasn't allowed (signed libraries) then maybe I'd edit the application code or binaries.

My preference is tools that give a rolling output as it let you capture the time-based pattern and share it with others, including in JIRA tickets and SRE chatrooms, whereas top's generally clear the screen. atop by default also sets up logging and runs a couple of daemons in systemd, so it's more than just a handy tool when needed, it's now adding itself to the operating table. (I think I did at least one blog post about performance monitoring agents causing performance issues.) Just something to consider.

I've recommended atop in the past for catching short-lived processes because it uses process accounting, although the newer bpf tools provide more detail.

strace is ok as a last resort, but "perf trace" and bpf tracing tools are the production-safe alternative. https://www.brendangregg.com/blog/2014-05-11/strace-wow-much...

It's not ridiculous at all. Who are you?

You are backing away from your other positions, for example:

> I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.

Do you now agree that TPC-B is too simple and microbenchmarky? And if not, please tell me (as I'm working on the problem of industry benchmarking in general) what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?

I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs."

I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?

I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks.

You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".

For a busy 64-CPU production JVM, I tested Google's Java symbol logging agent that just logged timestamp, symbol, address, size. The c2 compiler was so busy, constantly, that the overhead of this was too high to be practical (beyond startup analysis). And all this was generating was a timestamp log to do symbol lookup. For DWARF to walk stacks there's a lot more steps, so while I could see it work for light workloads I doubt it's practical for the heavy production workloads I typically analyze. What do you think? Have you tested on a large production server where c2 is a measurable portion of CPU constantly, the code cache is >1Gbyte and under heavy load?