By simple example, I mean this:
https://tech.popdata.org/images/flamegraph-example.svgBut looking at the one you're referencing, slightly out of order:
> highest-magnitude function I see there is __int__malloc.
Height shows stack depth, not magnitude.
> the actual culprit is Record::hasVariable, which is somewhere in the middle of the graph, and not any redder or less red, or wider or less wide, than many other functions.
> not any redder or less red
The colors are randomized to help with contrast (Brendan's website mentions this practice), so they aren't conveying any information.
> or wider or less wide
That's not really true. Hovering over Record::hasVariable tells that this bar covers 45.6% of the runtime. The only bars wider than that are the callers of Record::hasVariable (edit: rather, the stack through which Record::hasVariable is being called), i.e. the bars on which Record::hasVariable is resting.
> somewhere in the middle of the graph
Sure - being in the middle height-wise means that it's somewhere in the middle of the call stack. But there are some clues to its relevance:
1. It's close to the boundary between application code and standard library code. It does call MetaData::Cache::getVarsByName, which (going by the name) also is part of the application, but everything deeper in the stack (i.e. on top of those bars) is purely std:: stuff.
2. Domain knowledge. The text alludes to this: Record::hasVariable is a conceptually simple operation that's not expected to be a major part of the runtime.
This does not mean that Record::hasVariable must be the culprit. Maybe some function higher in the call stack (e.g. EditingAPI::Rules::getSourceDataAsLong) is calling Record::hasVariable way too many times? But it's a good place to start looking.