irogers

Just to advertise the perf tool has inbuilt flamegraph generation code these days (well leaning on D3.js). So `perf script report flamegraph` will convert a perf.data file into a flamegraph.html. Similarly there is `perf script report gecko` to write out the firefox profiler's json format.

Agreed this is awesome, obviously sanitizers fill some of this gap currently but they aren't great with things like reference counting that RAII makes a doddle. Fwiw, here is an implementation of a runtime RAII style checking on top of leak sanitizer: https://perf.wiki.kernel.org/index.php/Reference_Count_Check... There's an interesting overlap with the cleanup attribute that is now appearing in the Linux kernel (by way of systemd): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Somewhat related, "Data-type profiling for perf": https://lwn.net/Articles/955709/

For a collection of profilers you can also check out the Profilerpedia: https://profilerpedia.markhansen.co.nz/

There are also GC techniques to make the pause shorter, for example, doing the work for the pause concurrently and then repeating it in the safepoint. The hope is that the concurrent work will turn the safepoint work into a simpler check that no work is necessary. Doubling the work may hurt GC throughput.

This is awesome work and textual being able to support terminal or web (https://github.com/Textualize/textual-web) also gives hope that this can be more than a terminal app. I'm hoping that in the future features like this can be standard in Linux's perf tool, for example, Firefox profiler support was recently added as a Google summer-of-code contribution: https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Prof...

Wouldn't all your kernel stacks then end up in whatever this handler is? Why not implement your approach and mail it to LKML :-)

prodfiler clearly has a market. It would be interesting to see the approach as something standard in the kernel tree, perhaps it can be added to perf's synthesis, etc. There is already BPF based profiling within perf to avoid file descriptor overheads. If engineering resources are the issue then this could be a good GSoC project: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

Allowing processes to sniff each others stacks has some fairly obvious security issues.

For every running application turn DWARF data into BPF maps. Does this scale?

AMD will have support in Zen4 and Linux 6.1 (which is LTS):

https://lore.kernel.org/lkml/Yz%2FcpNTSacRMh1FK@gmail.com/

Further, precise events are fixed in Linux 6.2:

https://lore.kernel.org/lkml/Y5eQeR2tpZ%2FBos49@gmail.com/

DWARF bytecode is a full VM. Do compiler writers test their DWARF output? (my experience is not - especially for architectures out of the big 2 or 3) How does the kernel access the ELF file pages with the DWARF information in when in an NMI handler? You could mlock all your debug information when a program loads but the memory overhead wouldn't be nice. It is hard enough getting a build ID.

The elephant in the room btw is LBR call stacks, but they aren't exposed in the kernel/BPF yet. Userland perf has them and they recently became available on AMD.

This could be a great Linux perf GSoC project. Projects and mentors are being looked for: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

Agner speaks about memory renaming back on Zen 2:

https://www.agner.org/forum/viewtopic.php?t=41

Intel Alderlake has performance events for tracking it:

https://github.com/intel/perfmon/blob/974c69919b2a9dfd8278cf...

But even before this you had store to load forwarding on x86. I'm not saying you have, but before inventing a performance problem it is worth spending time trying to diagnose it with thorough profiling (e.g. [1]). The Fedora frame pointer patch did a thorough performance analysis and performance will be revisited again. Unfortunately there are a lot of arm chair performance experts who haven't spent time looking into the details.

[1] https://perf.wiki.kernel.org/index.php/Top-Down_Analysis

Twitter: "No need to recompile with frame pointers."

Web page: "always-on profiling powered by eBPF technology."

BPF stack traces are gathered using frame pointers:

https://github.com/torvalds/linux/blob/master/kernel/bpf/sta...

https://github.com/torvalds/linux/blob/master/arch/x86/event...

This seems relevant:

Future of Memory Safety Challenges and Recommendations https://advocacy.consumerreports.org/wp-content/uploads/2023...

""" Case Studies 1. The Python cryptographic authority is one of the most widely used cryptography libraries in the Python ecosystem. Many of the tools are largely built on OpenSSL. The popular cryptography library is written in C. About two years ago, the maintainers started the process of migrating some of their dependence on OpenSSL away from that to their own Rust code, particularly starting with areas around certificate parsing and parsing of other structures. These are some of the most classical places to find memory safety vulnerabilities in C libraries, and they wanted to mitigate the risk that they were having by relying on OpenSSL.

Another benefit was getting huge performance improvements, because the greater safety guarantees they were getting from the language allowed them to be more aggressive in doing things like not copying memory. Specifically, the safety guarantees of Rust mean that one can easily represent structures like X.509 certificates as an array of bytes, and then a parsed structure containing pointers into the original array. ... """

Problem with my open-source startup: https://docs.google.com/document/d/1kiW9qmNlJ9oQZM6r5o4_N54s... Worth a read and I'm sure Rui would appreciate ways to resolve this.

People try to get cute with 3-way compares. A real Java bug that inspired: https://errorprone.info/bugpattern/BadComparable

public MyFile implements Comparable { ... long timestamp; ... @Override public int compare(Object other) { return (int)(((MyFile)other).timestamp - timestamp; } ... }

The int conversion loses the sign of the subtract. The particular use of the code was sorting files to delete the X number of oldest.

More examples about the dangers with just ints: https://stackoverflow.com/questions/2728793/java-integer-com...

The comparable API in Java came from the C qsort API. It would have been better to just have a less-than method. Kind of surprised to see this in Go near a core API.

Fwiw, this would fail Android's Compatibility Test Suite:

https://android.googlesource.com/platform/cts/+/3ece5ae7a51d...

Not having cognitive load motivates something like the LLAMA work: https://research.google/pubs/pub49008/

The Linux Foundation also has projects in the Google Summer-of-Code (GSoC): https://wiki.linuxfoundation.org/gsoc/google-summer-code-202... In 2022 GSoC is expanding eligibility to non-students: https://opensource.googleblog.com/2021/11/expanding-google-s...

It can catch people by surprise that Java's narrowing conversions may not preserve the sign. For example, the following is broken:

class TimestampedObject implements Comparable<TimestampedObject> { long timestamp; int compareTo(TimestampedObject other) { return (int)(timestamp - other.timestamp); } ... }

ErrorProne catches this: https://errorprone.info/bugpattern/BadComparable

String should be an interface/protocol. When I log a message, I want to pass a string. If I have to append large strings for a log message I don't want to run out of memory, I should be able to pass a rope/cord [1]. We've known how to abstract this for forever and should work to optimize our compilers/runtimes accordingly. I'm not aware of a language which has got this right, for example, Java has the ugly CharSequence interface that nobody uses. StringProtocol in Swift (can I implement it?) makes you pay a character tax rather than to just pass a string. Rust/C++ give various non-abstracted types.

[1] https://en.wikipedia.org/wiki/Rope_(data_structure)

Given there is before and after code, perhaps the translation can be automated. If the whole code base were auto-translated, what bugs could be detected and fed back to the C implementation? I'm assuming there is going to be hesitance from the FreeBSD community from migrating their codebase from C to Checked C, but that doesn't mean this work needs to come to nought. On top of bounded memory accesses, it'd be interesting to see better thread, signal, interrupt, etc. safety support. I guess at some point this just looks like Rust.

The issue with BURS is that lookup tables can be 10s to 100s of MB. Most code generators will use dynamic programming, but hybrids have been considered: https://dl.acm.org/doi/10.1145/1133255.1133988 (Fast and flexible instruction selection with on-demand tree-parsing automata)