60x speed-up of Linux “perf”
eighty-twenty.org
eighty-twenty.org
I had done a bunch of research into this a while ago and filed an issue with the Ubuntu bug tracker, but was told it was due to ABI stability (which did not make any sense).
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1894407
Understanding that there is actually an underlying license problem is like finally being told all of the reasons for everything that happened on LOST ;P.
I feel for distro maintainers, real thankless job trying to square the circle on a billion different things like this.
I'm past believing that I'm smart enough to always be able to come up with/reimplement a competent enough solution for all my needs by myself every time. But copyright still has to be respected.
When I'm forced into that situation from licensing issues, it really makes me feel like my skillset primarily revolves around gluing together other people's well-designed code, and being left to my own devices exposes my weakness in implementing something from scratch.
perf is licensed under GPL v2 only and libbfd is a GNU tool that is licensed GPL v3 and higher.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815
Really wish the Linux kernel would start mandating new patches to be GPL v2 and higher and get buy in from the largest contributors. A decade later new code would like replace the smaller contributors code and we could consider the software GPL v3 compatible.
Since, realistically, neither Linux nor GNU will budge, maybe the practical solution would be to look into integrating with the equivalent library in the LLVM ecosystem (assuming it exists).
The GPL 3 is a pretty benign improvement on the GPL 2 (though I know Linus objects to the Tivo clause). I don't think anyone who was happy with their software being used under the terms of GPL 2 would be unhappy with it being used under the terms of the GPL 3 additionally, rather than exclusively.
But let's imagine the next version of GPL 4 was "Additionally the authors may at their discretion use it under the terms of the CC-BY-SA license". Nothing wrong with it as a license, but that's a bigger leap in terms of license changes.
It's not hypothetical that they could exercise that kind of power - they _did_ with the GFDL with the "wikipedia can relicense as CC-BY-SA" clause in GFDL 1.3.
I think most people would agree that that was a responsible use of that power. but I can understand those that don't want to extend the trust that all future uses will be responsible.
It says "the recipient can use this under GPL v2 or any later version as published by the FSF", which means if they recipient is happy with the rights they got under GPL v2 they can keep using it under that version for as long as they like.
The only person agreeing to future license changes is the publisher of the code, and they're the ones that chose to publish it under "GPLv2+" or whatever.
If the FSF goes rogue and changes the GPL to be incredibly restrictive (i.e., allowing proprietary redistribution, and I realise this can be considered permissive..) it might be possible to get it to be ruled invalid defaulting to the more permissive licence, especially if you have deep pockets, or if the FSF change the licence to be ridiculously permissive like 0BSD then it's not going to be legal in countries like Germany, either way any major change is likely to result in an international enforcement nightmare.
The scenario where FSF goes "rogue" would be if gplv4 became more permissive, e.g. if it stopped being a copyleft licence.
“You can do A, B, and C with the software I wrote, or whatever X may at some future date decide you can do with it” isn’t something everybody is happy with. It certainly requires some long-term trust in what X will or will not do.
>The blame is pretty clearly on the people who editted the license text to be GPL2 only.
They edited the licence, yes, but the FSF explicitly wants you to do this to make your intention clear(1). When you licence software under the GPLv2 (or 3, etc) you have a choice of 'GPLv2 only' or 'GPLv2, or any later version', however since the licence text only states 'Version 2' with the old short labels being just 'GPL-2.0' there's some ambiguity on whether you mean GPL-2.0-only or GPL-2.0-or-later.
The default assumption should always be v2-only, however as (at the time) the FSF were still recommending the short label of GPL-2.0 and the issue of using v2-only or v2-or-later wasn't really an issue you had a lot of v2 licenced software and patches using the default unedited licence with the FSF short label of GPL-2.0 and this is purely the fault of the FSF. It wasn't until the GPLv3 came around which some people didn't like (notably the Linux kernel, which is probably why perf is v2-only) that you got people editing their licences to make the intention clear, although for many projects they had no choice in the matter as changing to v2-or-later would require permission from every copyright holder that had contributed code to that project, again this is partially the fault of the FSF for not having enough foresight or making the choice of -only or -or-later more explicit and clear.
P.S. if you already know the history and context here this post probably seems a little patronising and I apologise for that.
(1) https://www.gnu.org/licenses/identify-licenses-clearly.html
It doesn't make sense to assign blame on those who bug-fixed the backdoor in the license that clearly would allow a third party to change licensing terms. Even(!) if that third party was rms.
GPLv3 wasn't just an update, it was a major change from GPLv2. Importantly, it limited developers own rights to use code of projects they contribute to how they want (including in devices that are locked or secured in various ways).
Plenty of people even outside of Linux are not going to be going to GPLv3.
The GPLv3 split also damaged the more copyleft side of things as I think some kernel devs predicted.
I think the momentum is currently more MIT / Apache - not sure if that could be where folks could be encouraged to release under to keep at least the open source part alive even if the copyleft part kind of goes away.
Anyone doing any stats on this? The more true open source players are going MIT / Apache style, the proprietary relicense folks are doing the (A)GPLv3 thing to drive licensing revenue given the risk aversion to GPLv3 that is out there. A lot of the GPLv3 codebases require contributor agreements so they can license outside of GPLv3 so they tend not to be true multi-contributor / multi-copyright holder codebases.
This is FUD.
- The developer's own rights to their own code are never limited. What is limited is the rights they get to other people's code.
- GPL does not limit use in any way, it limits distribution.
The limit placed by the GPL on distribution is that the recipient must also be given source code and the same legal rights to the code. The GPL-3 fixes a technical loophole where the recipient is given rights to the code but prevented by technical means of using their modifications to that code on the device it is intended for. And yes, tivoization is absolutely a loophole in the GPL2, i.e. against the spirit of the license - the FSF has always been about empowering users to modify their software.
If so, great, the claim is FUD.
Reality is GPLv3 folks have lied about almost everything - Ubuntu had to get special private communication from FSF to be able to ship a GPLv3 bootloader etc.
And no, the license is pretty darn clear to most of us, even if you are a major contributor to a project, you CANNOT use that project code how you would like. This is not FUD, this is part of the license design. That is a major change from GPLv2 which is what we are discussing.
A reminder that developers, not users, pick the license of code. That is also fundamental to copyright law. You can write a license that makes developers pay users $1,000. Users might like that. Developers may not choose it. That is what is happening here in many cases. Developers are choosing to avoid GPL for other options.
Again - this conversation would be helped if someone had some data. Anecdotally I'm seeing lots more MIT / Apache stuff than GPLv3 stuff these days.
Is this really necessary? The GPL allows you to read and study the software. Is it really copyright infringement if you take your understanding and make your own program? Are people really forced to come up with convoluted new ways to solve the same problem just to avoid any similarity to existing work?
- Striking similarity, or what most of us think of as infringement, where there's literal copies of someone else's code in your own. This is what things like content ID systems try to detect.
- Access plus substantial similarity. This is where you've looked at someone else's code (source or disassembled, doesn't really matter); and then produced something that looks a lot like that code if you squint a little.
Substantial similarity does not have a hard-and-fast rule associated with it, it's usually something that juries or judges decide. It's basically the "yeah just change it a little so the teacher doesn't notice" meme, but in legal form. If you were to read another program's code, and then make another program that looked an awful lot like the first, then you'd be infringing. But this is bounded by other exceptions to and rules of copyright - notably, functionality itself can't be copyrighted, and copyright over interfaces is largely prescribed by various fair use decisions (so emulation and re-implementation is largely still OK even under this standard).
If you read GCC, and then write LLVM, that's not substantial similarity. That's a different, legally distinct compiler design. The FSF could sue the pants off Apple if merely reading GCC source meant any compiler you wrote was infringing.
If you read GCC, and then write a compiler with the same internal representation as GCC; a parser that's structured the same way as GCC; and optimization passes that are organized the same way as GCC's; then you're closer to infringing.
Your responsibility isn't to create an entirely novel program by any means necessary. It's just to avoid doing the software equivalent of tracing over someone else's drawing.
GPLv3 says 'To “modify” a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a “modified version” of the earlier work or a work “based on” the earlier work.'
If you read and study a work, and then write something new that does the same thing, that may be considered adapting or copying from (parts of) the original work.
I don't have much of an understanding of copyright when it comes to cases like those, but my worry is that declaring that your open-source codebase is merely inspired by codebase X with this specific code in the case of these specific data structures would still count as infringement in a court of law. However, I don't intend to infringe on the licenses of the original, and do not intend to directly copy any code.
Rather than thinking about how to re-implement a thing that doesn't exist, I think about how to implement the tool I actually need. What I end up with may be less general, but it's actual functionality is often simpler and easier for me to grok/remember.
Or maybe rather, what does "use" mean here?
"Use" in this case means to have some feature that's implemented by those libraries reimplemented in an open-source project (MIT-licensed).
I am curious about how that compares with libbfd since that wasn't under consideration for us as it uses GPLv3.
This agrees with what I saw in `top` while testing: with the patch, I see `perf` using ~95% CPU, with `addr2line` using the remaining ~5%.
So speeding up `addr2line` probably wouldn't result in very much of an overall improvement for this workload.
[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815#28 [2]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815#38
"don't want to" isn't quite correct - they can't.
Perf is licensed under the GPLv2, libbfd under the GPLv3. The licenses are incompatible, which makes the combination unredistributable - which is what Debian would be doing with it.
Debian is legally not allowed to do this.
I wonder if a patch to addr2line would be accepted which allows the executable to stay running to accept multiple requests through stdin.
But shelling out to something or in some cases having an automated script create or download the derived work automatically is sometimes acceptable.
It’s a bit of a silly line in some ways. But it’s a long story.
My personal belief is that a court will only care about if your code is a wrapper (e.g, libgpg) or if it is its independently useful and just uses the library (e.g. PostgreSQL using libreadline) for a small part of its operation. But I am not a lawyer and this has not been tested in court as far as I know.
It can't possibly add any restrictions to the use of your project, because people can always just use it under the terms of GPLv2 if they want.
I guess it could remove restrictions, if the hypothetical GPLv4 was a total 180 and looked more like the MIT or Apache licenses, so I guess if that's a concern then you have a point.
If a third party forks a GPLv2+ project, adding features that are GPLv3+ licensed, the result is GPLv3+ licensed.
That means those wanting the GPLv2 licensed project won’t be able to use those new features.
As drran mentioned, I'm sure RMS would say the problem here is that Linux is licensed under v2 only. Linus of course would disagree.
That's a bit disingenuous. Linus has said no to the GPLv3, yes, but his reasons for doing so have merit.
Unfortunately it does not.
I've often wondered why more linux distros don't provide a similar tool to build ZFS into the kernel for you.
Ubuntu just bundles ZFS into the kernel statically - not even as a loadable module - and damn the licensing implications(!).
Take Yosys, an open source synthesis platform that allows external commands by adding shred libraries. It’s MIT licensed.
There is the Yosys-GHDL plug-in that allows using VHDL instead of Verilog compilation. It’s an independent open source project that can’t be merged into the main Yosys tree because GHDL itself is GPL 2.0.
Is the author of that plug-in violating the GHDL license? The plug-in glue code is GPL2 as well.
Can the GHDL authors ask the plug-in authors to take down their code (which has been forked many times on GitHub, of course.)
In this case, perf is (like the Linux kernel, in whose git tree it lives) GPLv2-only, and modern versions of libbfd are GPLv3-or-later, and it's not possible to comply with both at the same time. The GPLv2 has a "no additional restrictions" clause, and the GPLv3 asks for things the GPLv2 does not. So a distributor of a combined system including perf built against libbfd (like a Linux distro) cannot comply with the licenses.
(I suppose perf was written against the libbfd API back when libbfd was GPLv2, avoiding the question of whether perf is an illegal derivative work of libbfd. Or the backtracing API is de minimis, which means the source is fine but a compiled binary as par of a distro that also includes a modern libbfd is not.)
One possible answer here is to ask the perf authors to relicense under GPLv2 or later. Many years ago, at a startup that no longer exists, I wanted to reuse code from the Linux kernel "dm-verity" module in GRUB, which had moved to GPLv3, and I got a pretty quick answer from Red Hat (who was the only copyright holder for those files) saying that would be fine.
In this spirit, is there any legal issue with downloading perf and libbfd separately, compiling it, and running using it for yourself only? I assume not?
What about using it then as a tool within your company? That might be seen as distribution, and thus not allowed?
Debian could even ship you something that compiles perf against libbfd on your own system. This is the approach they take with ZFS and Linux, which are under incompatible licenses (CDDL and GPLv2): https://bits.debian.org/2016/05/what-does-it-mean-that-zfs-i...
A company is generally a single legal person and so there's no "distribution", but https://www.jolts.world/index.php/jolts/article/view/66/125 is a law review article that discusses the complexities here. If you employ contractors, or if the company as a whole is purchased, the answer is apparently blurrier.
That makes it arguably not a system library, so it's fair that Debian would want to steer clear of that interpretation.
It depends on gcc, for some reason, and that depends on binutils?
(and I have no idea if anything here uses libbfd, since it's just a part of the binutils package and not split out)
In a society with different laws, there would be no problem. I wonder what would be greater - the productivity gain when we would not have to make workarounds due to licensing, and when we could just legally access the source code of everything. Or the loss, because people might have less incentive to innovate, or they would just keep the source hidden.
The economic activity & value directly suppressed by IPR is unbelievable. Hard to even mentally scratch the surface. It's extremely economical (for society) to digitally copy and distribute useful information.
Most Western IP laws are draconic in the purest sense of the word.
Unfortunately their is a very influential sect of "corporate-law fearing" IP fanatics at the heart of many FOSS tech traditions (RMS, et al). Not all of it is pragmatic.
IP laws in most countries are also not decided by a democratic process but rather pushed in by trade deals, which also make it almost impossible to change those laws.
E: Downvoters, have you read the article? The program started a new process for each address lookup instead of using a library because that library is GPL.
Shall we have a discussion?
If they don't want to do that, then being authors and owning their copyrights, that's their right to do. That's nothing I would blame on the GPL.
I mean we should be grateful that these authors made their works available free of cost in the first place.
So GPLv3 bans that. That, by itself, is inherently an additional restriction not allowed by GPLv2. They don't need to go out of their way or have a conspiracy to deliberately make it incompatible. Especially since the GPL2, as distributed and suggested, includes the "or later" text which resolves this incompatibility to allow GPL3 software to use GPL2 code. If anything, this encourages GPL2+ as the default GPL license if maximum compatibility is your goal.
Others (e.g. Linus), are more focused on getting the code changes so if they want to use them in the original project they can, and don't feel being able to install it on the device it was built for is as high up on the priority list.
https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://sfconservancy.org/blog/2021/mar/25/install-gplv2/ https://events19.linuxfoundation.org/wp-content/uploads/2017...
Perf does not permit the license to be redistributed as anything except GPLv2, so it conflicts with GPLv3 code.
Linux doesn’t use “or later”. https://github.com/torvalds/linux/blob/master/COPYING:
The Linux Kernel is provided under:
SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note
LICENSES/preferred/GPL-2.0
LICENSES/exceptions/Linux-syscall-note
Documentation/process/license-rules.rst
All contributions to the Linux Kernel are subject to this COPYING file.
But this is an example of the ugly side of open source licensing. A lot of people don't have strong feelings about ensuring the distribution terms of their code, and just don't care how it's used. And in those cases, it can be annoying to have e.g. opensource.org insist that GPLv3 is the best option.
Edit: To be more specific, I mean that this problem could be avoided if both softwares were released under a public-domain-equivelent license. But of course, that will never happen.
I'm not sure how you can rationally make an argument for a framework that literally limits the use of improved or more performant code/technology/understanding simply because it does not share a philosophical view of IP/ownership/sharing.
This is the core of the problem with IP in the real world as well, and I find it mind-boggling that we don't look at the SYSTEM and see it as problematic.
What are they gonna do? Sue you from beyond the grave?
I actually consider putting something into my will that states that all my open-source contributions are relicensed to "public domain" (CC0 [1]) once I die.
[1] https://creativecommons.org/share-your-work/public-domain/cc...
Can you explain how you reach this conclusion? Assuming the GPL did not exist and the author would have made stuff proprietary, would that be better ?
Some people like me would share code with GPL3 or later, why the f* should we give it to you as BSD? are you running on your devices only BSD code or MIT code ? or are you running proprietary software but for some reason making all stuff BSD will make your job easier since you could mindlesly copy paste shit in your proprietary stuff?
That's a false dichotomy – there are many more open source licenses which wouldn't have led to the problem described in the article.
> E: Downvoters, have you read the article? The program started a new process for each address lookup instead
> of using a library because that library is GPL.
>
> Shall we have a discussion?
> non-bfd, without patch: 7m59s
> non-bfd, with patch: 15s
> bfd: 15s
The same could have happened due to other reasons for alternative implementation, that range from availability of more than one implementation with different advantages and disadvantages, different OS the code needs to run on, or simply preference.
Blaming the license in this case is just short-sighted and with the wording used it just shows a bias of yours against the GPL, but not actual will to participate in a meaningful discussion on the linked thematic.
--- You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange;
You can't just give out binaries without source, but you can choose not to release anything. The thing is, as usage models shift to include a lot of what the FSF very reasonably relabels SaaSS (Service as a Software Substitute), you can release nothing but provide the functionality over the network and reasonably expect people to use it that way. Since you haven't distributed the software itself to anyone, the GPL doesn't put you out of compliance for not showing them your code or letting them distribute it further.
This is the situation the Affero GPL (AGPL) was intended to address—it requires (or tries to require) that source code also be accessible to anyone who is given access to the software over a network. (Imprecisely speaking, that is; do check the license text if you want the specifics.)
Heck no. "as a" already means that. Rewriting the acronym is far more petty than reasonable.
“Service as a Software Substitute” pushes the ‘service’ part to be the most salient. It depicts something which is fundamentally a service, where ‘substitute’ once more emphasizes that you do not, in fact, have the software itself, even though it is taking the role of software. The FSF considers this very important, because they wish users to be able to copy and modify the software they use, and pseudo-distribution purely as a service does not naturally allow for this. If that is not something you care about, then the emphasis will seem strange, yes.
And in this case I’m not sure how gpl helps. With gpl2 you only need to distribute source code if you distribute binaries - so they have no legal obligations there. And Amazon isn’t really making meaningful changes to elasticsearch and friends anyway, so having the license require them to opensource their changes is a bit moot.
[1]: https://theceme.org/richard-godden-platform-capitalism-nick-...
You'd need a method of determining revenue and then apportioning revenue amongst the many parts of a system. Then there's transfer pricing issues. Not to mention audit requirements. See also Hollywood accounting.
One of the ways open source gets adoption is because using open source with an acceptable license is often much less hassle than paying for commercial software and complying with commercial software license requirements.
Either way, I don't get paid a royalty for work for hire from my past employers, so I don't expect a royalty from anyone else. And I've not worked on a project basis either; so I'm getting paid for having my butt in the seat and anything that happens afterwards is a happy accident.
I've open sourced some personal stuff too, although I don't know that anyone has looked at it. That stuff is usually more like nobody should need to write this again. Not much commercial market for a PPPoE client that can handoff to a standby machine anyway, but maybe it will be useful for someone, some day.
I've got another project in the works, but it's mostly a bit of glue around other people's open source code. If I wasn't retired, I'd try to get an employer to pay me to write it (and it would get done faster!), but I can't see why anyone would pay for just the software. Consulting on the software, sure; but then again, if you were to rely on it, you'd probably want to cultivate in-house expertise to reduce dependency on outside help.
List and more references: https://github.com/LibreCybernetics/awesome-copyfarleft
Not contribute, but share the sources if they distribute a binary which uses code derived from GPL-ed one. Wireless router vendors used to share modified sources as an archive on some obscure ftp without comments and documentation (so you'll have a hard time building a binary from these sources). It's better than nothing, but this is not a contribution.
Enter MPLv2, with the best of both worlds:
- not viral copyleft
- but users DO have to contribute back
So in that sense it's only very marginally 'better' than a fully permissive license.
The incompatibility is caused by "lack of restrictions" clause. Without that, GPL becomes essentially BSD with all the corporate thievery that entails.
However, in the case of the Linux kernel (which "perf" is distributed with) changing the license is not an option --- no CLA and, even if it was practical to ask so many people for permission, many contributors are dead now. So here the only option is using dependencies which are compatible with GPLv2. So maybe we need a new library to replace libbfd, which would be more permissively licensed.
In absence of copyright anyone could publish binaries build on GPL code without sharing the sources. So a central feature of the GPL would cease to be if copyright was in any way disabled. Saying the GPL is about disabling copyright is about as true as McDonalds being about healthy diets.
See this video of him explaining his take on GPLv3 and the "or later" clause at Deb Conf if you are interested: https://www.youtube.com/watch?v=PaKIZ7gJlRU
Linus does not think GPLv3 is a good or fitting license for the Linux-kernel and that it changes too much things to be considered a new version of the same license as GPLv2.
So he refuses to add the “or later” backdoor which would effectively relicense “his” kernel with a license he does not approve of.
Is that not his right?
MIT, X11, BSD (2- or 3-clause), and more similar ones I can't name off the top of my head: these are the basic do-what-you-want license, the only requirements are things that every (good) license already has, such as standard limitation-of-warranty clauses and retention of copyright notice requirements.
Apache (v2): This is the next stage up, which means that the text is lengthier and somewhat denser legalese, but also covers more topics such as trademarks and especially patents. The patent clause here includes a provision that any patent licenses are revoked if you sue the authors for patent infringement.
MPL (v2): This is a weak copyleft requirement, which means that you must provide any changes to the source code when you distribute the binary and additionally the resultant code must be licensed under the same terms, although it only applies on a per-file basis. EPL, CDDL are broadly similar to the MPL, with a few differences in the legal minutiae.
LGPL: Weak copyleft again, except now it's on a per-library basis. [Although, to be honest, the definition of per-library basis isn't entirely clear for non-C/C++ code.]
GPL: Strong copyleft, which means that you pretty much have to use GPL if you reuse the code.
AGPL: Even stronger than GPL, you have to distribute sources to anyone who uses your code over the network.
With the GPL family, there's a distinction between version 2.1 and version 3 that retains relevance, because some people objected to the changes in GPLv3 (notably the anti-Tivoization clause and patent clause changes) and refused to move to GPLv3, with the Linux kernel being the most notable project to refuse to do so.
As for relation, well, any license more complicated than Apache includes lots of legal minutiae that makes it somewhat hard to render judgement if two licenses are compatible or not. In general, though, you can usually use an earlier license in this list in a project that uses a later license, but usually only if both licenses are the latest version (as the most recent updates added some compatibility escape hatches).
IMO, the gist of the GPL boils down to:
> If you "distribute" a "derived work" of this code, you have to release your code as well (under the same terms.)
AGPL not only interprets "distribute" to mean "offer up as a service over the network", but it also interprets "derived work" the same way. This is why anyone who uses (say) MongoDB in a company basically has to pony up for the commercial license, lest they be required to open source huge parts of their company.
I don't understand this part. Why is this useful?
- https://en.wikipedia.org/wiki/Comparison_of_free_and_open-so... has a feature matrix for about 40 licenses
- https://joinup.ec.europa.eu/collection/eupl/solution/joinup-... has a comprehensive set of features you might want a license to have, and shows matching licenses.
You still would have to read up on what terms such as “trademark”, “copyright” or “copyleft” mean.
Think about all the time that would be saved when people no longer need to think about all this lawyer bullshit.
I don't find either to be useful lines of thinking.
Don't compare this stuff to police brutality, looting, murder. They're not even in the same realm. Intellectual property is just ideas and absurd notions of ownership. It's like trying to own numbers. The real brutality is sending people to jail over this.
Says who? The copyright monopolists?
There's plenty of historical evidence to the contrary:
https://news.ycombinator.com/item?id=28330810
It seems obvious in hindsight. Less monopolies lead to more competition and better products. Even the US engaged in such infringement.
It'd be nice if perf record had a fundamentally faster way of working. I found a nice description of how it works in the README.md for cargo-trace: "perf relies on perf_event_open_sys to sample the stack. Every time a sample is taken, the entire stack is copied into user space. Stack unwinding is performed in user space as a post processing step. This wastes bandwidth and is a security concern as it may dump secrets like private keys."
cargo-trace is apparently dormant now, but I found it really interesting. It does the unwinding via eBPF instead, which should be quicker while recording, not generate as much (sensitive) data, and not require as much post-processing. (Symbolization would still happen in post-processing.)
The idea of a more efficient compressed encoding seems generally applicable. I imagine just piping through zstd would be an improvement over plain perf record directly to a file, but it sounds like your tool's splitting makes zstd more effective. It'd be handy to be able to just do perf record ... | fancy-recompress > out, and even better to upstream the format improvement into perf itself. I feel you on "ran out of interns"; there's always more to do...
But yes, in a pinch piping to zstd has far less overhead than writing directly to disk.
I suppose you could make a library out of that.
reading/parsing dwarf data, on the other hand, is likely to be slow. not totally sure, but maybe i/o could be sped up by mmap'ing in the dwarf data and maybe part of the parsing could be cached?
fun story, i once solved a similar performance regression in a machine learning context, where calling code would serialize an entire model and pass it to inference code, which would then deserialize it all, make one inference, and then tear it all down. if online/streaming is not required, a huge speedup can come from just batching the work.
There needs to be a way to dynamically link a binary into your own address space, and call it's logic repeatedly, without incurring all the process startup overhead all the time.
Pretty much, have all command line utilities be linkable like a library.
https://github.com/AFLplusplus/AFLplusplus/blob/stable/instr...
If you want to try the patch on Ubuntu, I recommend using the Debianish technique of "apt source linux-perf-5.10" (or whichever version of the kernel you're running) and applying the patch. Then "make" in the tools/perf directory, and it should work...
http://neugierig.org/software/blog/2012/03/heap-profiling.ht...