Dstat project ended due to RedHat replacing it with its own dstat tool
github.com
github.com
Stealing the project name was a mistake. The authors are right to be upset by the fact that people used to get their project when they did "yum install dstat" but now are getting a completely different (and not completely compatible) project. I think a lot of the friction could have gone away if they had just named it "pcp-dstat" or something similar, but instead they essentially took this person's namespace without discussion, and really for no good reason that I can tell.
The other issue is that forking a project without even talking with the people running it is a very touchy subject. Ultimately it does look like the project was dead- even to the point where people were asking to contribute but not getting responded to, and the fact that python2 is EOL and python3 support wasn't added until after this drama started shows that the project wasn't really "active". So while I do think they should have reached out, I totally understand why that slipped their mind.
So at this point I think RedHat should apologize and fix the package name.
This webpage listing the contents of the fc30 pcp-system-tools package: https://fedora.pkgs.org/30/fedora-updates-x86_64/pcp-system-...
lists a file called '/usr/bin/dstat'. The pcp-system-tools package obsoletes a package called 'dstat'. If you go to https://pkgs.org/download/dstat, and look under "Fedora 30," you get pcp-system-tools.
[Spoiler: 'yum install dstat' installs pcp-system-tools, which has a symlink /usr/bin/dstat -> /usr/libexec/pcp/bin/pcp-dstat.]
> That said I don't fault you for make a new tool that does all the new fancy things you mentioned above, I do think it's a little sketchy to make a new tool called dstat without at least asking here first. I know that when I hit Fedora 29 and did a yum install dstat and I get a bunch of pcp stuff I was pretty confused.
The way they have things aliased right now, if you type in `yum install dstat` you will get this new program (including all of its baggage).
The decision to do this was made before June 2018, 18 months after the last activity. One of the reasons cited was lack of activity, but that is no thanks to them, I guess.
That is why I am convinced the goal was to replace it from the onset, there was no interest in helping out the project. In fact "no activity" was the right excuse to make their action seem legitimate. Attempting to contact the project could have jeopardized that plan.
They could have just removed "dstat" from the distribution, and added a note that users can now use pcp-dstat. But now they made it impossible for users to add the original dstat. Let alone the support nightmare of having a different tool with the same name. There's no winning this one.
>there was no interest in helping out the project.
They saw that people were filing issues and making pull requests and that those people were getting total radio silence. I can understand how someone might think "well, people are already trying to 'help the project' and the maintainer isn't even acknowledging it, so it's a waste of time".
And while again I totally understand and agree that they should have tried to contact you, you've known about the removal for like 9 months and you apparently never reached out to them either? Like, a couple words in reply to a bugzilla or an IRC message does not amount to picking a fight with a company.
I am sure they made a sound business decision, and I think as a result of that I made the right personal decision. And here we are now having this meta-discussion with people not having a clue. Welcome !
You seem to imply there was no communication, and I stopped the project out of the blue. That is a misrepresentation.
Nobody stepped up to take over maintainership, and I don't see anyone doing that now. But if someone wants to try, I can unarchive the project and restore the PRs and issues.
If not, the king is dead, long live the king!
What they should have done is tried to reach out to you before forking.
They included in RHEL and sold it as part of their product when it suited them, and now they have replaced it because it suited them. That's fine, I don't have to agree.
A: "Whoa, there is no dstat in RHEL8? Customers are gonna complain."
B: "Yes, it's because it's Python 2 only and upstream is dead. Tell customers to use PCP"
A: "Srsly? Nobody thinks of the scripts? Besides, dstat output is so nice."
C: "Well, there is some toy example to print pcp data in dstat format."
A: "Do we really want to tell customers 'there is a toy example'?"
C: "Give me a day or two to pimp it up."
And then C got a bit carried away. :-)
And it's not like I have disappeared from the face of the earth. I am quite active on GitHub.
Changing the name space would have been problematic, as so many people use automation tools now to install package dependencies, rather than break their work because someone has zero intention of maintaining a project, they did the right thing and used the same namespace.
Had this been a case of RH disagreed with the developers direction or any factor around an active project, it would have been a shitty move, but considering the developer ignored any offers to help and let the project to fall into a stale un-maintained state, I don't see any grounds for him to throw his rattle out of the pram and blame RH here.
I assume people offering to help only wanted to see their own contributions merged.
I shouldn't be surprised people here take positions while not having been involved in the project or know the whole history.
In my opinion, stopping the project was the only sensible thing to do at this point. Red Hat replacing the tool with one of their own (which I wholeheartedly disagree with for various reasons) created a dead-end. It's the final blow to a dying project.
Wrt. Python 3 support, I simply did it because it was so easy to do, not because I thought this would be a game-changer. But hey, please do read into whatever fits your narrative :-)
I can also see where someone might not want to endlessly re-debate an old security decision.
CWD is far worse a risk, because it changes every time the command is run. By comparison it could be assumed that a secure install of the software is in a trusted location, and this just gets the parent root of that location. Of course, it’s better if it’s documented, but I see no difference between this and $JAVA_HOME’s approach of ./bin, ./share, etc. It could be argued this is better—no environment variables can override the default. If you install the app to /tmp/bin that’s your own fault...
That said, going up a directory would seem to imply that dstat could easily be loading completely arbitrary code from a directory it has no exclusive claim to and thus cannot meaningfully trust. Indeed, as I understand `share/` that is exactly the point. This really doesn't strike me as being a safe thing to do, and I can readily see why the author would reject it on security grounds. This seems to me to be a distinction with little meaningful difference, though I can see where some people might disagree.
As I tell PMs, designers, and engineers I work with far too often for my own comfort, someone else's poor security decisions are no excuse for our own.
The only bug in this PR that I can see is it might assume the system plugins are more important than the local overrides. Generally you’d put this sort of path in the least priority, to allow for /usr/local to override. That said, obviously hard-coding paths is worse than simply having a default, presumably relative to the current binary/file, and allowing end users and packagers to override/configure the defaults to suit their security preferences, with the default being that the entire package is unzipped with expected relative paths to files... or that’s how I would look at it. The only way to greater security would be to ship a Docker container, Snap package, or similar and mount your own filesystem overlays. :) Or, perhaps keeping a trusted list of plugins somewhere.
edit after your reply: if you install in /tmp, you'll end up with /tmp/bin/dstat and /tmp/share/dstat. you're concerned that an attacker could smuggle something into /tmp/share/dstat, but /tmp/bin/dstat is of no worry? what exactly is the threat here?
> access control of `../share` in an unknown part of the filesystem is a matter for some concern. Given that the binary can be put basically anywhere, it would seem to be perilously close to CWD.
another edit since i cannot reply to you: do you have any examples of the "threat model [which] includes that you can't trust every part of the filesystem you're working from"? something concrete, specific. a particular install prefix that would let you create $prefix/bin/dstat but $prefix/share/dstat would be dangerous.
aaand, see my reply at https://news.ycombinator.com/item?id=19989237
EDIT: A sibling points out the issue in more detail - https://news.ycombinator.com/item?id=19989237
i'm still convinced this is cargocult security.
If so, it could be argued the bug already existed and this patch simply uses the existing behaviour...
You're right though, I didn't see the issue was still there. The home directory should also not be there either. I think I looked at this too hastily but now I'm confused too.
os.path.realpath(__file__)
I know that the word has changed a bit since then and that it's a bit different to appropriate a soul-less corporation's command names vs an individual volunteer.. It just happened to strike me as odd at the moment I came across this thread that we're so up-in-arms about this when we're all happily using utilities that were produced by the very same process every day.
Also, most of those utilities were named things which solely described what they did. It's a bit different to make a new "cut" vs a new "clhodapp".
The "Vixie cron" that you might even still be using is a "PD" clone of the Unix cron program, reimplemented from scratch with the same name. The van Smoorenburg "init" that you might have heard of was a clone of (an at the time sadly already out of date) Unix init program. All of the stuff in GNU coreutils and other similar GNU toolsets was a deliberate reimplementation from scratch.
And then there are the Thompson, Mashey, Bourne, Almquist, Bourne Again, Korn, Z, and Watanabe shells. And BusyBox and ToyBox. And /usr/lib/sendmail . And the "mailx" utility. And "awk". And "vncviewer". And "getty". And so on.
* https://unix.stackexchange.com/a/508142/5132
* http://jdebp.uk./FGA/inittab-getty-is-history.html
* https://news.ycombinator.com/item?id=19988382
> "He missed a big one: you have no way to stop Linux distributions from hacking up your software, and you'll suffer the consequences of whatever they do."
Maybe Debian is more respectful since it's open source not backed by a company with enterprise customers?
(Disclaimer: I work for Red Hat)
By contrast in my personal experience as a developer of something that got packaged in Debian, I discovered later that that package had a bunch of patches added, a man page written (!) and so on, none of which was bad, but also none of which was fed back to me at all.
(In case not obvious: I work for Red Hat on Fedora.)
The change page says "The original dstat utility has reached end of life - it does not support python3 and there are no plans to update it." This may not have been right after all, but I don't think it's in bad faith. Perhaps FESCo should have dug a little deeper, but there aren't really any particular red flags indicating that that's warranted.
I'm open to suggestions, though -- we do want to continually improve our processes, and we do want to work well with upstreams. And, y'know, be better to long-time Fedora ecosystem friends.
And that is fine if it would stay pcp-dstat as-is.
The most upsetting to me is that Dstat is no longer a python tool you can drop on a JeOS, Synology NAS or a WRT router to get it to work, you now have to install PCP and all its dependencies to make it work, which goes against the original design goals. (i.e. we used to support Python 1.5 for a long time to accommodate RHEL2.1 during its life-cycle)
Also, by taking the Dstat code, removing the plugin mechanism and replacing it with a PCP backend, writing a python plugin for Dstat is no longer possible. This is promoted as being a feature as "your plugin is now a config file" which is a bit disingenuous as you have to write a PCP backend which is a lot harder.
And it is not even a drop-in replacement, it only implements the built-in counters, not the full set of plugins (i.e. --top plugins are missing). So the argument that it needs to work as-is for existing customers does not hold true either.
By taking that name (with the Red Hat clout) there is no chance of anyone taking over maintenance without having to deal with 2 products using the same name, which I guess is forcing your wish on other distributions too.
Wrt. the change was properly announced. I bet the checklist was properly checked. Or to quote Douglas Adams:
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
In retrospect, it would have been nice for the devs working on this to contact you directly and explicitly about the proposed change. I'm sorry that didn't happen.
But here we are now -- at this point, what outcome would you like?
I don't have any wishes. Just a bad aftertaste but that will go. I don't like to dwell in the past, regardless of the fact I have to defend myself in this forum to total strangers who seem to know better :-)
And that will pass as well. We'll see...
https://github.com/performancecopilot/pcp/commits/master/src...
But yeah, it's a lot harder to fight Red Hat (soon IBM!) over that if the project is inactively maintained and there's no registration.
Still, whether legal or not, the name squat is not a good look for Red Hat.
Has the name not changed to "pcp", or am I missing something? "pcp dstat" appears to be a command, but that seems like a reasonable abbreviation for "hey pcp, please give me a dstat-like interface".
[Edit] Correction. https://github.com/dagwieers/dstat/issues/156 says the following, so there is a compatibility symlink using the name "dstat". However the project name appears to be "pcp-dstat" or "pcp dstat".
"Since there had been no meaningful work on the old dstat code in years, github PRs were being ignored, community developers requesting access to help were being ignored, and we have customers using it ... we were asked to make this as seamless a transition as possible, provide that compatibility symlink for /usr/bin/dstat to pcp-dstat, and remove the old package from RHEL and Fedora."
This is not true, I have not been contacted. I learned from Fedora's decision to replace Dstat with PCP months after it was already decided.
So it's not like I have had a choice. The choices I have today are:
1. Continue with a project, while Fedora/RHEL is shipping a tool by the same name (with 90% of my code)
2. Rename the original Dstat project, which would be silly
3. Discontinue the project
Option 3 is the path of least resistance. Option 1 and 2 would not bring any joy. At least someone will be paid for maintaining the tool.
https://github.com/dagwieers/dstat/pulls?q=is%3Apr+is%3Aclos...
And as a result I don't see a point continuing a project with the same name.
>>> This is not true
>> GitHub PRs being ignored seems totally true.
> Sure, and Red Hat being paid for RHEL shipping Dstat for a decade could have helped out. But instead they decided to replace it.
It's one thing to be upset due to a belief that Red Hat/others are at fault, but why lie to make your point?
It's pretty clear we didn't accepted any PRs since December 2016 until Red Hat decided to replace our code, which started early June 2018. That's 18 months.
If I am upset about anything, it is this: https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c7
> It's pretty clear we didn't accepted any PRs since December 2016 until Red Hat decided to replace our code, which started early June 2018. That's 18 months.
You are asking: "why didn't RH help out", you also didn't accept any pull requests. That surely would answer the question about why they didn't bother with pull request.
The question is then, how exactly do you think they have helped?
Maybe you should read the announcement and leave it at that.
But maybe people fancy the new PCP Dstat and accept their losses. In any case, money rules the world, Open Source ran by volunteers is dying and becomes less and less attractive.
* https://packages.debian.org/search?suite=jessie&searchon=con...
Red Hat's announcement (posted in February 2019), also cites lack of Python 3 support as another reason for replacing the original dstat, but Python 3 compatibility was added last January.
More color: https://github.com/dagwieers/dstat/issues/156
And given that PCP is using most of the original code, they must have made the same changes to get Python 3 support.
The king is dead, long live the king.
Feature complete, bug free software doesn’t need to have patches released every week. That is the ultimate goal of a small utility like this.
It’s a shame that so many developers mistake this state for “unmaintained”or “abandoned”.
dstat, up until earlier this year (a couple weeks before that announcement), did not support Python 3 - and the author may have just merged RedHat's changes anyways. RedHat's "fork" supported Python3 for months at that point.
Yes, it was a useful tool, and what RedHat did was shitty but I can understand the motivation for not wanting to keep a Python2 install around just for one tool.
I don't use a Red Hat distro but would this really only be one tool? On Arch I see the following packages depend on python2: mercurial [required], git [optional], texlive-core [optional], graphviz [optional], ...
Name : mercurial
Version : 5.0-1
Description : A scalable distributed SCM tool
Depends On : python2
Optional Deps : tk: for the hgk GUI [installed]
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 25.95 MiB1. Some of mercurial's bundled plugins are not Py3 compatible (I recall reading a discussion as an outsider a while back that some plugins are planned to be ported to Rust, not Python 3, but that may have changed) 2. Nobody involved in packaging it for Arch has had time to verify mercurial 5 on python 3.
At the point when people started to work on pcp-dstat, it was clear that if you wanted X in the base distribution it had to be Python 3 compatible.
RedHat 8 that just got released this month doesn't have it. Not sure when is the next release/update cycle for Arch, that will drop it. Keeping in mind that Arch probably doesn't care about doing things last minute or even after the due date.
Especially since RHEL 8 is gonna stick around for the next decade, it's a good idea to move as much stuff as possible to a newer platform that will continue to have security updates at least provided and written by the authors of the platform.
Note: they don't overwrite/replace those utilities on the command line. You can still use the traditional utilities same as it ever was. The pcp versions are called like so: "pcp dstat" or "pcp atopsar".
The big things for our customers isn't just support, it is stability. When you build your systems on RHEL 8 (just released), you will get 10+ years of support and patches for that system. Those changes are all guarenteed to be binary compatible, so the risk of patching ever impacting systems is low. If there is a critical security vulnerability (say Heartbleed in openssl), Red Hat will backport the fix to every supported version of RHEL. That means you get the security fix without (1) having to upgrade the entire openssl dependency and risk breaking something else and (2) without pulling in newer changes that might have their own vulnerabilities.
Going along with the stability, vendors can QA their products against RHEL as a known target.
Last reason for now, if you are in an industry with any kind of compliance requirements, like FIPS, Red Hat jumps through the hoops to get the certification. That makes RHEL the only option in many environments.
Debian never upgrades anything (unless security issue). All packages are frozen at the time of the initial major release. Have fun using curl/htop/nginx/libreoffice from 5-10 years ago, plenty of obscure bugs and missing command line flags.
RHEL actually maintain their system packages.
I have exclusively used the Fedora line in enterprise environments (CentOS, OEL, RHEL).
The point about long term support is definitely #1.
Why should repo activity without context be used as an indicator of anything in discussion instead of focusing on what Redhat has done?
This is openly hijacking an open source project by a billion dollar company because it can and makes a mockery of not only open source colloboration culture but basic professional behavior. Has Redhat reached out to the author, made any requests, tried to work out some way forward, offered to pay for the brand name? Cmon this is simply indefensible.
>Why should repo activity without context be used as an indicator of anything in discussion instead of focusing on what Redhat has done?
"Activity" doesn't just refer to commit traffic. People were filing issues and making pull requests and not getting any responses for more than a year. And furthermore it was a Python 2 tool and the Python 2 EOL date is fast approaching.
I agree that the packagers should have at least left a note on GitHub (even if they thought it would go unread) - but clearly it was not a "maybe it's just mature and nothing needed to be changed" situation.
Are you saying anyone running a Python2 project can expect to have Redhat reimplement it in Python3 secretly and have people defend it on HN? This kind of behavior is simply indefensible.
It's obvious there is a culture clash of open source developers sponsored or working for corporates conflating heavy activity on Github that they are paid to perform as their day jobs as the only open source model. But open source was traditionally about people having other jobs and using their spare time to develop open source without profit motives because they believed in the movement. They certainly did not face corporates and paid developers analysing their frequency of contributions to dismiss their efforts and justify an unethical takeover.
Its based on a entire language which is about to go EOL (Python 2)
Why should he change the project name when he originally have it first?
Also why did you have to frame it as rage quitting?
A simple name change and he can continue to do his work. It’s the idea of turning the other cheek. That or trademark the name I guess.
I had to look this up.
> refrain from retaliating when one has been attacked or insulted.
I don't think he have retaliated in anyway, at least from the github commit message and the blog post. He just decided it wasn't worth his free time to deal with this problem. Maybe there is a disagreement with the definition of rage quitting but it was his open source project and his free time. I don't think it's rage quitting if he wants to quit and do something else with his free time.
Rage quitting have a negative connotation such as flipping table or that scene from the movie Half baked where the dude cussed everybody out before quitting. He seems to have a career, whether you care or not, labeling at such may be detrimental to his career.
Regardless of if you care or not, I hope other people refrain from speculating or do huge leaps of conclusion base on little or no facts that can damage other people's ability to make a living.
Yes, RH should and could have filed and issue saying that they started working on a fork/port-to-PCP, and they intend to provide the dstat executable. But usually at that point, the work is already started, and there's almost nothing to be done. How long RH should wait for that ticket? What if the maintainer says, wow, cool, I'll make a py3 version in 3 months. But RH needs it in 2? What if the maintainer disappears again?
It's not great, but it was abandoned. :(
I don't get paid for doing this work. Red Hat does. They get paid for offering it to customers that apparently demands that it ships with RHEL8. They have been shipping it with RHEL since RHEL3.
They haven't contributed to it ever in that timespan.
They could have paid me for maintaining it, they could have offered to maintain it for me. (Because it is a lot of work to accept and verify pull requests). I am a freelancer, I have to work for a living I don't have a lot of free time.
Instead, they ripped out the plugin backend, added the PCP backend and now you can't use it as a drop-in tool that can't run PCP. Your synology NAS, WRT router or JeOS platform. Because Red Hat does not care about anything other than RHEL. They are business-oriented. And that's fine.
Is the Open Source ecosystem better off ? I sincerely doubt it. It's being replaced by paid-for engineers with business interests. In the long term it is killing the community.
Well... it hasn't been so far. But this isn't the old Red Hat anymore. This is IBM Hat. And for all the talk about keeping Red Hat mostly independent or whatever, I think we all know that's bullshit. Acquiring companies always give that little dog and pony show, and it never holds up in the long-term. There's no way that IBM culture won't wind up infecting Red Hat. Now whether that's a good thing or a bad thing is a question I'll leave to the philosophers. But I'd be very leery of assuming anything regarding what RH will or won't do going forward.
There is more than enough people who have their own minds. That is the RH culture, if this freedom is touched they will just quit.
IBM won’t affect RH in any way, Jim is not an idiot because he knows this would destroy RH and IBM are also not idiots because it would destroy they purchase.
I am not saying that "every decision is mysteriously pushed by IBM." I'm saying that IBM culture will - over the long run - infect Red Hat. To pretend otherwise is, IMO, pretty naive. Over the decades, company after company after company has been acquired, and nearly every time the acquiring company makes the same promises about maintaining autonomy for the acquired company. And sometimes it holds up for a few months, or even a few years. But every single time, at least that I can recall, in the long run the acquired company eventually gets totally absorbed by the parent and starts to take on their characteristics. I haven't heard any cogent argument yet to justify believing that this time will be different.
IBM are also not idiots because it would destroy they purchase.
Like the way IBM managed to avoid destroying Lotus, Informix, Tivoli, Rational, Sequent, Truven, Explorys, Phytel, etc.?