Reimplementing the Coreutils in a modern language (Rust)
fosdem.org
fosdem.org
I love these re-implementations of things. Technology advances by the continued reinvention of the wheel, and even if some efforts end up being of merely didactic interest, it is still important to make them. Coreutils is one of the pillars of our civilization, and thus it should be re-implemented several times on many programming languages.
Looking at the source code, it is impressive that this re-implementation is essentially complete! Look at this: https://github.com/uutils/coreutils/tree/main/src/uu It's only missing some obscure and fringe things.
An important feature of GNU coreutils is, somewhat sadly, not reproduced by this re-implementation: the copyleft license of the original implementation. This takes out some freedom of the users. For example, when I use GNU "ls", I know that I can always look at its source code and change it to my whim (or hire a programmer to change it for me). However, if I use uutils "ls", this is not necessarily the case, for it may be a re-distribution by some middle-men that has modified it slightly and does not provide the source code. I suppose that the removal of this freedom is accidental, because it was a nice thing.
It's not just a matter of having all the individual tools, it's a matter of having them be functionally complete and behaving either the same as GNU or POSIX - note the graph part way down the readme that talks about having only slightly better than half of the tests passing. And the tests probably aren't truly comprehensive, especially for things that people didn't realize were implementation specific or non-guaranteed behaviors. It's good to see them making progress, I just want to someone temper the idea that they're close to done.
(as a side note, I did not want to turn this thread into a licensing flamewar, even if I dislike the license change. It is more interesting to talk about the tools themselves! How does the performance compare to the GNU implementation? Are they faster? Do they use less memory? Are the executables smaller (e.g., when compiled statically)? Do they compile faster?)
Solution: Do not use "re-distributions" by middlemen.
For example, when I use BSD UNIX I get it directly from one of the two original BSD projects. I compile everything from source.
NB. I use both Linux and BSD.
Usually open source routers use a single "crunched" or "multicall" binary for utilties such as those in Linux coreutils, the subject of this thread. The process for making crunched binaries in NetBSD is really easy. Compiling busybox is probably not too difficult either.
The router (gateway) is the most important computer on the network IMHO. If one buys a router that one cannot control, then to me that suggests the buyer does not really care about controlling the computers that use it.
Maybe a computer owner must use a commercial OS that includes pre-installed software. It might be nigh impossible for the computer owner compile that OS from source themselves, let alone the pre-installed software (e.g., humongous web browser).
However a router is a much smaller system, no need for graphics or a web browser, and one can compile this from source relatively easily.
Not necessarily. Some people prefer the MIT/BSD/ISC family of licenses over GPL.
Most of us have employers who are afraid of copyleft. That has made us afraid of copyleft, because we feel like our job is threatened by copyleft. But the erosion of copyleft over the decades has produced phones that track us, websites that control our dating lives, our food, the entertainment we consume, social media that manipulates our emotions and our friendships, and countless other things without us having any recourse or control.
For a while, copyleft was good. It gave us Linux on laptops, where we have the source code for nearly every part of the stack. Without copyleft, we have locked phones, tablets, game consoles, e-readers, refrigerators, cars, pacemakers, and many other devices that are difficult to really call our own because they have software that ultimately obeys the manufacturer, not the consumer.
Well, the GPL forbids me from sharing opensource code between work and hobby projects. Especially GPL3. Thats the worst!
I think its remarkable how proponents of GPLed code insist that GPL is about freedom. The GPL is a list of quite restrictive, opinionated, and frankly ambiguous terms that together make me less free to use opensource code to build software, including other opensource code.
The fantasy of the GPL hopes that all opensource code was GPLed. And then we would collectively create such a large moat that companies have no choice but to GPL their software too. And it essentially requires every single scrap of opensource software out there uses the GPL, so we can make a monopolistic straight jacket for software.
This is a suffocating, and impossible dream because (spoilers): Not all programmers want that! Companies hate it. And opensource programmers like me hate it. I want to work on software that I can use both at work and for fun.
So (of course), we make other implementations which aren't GPLed. And now companies can choose if they want the GPL version or the MIT version and - gosh - they choose and invest in the MIT version. MacOS gets built on top of FreeBSD. LLVM gets massive financial support from companies and it becomes excellent. Microsoft has a competing software stack for just about everything and increasingly thats all opensource too. And React. And Chromium. And so on.
The moat is broken. GPL has failed in what it was trying to do. The GPL probably needs 90%+ of opensource code out there to be GPL licensed to achieve its mission, and yet according to a random google search[1], its actually used in about 5% of actual projects. Nobody is forced to GPL anything these days. The GPL hasn't put effective pressure on companies to opensource their code for decades.
Yet another MIT licensed coreutils implementation changes nothing.
[1] https://gustavopinto.medium.com/why-the-mit-license-is-much-...
FWIW, I do appreciate the part of your argument whereby copyleft lost; I just don't agree about why: my best guess is that the lack of GPL3 back when copyleft has power was the critical lack of foresight that has led to companies like Apple and Google to have the 15-20 years required to build empires on top of the work of open source developers and then use the resources they accumulated to rebuild--part by part--the entire stack until they simply didn't need to rely on the open source community anymore.
And therein lies the problem. Open source developers are already doing a bunch of work for free to benefit others, and this license asks them to voluntarily make that work more onerous. Other than an appeal to their sense of morality (which only works if they happen to be a Stallman disciple), what's their incentive to use it? It's simply demanding more from those who are already giving.
If you are giving away source code, the GPL doesn't ask you to give more. It does require your downstream to continue giving should they take and extend or modify.
It also just isn't something I want to impose on downstream developers--I want people to use my code, and if they make some money, good for them. It's not like they took anything out of my pocket by doing so. Unfortunately there's a class of entitled moralists who think this is the wrong attitude for me to have, and that I should only distribute code if I pledge to enforce their idea of "freedom" regarding its use.
I think that's a bit harsh. If you think the opposite - that MIT/BSD licenses should be preferred over Copyleft - then you should look in a mirror when saying that. Fortunately I suspect it's just your own preference and that's great. My opinion is that MIT licensed code is appropriate for some cases but I prefer GPLv3 as a default.
I'm rather angry at other licenses (MPL and Apache in particular) for existing. Maybe they bring some nuance that I don't understand yet, but their existence creates more options and concern over license compatibility. I think BSD/MIT or GPL/LGPL should be it, the rest IMHO are just fragmentation.
Sure it does. It essentially asks me to give away the ability to profit from my work later or get hired by companies making use of my projects. (Because companies generally refuse to use GPL code at all).
It’s hard enough keeping up the motivation to write and maintain an opensource project. If no one uses it and I can’t find a way to make it financially self sustaining on top of that? Bleh no thanks!
You mean, they refuse to use Linux?
Companies seem happy using GPL software (like linux) only so far as they're allowed to ignore all the share-alike clauses of the GPL.
apparently the notion of Tivoization was lost in a very short period of time
https://en.wikipedia.org/wiki/Tivoization
When I license something under GPL I'm not asking people to give more, I'm forcing them to not take without giving back.
They are free to use my software however they want and to modify it however they want, unless they want to redistribute it.
Plain and simple.
What's so hard about it?
it's the exact opposite.
Apart from the wording, which sounds dismissing for no reason, Stallman has no disciples, people who describe them like that usually have some agenda that does not respect other people's freedoms, since we know discipline is hard, GPL forces people (corporations mostly) to stay away if the just want to free-ride on voluntary work.
Want to use GPL code? No problem! it's yours, no strings attached, there are only 2 things you can't do: put it under a different license, redistribute it without incorporating your changes as GPL.
Which is completely fine morally and ethically, if you come to my house and are my guest for free and I only ask you to not rent the property for money and if you do to share the money with me, would you think I'm appealing to your sense of morality or that I am being fair?
Compare it to, let's say, any other proprietary license where, for example, Oracle doesn't allow you to run the software you have bough on different hardware configurations than the one you bought the license for and try to establish which one gives you more freedom.
There's also, OTOH, a popular trope that says "I license it <whatever free license not GPL> because I have no problem with corporations using it and I can brag about being the author"
Well, that's a pipe dream, the most delusional there is.
Take this as an example among so many that it's really heart shattering: https://twitter.com/FiloSottile/status/1469441487175880711
An hypothetical discussion would go like this
- coder: I wrote this code, hope you like it, it's free to use
- megacorp: you mean I have to pay zero $?
- coder: I mean it's free as in freedom!
- megacorp: can you speak $ please? is it zero or not?
- coder: well, yes, but that's not the p...
- megacorp: then ZERO it is and I will also blame you for any bug or if you do not add the features I need
- coder: but that's not fair...
- megacorp: does the license prohibit it?
- coder: well... no
- megacorp: then unfair it is!
how THIS (which is far from rare or no heard of outcome) incentivize people to keep on coding for free is the real mystery to me! not that the GPL asks people to follow standard human decency and respect for other people hard work. That's completely understandable to me!
Edit: Nice ninja edit. No, I don't use non-copyleft licenses for bragging rights. What is it with GPL ideologues in this thread and ascribing ulterior motives to anyone who disagrees with them? Here's why I don't use copyleft: because I don't give a shit who uses my code or for what. It's really that simple. I guess I don't have enough petty envy in me to care if, god forbid, somebody else makes a buck off my code.
And so what if some corporation blames me for bugs? I can tell them to fix it themselves or pound sand. That's the beauty of requiring nothing from them in return. They have no leg to stand on when making demands of me.
> Other than an appeal to their sense of morality (which only works if they happen to be a Stallman disciple)
> what's their incentive to use it? It's simply demanding more from those who are already giving.
I think you meant more than simply "why I choose not to license my own project as GPL"
you're actually bashing on GPL because you either
- are ignorant of what GPL license says
- have an agenda and/or are paid to spread FUD
- all of the above
If I want to be charitable I would say that the last hypothesis is that you are not aware of what you write.
EDIT:
you also rote
> Nobody is "afraid" of copyleft, that's a rhetorical device you're using to discredit those who disagree with you
Which is not true, a lot of businesses are afraid of copyleft, in fact at work not many people can publish their code under a copyleft license, because the mainstream mindset is that code is an asset and keeping it secret gives companies an advantage. Which clearly is not the case, because most of the code is simply a rewrite - usually bad - of something that already exists in the copyleft World. Everyone working in the field knows that. So it's a decision based on fear not on risk assessment. The company I work for is specialized in risk assessment and even they cannot escape the myth that code must be kept secret or the competition would steal it. I wonder what competitor would steal that pile of garbage that only works under very specific requirements, which are mostly there because 40 years of legacy and incompetence.
You seem to have an idea of copyleft (GPL in particular) that is far from what it is and it only exists in your mind.
Think what you want. Nobody else was confused by my meaning.
> have and agenda and are paid to spread FUD
LMAO you guys are fucking lunatics. Have a nice night.
you are very confused and don't even understand what you wrote my dear friend.
GPL is not based on "asking those who are already giving to give more" but on collaboration. Which I understand is not a thing for you, so you better stay away from it, but that's not a GPL's fault, that's exactly the purpose: keeping away the free riders like you!
besides "nobody else" is a very poor defense, only kids use that. It doesn't make your statements less false.
> LMAO you guys are fucking lunatics. Have a nice night.
Who are you talking to now?
I am one.
Reminder: the World is vast, USA is just a small percentage of it, not even one of the most interesting by any metric, it's lunch time here.
Sleep on it, if tomorrow you still see the invisible "guys" you mentioned, call a doctor. You sure you're ok?
"Do what the fsck you want with your code; as long as my code and any modifications to it are available under the MPL, I don't give a sheet what you do with the rest of your code."
Also, when another license is incompatible with the GPL, I see it more as the GPL being at fault, because it's literally only compatible with itself unless you explicitly add exceptions. It doesn't matter if another license grants the exact same freedoms as the GPL, it will be incompatible with the GPL because it has a different name.
so if you chose BSD you are exercising your freedom, if people use the GPL the way it was intended, they are lunatics or ideologues.
Nice try, little "Nazi schatze" who "fight for fatherland" (cit).
> what. It's really that simple. I guess I don't have enough petty envy in me to care if, god forbid, somebody else makes a buck off my code.
except that there are INNUMERABLE cases of people having created very successful projects (please provide proof that your code served more than yourself) that are lamenting the fatigue of maintaining it while commercial entities make a lot of money out of it, without giving back, not even pennnies, let alone workers or code.
That's why GPL has its merits, it prevents this kind of behaviour which is to be expected from corporations and other private for profit entities.
If you think that's ideology and your case will be different, you're simply delusional.
As soon as 20 people will start using your code and file bugs, ask for features, open issues on something that is not related in any way to what you wrote, because they did not understand it or were told to nag you, you'll lose your shit. 100% guaranteed.
For what?
Hi, people have made money using my code and I also don’t care. I’m happy more people are using my software. I like the bragging rights and I’ve gotten plenty of work off the back of my opensource work. I understand this is upsetting to you. If you don’t want people using the software you write for free, maybe don’t write free software? That’s sort of the point.
There is a problem of freeloading in the opensource world - we need better ways to fund opensource code. But I don’t see how the GPL is a viable solution to that problem. Not at all. More likely, if I release my code under the GPL not only won’t I get paid, but also nobody will use my software. That sounds like a strategy which just makes everyone lose.
looks like everyone's missing the point.
> I understand this is upsetting to you
Again, maybe I am on another level of comprehension, so I don't understanda why it is so hard for someone to get it, but I am not upset by that, at all.
I simply know that those who think "it will be fine" are delusional and don't know what they are talking about!
So I just will paste some link to relevant news here, maybe it will make things clearer.
It includes the opinion of Antirez, father of one of the most successful OSS ever: Redis. Maybe his words will open your eyes and tear the veil of Maya.
(spoiler ahead alert!)
Basically you work for free and people don't even thank you and the maintainer ends up being doxed or blamed or pushed aside and in the long term the only solution to keep sanity is to resign
https://www.jeffgeerling.com/blog/2022/burden-open-source-ma...
https://www.theregister.com/2022/01/13/opensource_apacheplc4...
https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-...
https://old.reddit.com/r/linux/comments/z14tt2/reason_why_op...
https://github.com/isaacs/github/issues/167
http://web.archive.org/web/20221217180915/http://antirez.com...
> But I don’t see how the GPL is a viable solution to that problem. Not at all
GPL doesn't permit freeloading, that's how
Do you wanna know who the freeloaders are?
Just make a list of companies that do not accept GPL code.
Those are the free loaders.
For people it's even simpler: you don't want to contribute to the project because it's GPL? then you are a freeloader.
Which is not the same of saying that if someone creates a project under Apache license he's a freeloader. There are many reasons to start a project under more permissive licenses, but if you plan to write something that has chances of being successful, think about what you're doing and who got your back.
It's one thing to create Go and make it opensource with Google backing you, another entirely to maintain log4j or GPG or OpenSSH on your free time for years or decades, without even a thank you and people constantly opening issue like "this thing is shit it should be rewritten in Rust" or "this project doesn't have a COC/ the COC is not inclusive enough I will blame you all over the internet" etc. etc.
My Apache/BSD projects are under that license because I know it's code that will be used in a context where GPL would not be accepted, but I also refuse to offer any kind of support whatsoever, basically once you get it, it's yours, I won't even close your issues, I will simply ignore them, that's how much I care about it.
Because I don't care to work on stuff that people are not forced to contribute back to, unless it's for myself.
Need a feature? Show me the money and I will think about implementing it.
> . Not at all. More likely, if I release my code under the GPL not only won’t I get paid, but also nobody will use my software
textbook straw man
you will get paid in code.
Linux is GPL, Vim is GPL, Emacs is GPL, GCC is GPL, Gnome is GPL, KDE is GPL, OpenJDK is GPL, Telegram is GPL, VLC is GPL, Blender is GPL, uBlock Origin is GPL, etc. etc. it's notorious that nobody uses them...
If users want to have freedom, they're free to develop software on their own time or their own dime.
Well, apple built on top of FreeBSD instead - which already existed and worked great. And Microsoft - the most important competitor 20 years ago - just happily built their own competing implementation of … well, everything.
The moat wasn’t effective long before Google and cloud services took over. The lack of forsight around needing gpl3 was just a nail in the coffin.
BALONEY
it's hard to even claim that copyright law itself takes freedom away from you. But in the context of copyright law, every license including BSD/MIT and GPL grants you freedoms; none of them takes freedoms
You still retain copyright unless signed to transfer it. So if it's your own code you can share. Many projects are even dual-licensed so they can develop in open source manner but also be possible to be sold for commercial use. Now if you just want to take code from someone's else GPL-licensed project without sharing your own, well, unfortunately not possible.
I won't buy a house in an HOA. I'm not afraid of HOAs, I just have better things to do with my time than keep up with their demands.
I just had a back-and-forth with someone who outright lied about the legal enforceability of the GPL and it's all beginning to leave a nasty taste in my mouth. The BSD/MIT world is good, but some of its boosters are beginning to look like thugs for companies that profit off of unpaid labor.
Edit since I can't add new replies for some reason:
The anti-copyleft FUD I'm talking about is here:
It would lock them out of profitable business models, so they make a perfectly rational decision, as a business, to not use it. You can call that fear if you like, and I can call that "abuse of semantics" :)
> I don't want to be unpaid labor, I don't want people to recast being unpaid labor in terms of promoting freedom, and I damned well don't want people calling the licenses that keep me from being unpaid labor "fascist" as they did down-thread from here in this very discussion.
That's completely fair. If you develop open source code and you want to license it as copyleft, go ahead. It's your code.
The flip side is that if I develop an open source project, I'm free to make it MIT or whatever because I think the contributions of large companies to my project are valuable. Yes, they can make enhancements and not share them with me. But in practice, they will often merge them into my project because maintaining their own fork is expensive.
On the other hand, if I have to make my code copyleft because it uses your code, they will almost certainly not use or contribute to my code at all, because that would require theirs to be copyleft as well. Therefore I don't use or contribute to your code.
My decision not to use copyleft code isn't based on fear. It's because I know it will take my project in a direction I don't want it to go, and make it unusable for people who I would like to use it.
As for being unpaid labor, that's a personal choice as well. I don't care about it, personally. I'm working on my project either way, because I want to. It's no skin off my nose if somebody else finds a way to make money from it. The minute working on it becomes a burden, I'm free to stop.
I'm fine with people stating they don't use copyleft licenses.
I'm not fine with people spreading FUD about copyleft licenses.
I'm, in a broader sense, not fine with people spreading FUD that Just So Happens to align with the business interests of companies.
>>> I suppose that the removal of this freedom is accidental, because it was a nice thing.
>> Not necessarily. Some people prefer the MIT/BSD/ISC family of licenses over GPL.
> Most of us have employers who are afraid of copyleft. That has made us afraid of copyleft, because we feel like our job is threatened by copyleft.
There's no anti-copyleft FUD here. All that's here is someone assuming someone else isn't a fan of copyleft because of "fear."
In short, you're arguing against anti-copyleft FUD, and I'm arguing against anti-non-copyleft FUD. Both are important for a FUD-free discussion.
Good heavens, you're still going on about this two days later. There was a disagreement. No one "lied".
This is not appropriate behaviour.
Here's another analogy: I ate eggs instead of cereal for breakfast. I did this because I wanted protein and not carbs, not because I'm afraid of cereal. Likewise, I just don't want to do the things that copyleft requires, so I use MIT and the like instead.
I also don't really think most of the examples you cite are all that strongly related to "erosion of copyleft" (if such a thing has happened in the first place; I'm not so sure it has – I'd have to check license counts of packages but BSD licenses and MIT license have been popular for many decades).
That sounds a bit extreme, do you feel that way in all things (murder, theft, slavery,...) or only in terms of limiting the harm entities can cause through distribution of closed-source software?
Personally I feel like forbidding things is fine. It won't prevent bad things from happening, but it can protect us humans from our own psychological weaknesses. I think of antitrust legislation as an example.
Painting it as some kind of irrational fear is misleading and counter-productive. It is perfectly logical for Apple to forbid GPLv3 code.
Quite frankly, GCC committed suicide by switching to the GPLv3. Both Apple and Google have entirely abandoned it. Apple invested tens of millions of dollars into building Clang as a direct result of the license change. And now even the BSDs and many embedded Linux platforms have switched to Clang, leaving desktop Linux and a few esoteric systems as the only remaining use cases for GCC. Yes, GCC is still around, but no serious company will invest in GCC for anything new ever again.
I'm pretty sure we've had this discussion before on lobste.rs, yet you continue to mischaracterize and lump all opposition of copyleft into some irrational position based on fear, or that we somehow can't think for ourselves because our employers don't like copyleft.
My full position: https://github.com/BurntSushi/notes/blob/master/2020-10-29_l...
We've gone full circle.
Copyleft is very nice but I think the competition that liberally licensed software means to proprietary alternatives is way higher than the competition that liberally licensed software means to copylefted software. Meaning, if there is GNU coreutils around but no liberally licensed alternative, there will be a market for alternatives that proprietary vendors can use. Thus, somewhere out there will maintain a proprietary coreutils library and charge money for it. This code base will always exist as a risk to the GPL licensed coreutils package. If there is a free (as in money) alternative however, the proprietary vendor will have a much harder time to convince some company to buy their product.
I think copyleft shines the most when there is a lot of competition with proprietary solutions. Without copyleft, the competing solutions could just incorporate improvements of some package into their own. With copyleft, the copylefted software package has a chance to fight. But in a domain where Free Software has won, and I would consider coreutils to be such an environment, one can also have open source software.
And where is that?
Coreutils might not be important enough for people to pay for its maintenance, as is seen by people leaving the proprietary unixes. Idk. That sentence was mostly there to explain my more general argument.
BSD is not proprietary
In practice, when the Open Source distribution is the dominant implementation, the source tends to be available and modifiable even when the source is from a for profit company. One might look at all the .NET stuff from Microsoft as an example. Clang and LLVM would be other great examples. For the BSD license, the operating system family it was designed for has been available as Open Source since before the GPL has existed. You can still get their equivalent of the core utils in source form today. In fact, there is a Linux distribution going Alpha very soon ( Chimera ) that is going to use them.
If most distributions were to adopt Uutils ( this Rust reimplementation ), I fully expect that not only would the source remain available for them but that there would remain essentially a single version that they would all share. I do not think that the GPL is require to maintain that momentum.
The big difference is that some commercial company could take this and extend it as their own as you say. In that case though, they would likely just have gone their own way anyway which would mean less sharing, not more. You could argue that MacOS for example is what is wrong with BSD as they took some BSD stuff and have not shared everything back. They have shared some stuff though, likely more than if there was no BSD code to borrow. With the BSD code, it seems very unlikely that MacOS would have used GPL.
Anyway, it is all just opinions and preferences. I may not be any more right than you. Who knows? What you are saying though is not 100% obviously right to me either.
You are still free to go and work off of the upstream version of uutils.
Copyleft fanatics feel a lot like libertarians thinking they need to bring a utility belt full of weapons for their trip to the supermarket.
[0] https://github.com/NixOS/nixpkgs/pull/116274
[1] https://github.com/NixOS/nixpkgs/pull/116274#issuecomment-85...
But it's also hard to communicate what I mean by that without diving into a bunch of detail on some case where it proved useful to me.
The ability to give uutils a shake at scale is a paragon.
At some point I imagine it'll be able help ferret out a good fraction of remaining long-tail issues with oilshell's mostly-bash-compatible `osh` in short order.
I did a little cleanup on their `dd` tool a while ago[1], and I would be pretty cautious about actually relying on it anytime soon.
https://www.cs.dartmouth.edu/~doug/reader.pdf
Lots of cool details about stuff like the gradual migration of the kernel from 100% assembly C and the origin stories of a lot of these ubiquitous utility programs like cp, ls, etc.
edit: still got it :)
What's the status of upstream GNU coreutils' attitude on incrementally rewriting in a safe language like rust? Isn't one of the selling points of Rust that it can incrementally be adopted in a C source tree?