An M1 for Curl
daniel.haxx.se
daniel.haxx.se
I make a (closed-source but completely free/ad-free) minimal macOS app for Hue, and every other day (sometimes a week goes by) I get a lovely price-of-a-coffee-sized donation from someone in the community. And when I ask for their name to include them in a credits or supporters panel, most don't even want me to mention it or credit them.
One time, I struggled to replicate an issue for which I didn't have the necessary hardware (multiple Hue bridges), and the bug reporter simply offered to buy and send me "a bunch of bridges and lights" to make debugging easier. I couldn't accept it, but it's gestures like that, combined with positive feedback from people wanting the project to keep going and succeed, that are such a tremendous motivational driver for working on "free stuff". I had totally underestimated that (coming from the SaaS side).
Why not?
So I only accept substantial donations if I'm very sure I'm going to be able to do a substantial amount of work on the project going forward.
The sender didn't reply to messages.
I really wasn't sure what to do. And without a document saying it was a gift from the sender, I had to give half away as taxes, and then there was no possibility of returning it (I wouldn't be able to reclaim the taxes).
Recently I donated my old top-spec race MTB to a bike club, under the condition it was to be gifted to one of the Ukrainian’s who’ve been displaced and now live in our community. Nobody knew I had the bike. Within a few days I got a message back saying there was a teenager that loved racing bikes back in his home country, and can now look forward to the 2023 season with additional financial/logistical support from the club.
I’ll probably bump into them on the local trails and seeing their beaming (and muddy!) smile is worth a thousand times more than publicly being connected to the donation.
To my mind "this dev tool doesn't work, only on Apple, only on Apple's flagship chips" is the opposite of what they'd like to see happen.
Remember, Steve Jobs was good friends with Larry Ellison who famously said that open source is there to be exploited. These guys all think the same way.
There are plenty of licenses which are free only for non-commercial use. The author choose the “free for all” license for a reason though.
> Yet, there are many curl users on the different Apple platforms and sometimes these users run into issues that are unique to those platforms and are challenging to address without direct access to such.
In other words, it's to Apple's own detriment if the author of curl doesn't have access to Apple hardware.
curl is offered for free, and the consequence of that is the author of curl is under no obligation to support Apple platforms or shell out money for Apple hardware.
First sentence of the article: "A generous member of the wider curl community stepped up and donated an unused Mac mini m1 model to me to be used for curl development."
Edit - Looking at https://opencollective.com/curl, the project received $64,000 in donations, and the developer is employed, full-time, by a company to work on cURL. You'd have slightly more of a point if this were not the case, however as it stands it looks like a whine about Apple.
If you read the other comments on this submission, it's not just my opinion.
> Doesn't make it any more (or less) valid than the question I'm posing; why is the developer entitled to Apple providing hardware for testing?
That's the wrong question. The developer is not entitled to Apple hardware and never claimed to be entitled.
The issue is explained in my earlier comment in this thread: https://news.ycombinator.com/item?id=34183736
To repeat myself:
> Yet, there are many curl users on the different Apple platforms and sometimes these users run into issues that are unique to those platforms and are challenging to address without direct access to such.
In other words, it's to Apple's own detriment if the author of curl doesn't have access to Apple hardware.
curl is offered for free, and the consequence of that is the author of curl is under no obligation to support Apple platforms or shell out money for Apple hardware.
> You develop software for MacOS, do you expect a free Mac?
This is a straw man. Apple does not ship my software with macOS, unlike curl. Also, I make money selling Mac software, unlike curl.
You can only use it for iOS apps if you have Apple hardware. I am one of the maintainers and thankfully I got a Mac Mini donated as well when I lost my company Mac.
If I wanted to donate $50 to a single person who has developed a critical bit of open source we as a company gas relied on for ten years, it would be a major hassle.
There's no process for buying a laptop for an open-source dev who's code the business depends on but who is not an employee, so especially in German companies, you don't do that.
You would usually proceed as a donation from inventory or you'd give them some type of credit for purchase (might come off the marketing dept)
When donating something never go with the more expensive option (in this case, actually going and buying something you make on the market)
From a corporate accounting perspective, a "purchase" is "we pay money and we get a thing". Sometimes the thing needs to fit a stereotype; "a hardware", "a service", "a contract", but the dance is the same; request a quote, get a quote, place an order, get an invoice, pay the invoice, get a receipt, get confirmation of delivery.
Iff the transaction fits that mold, then it's generally straightforward. If it doesn't, then you need to find someone with the power to override the organisation; in an organisation resistant to internal tampering ("corruption") that can be quite difficult.
I wish we could just buy licenses sometimes ... I really hate fucking around with procurement processes
> We don't donate to OSS software which we use, because we're legally not allowed to.
> I routinely send key projects, particularly smaller projects, a request to quote me a commercial license of their project, with the explanation that I would accept a quote of $1,000 and that the commercial license can be their existing OSS license plus an invoice. My books suggest we've spent $3k on this in 2015. My bookkeeper, accountant, and the IRS/NTA are united on this issue: they don't care whether a software license is OSS or not. A $1k invoice is a $1k invoice; as a software company, I have virtually carte blanche to expense any software I think is reasonably required, and I think our OSS is reasonably required.
> I would do this more often if OSS projects made it easier for me to do so. Getting me to pay $1,000 for software is easy; committing me to doing lots of admin work over the course of a week is less easy. Take a look at what e.g. http://sidekiq.org/ , which is an OSS project with a commercial model, does. Two clicks gets me to a credit card form. If I actually used Sidekiq, Mike would have had my credit card on file the day that form went up.
Of course, Patio11 was running his own show, and a larger company might demand more, but it is worth asking.
1. For the given amount of $5000 for the period 2023-01-01 to 2023-12-31 you are entitled to
2. Create a feature or bug request on our support system
3. Have a first response back in 96h
4. No warranties explicit or implicit that we'll be able to solve your problem blah blah blah etc
Way too risky.
But yeah, it should be in their best interest to reach out directly to the developers of software like curl, zsh, git, Python... Everything they've been shipping as a part of their OS for 20+ years.
https://developer.apple.com/library/archive/documentation/Ne...
But it's a big company, so presumably some team used curl for some project, leaked that in an error message, and whoever runs the Apple Support social media account apparently didn't know about that after someone contacted them about the error message. Or something like that, it's hard to figure out what was going on without more context.
Any time you see a behemoth like Apple not doing something it's because they don't want to.
They don't consider whatever gains they get from it to be worth the time and money.
The numbers make me think it's using the 'efficiency' cores, or might indicate a lot of memory pressure issues (which seems unlikely at 16GiB)
There's a better write-up about compilation performance here: https://build2.org/blog/apple-m1-compilation.xhtml
For macOS there is a `taskpolicy` command to set the usage of efficiency and high performance cores. I'd expect that it's using high performance cores by default but you can force it by running `taskpolicy -c utility <program>`.
You can also force it to be run on efficiency cores by running `taskpolicy -c background <program>`
In raw performance, M1 cannot match workstation-class CPUs. Since Apple was almost exclusively offering only ultra-mobile Intel chips with horribly dated GPUs, M1 was a giant leap in their offering, but in the desktop PC market there are more powerful chips available. And even some old ones can perform quite well.
I have an M2 Macbook air (IE; fanless little thing) but it completely desicrates the memory of its predecessor (Precision 5520 w/ Xeon E3-1505m v6) which has a 45W TDP.. (IE. double and a bit).
In single core it continues to readily beat my Theadripper 3970x workstation which has a 280W TDP... (more than 12x the TDP of the M2, though it definitely beats it in multi-core...)
I think something else is at play here. Be it a terrible filesystem or (as mentioned in another comment: the use of efficiency cores for compilation).
> time make -j
real 0m8.716s
user 0m29.786s
sys 0m15.102s
> time taskpolicy -c background make -j
real 0m42.196s
user 1m13.041s
sys 0m30.957s
One thing that _is_ notable is that it isn't really managing great parallelism (particularly visible in the efficiency core one).
(Only looking at the actual build, as IME autoconf has remained reassuringly slow for the last 20 years, in defiance of all progress in CPUs, so I'm not at all surprised by that one...)
Is your RAM running at 3600MT/s (i.e. IF @ 1800MHz)? If not, that's 30% loss for some workloads.
Also, compilation is a multi-threaded process. It doesn't make sense to blame things on single core perf.
I’d bet that 90% of the issue is that make, /bin/sh, and autotools all use fork() on macOS instead of posix_spawn() (I think that’s still the case…), and fork is slow on macOS. But not quite slow enough to get all that fixed by default.
Performance doesn't scale linearly with TDP, and that's always been the case. Take any laptop vs desktop CPU, and you'd see the same ratios at play.
Nonetheless, the M1 will most definitely be drawing much less power, its essentially an Atom processor after all. Its just gonna be hard to really put a number on it - even if you could read out the consumption from the BIOS, as they're doing things differently (integrated GPU and SSD controller for the apple silicone chips etc)
are you certain you arent swapping these? Intel CPUs are famously running up to 100C before stopping their boost clock.
One need only watch a single episode of Linus Tech Tips.
people have reported barely any cooling requirements on the M1 and M2s comparitively.
Its a quiet old model, and consumer CPUs of that generation where generally targeting ~60-70°C. Nowadays they're all going for 95-100°C, wherever its Intel, AMD or Apple
Having said that though, as someone who got an M1 MBP (10-core) a few months ago, I also found that CPU-wise (ignoring memory/cache bandwidth, which is much faster on the Mac), it wasn't really any faster than a 2012-era dual-socket 4 core Xeon at 3.3 Ghz for compiling and raytracing (the latter doing lots of FP calculations which I'd expect the M1 to compute OOO really well with how wide it is) when the code was CPU-bound. The power usage is very different though obviously. Hyperthreading also helped the Xeon quite a bit.
Battery life is very impressive though, and haven't really heard the fans yet.
That performance level is maintained when the laptop is unplugged too, which is hard to find in a laptop. Most of them will throttle themselves pretty hard the moment you’re no longer tethered to a wall because otherwise they’ll burn through their batteries in a flash.
I understand from the CPU model that the main development machine of the author is desktop with decent ventilation for the hot and power hungry i7 CPU, which makes a huge difference.
E.g. thinkwiki says its an i7-2640M , in an X220T laptop from 2011. Passmark gives it a single threaded rating of 1516. The current best CPU I can see in a thinkpad with a quick lookup is a i7-1185G7, with a single threaded rating of 2843. That's almost double, but we're comparing with a system of 12 years ago.
Laptops have become a lot more power efficient, GPUs, multi core USB and WIFI are better. But the i386 days where you could more than double your CPU performance after 1.5 years are clearly over. That x220, if upgraded with SSD and 16GB ram and a new battery still does quite well today, and I paid about €500 for it over its whole lifetime.
...but that would deflate his ragebait post.
You're welcome to confirm your assertion at anytime thanks to Daniel and nearly 1k others generosity. Feel free to let us know what you find.
If you want your employer to contribute, check your expense policy. Most companies have a threshold below which you don't need a receipt or even explicit approval. At my last company, that was $50 so I convinced Daniel (of curl) to add a $25/month level specifically to take advantage of it. :)
For example Apple sent patches to some open source projects to add support for Apple Silicon: https://twitter.com/markvillacampa/status/127520044676491264...
That must be the only surviving OCZ Vertex4 in existence.
Their engineers are prohibited from contributing back to the ecosystem so it's not possible to engage with them in a reasonable manner.
They don't acknowledge popular open source projects. Is it too much to ask to even have a single engineer talk to popular projects like Homebrew?
They make us pay to build on their platforms. CI is more expensive, harder to get access to. Emulation is prohibited.
Everywhere I look, it looks like they are a negative force. Sucking out good will and not giving anything back.
They've donated $1,000–$4,999 to The FreeBSD Foundation this year[1] :-)
[1] https://freebsdfoundation.org/our-donors/donors/?donationTyp...
The amount is a lot less than I expected, given the size of Apple's pockets.
Things have changed and this is no longer true. A significant number of projects receive frequent contributions from Apple engineers. The majority of active committers on Apache Cassandra are at Apple for example, contributing major features such as a new transactional system[1].
My team actively contributes to Spinnaker, and we're currently hiring someone with a focus on open-source contributions, quoting the second "key qualification" in the job listing[2]: "Interest in developing with open source software and contributing back to the community".
[1] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15...
[2] https://jobs.apple.com/en-us/details/200361051/apple-cloud-p...
That's a bit odd phrasing, that has guilt undertone. Individuals are unlikely using open source for personal financial gain, but somehow this sounds like they actually "owe" something, kind of shifting the blame from corporations that are actually making money with open source software and often not paying contributors anything.
Also hiring someone to contribute to open source, makes that person likely represent interest of the corporation rather than the project they will contribute to and these are not always aligned. It also takes time of maintainers to actually review and gauge value and viability of the contribution rather than actually spending time on development of the project.
I think much better idea to "contribute back" is to actually pay the existing contributors rather than having own pet one.
> hiring someone to contribute to open source, makes that person likely represent interest of the corporation
That's true, and this is why my team is active within the project by having a member on the Technical Oversight Committee and several others in SIGs. The point is not to push our own agenda, we have made a lot of changes that are mainly relevant to our use cases and that don't necessarily make sense to upstream. Whenever we work on features that we know others would benefit from, we engage with the project's relevant committees and make the changes requested in pull request reviews.
To be clear, I don't speak for my employer and this is only my own experience. Not all teams are like this.
> I think much better idea to "contribute back" is to actually pay the existing contributors rather than having own pet one.
Contributing features and supporting existing contributors in other ways are not incompatible. We have a whole team working on this code base full time, and so do other companies. Of course all of these people collaborate. This project is not really run by volunteers but mostly receives contributions from a number of companies (originally created at Netflix, large parts came from Google, etc).
edit: See TOS members here for a list of the companies involved: https://cd.foundation/blog/2022/11/01/new-spinnaker-technica...
...
Swift?
LLVM needs to be extended to support their platforms and it's too expensive to maintain a fork.
WebKit was forked from a LGPL project (KHTML) so they have to keep publishing the source.
Edit: I was wrong, twas the Goog, in fact.
https://wiki.openstreetmap.org/wiki/Organised_Editing/Activi...
Firstly, they have a site that lists some of their work https://opensource.apple.com/ but going further, projects like Chrome wouldn’t exist without Safari having forked KHTML and then massively worked on WebKit. Clang + LLVM would likely not exist in their current form either (clang started at Apple, llvm was earlier but a lot of early push came from them). Which in turn would have changed the landscape today regarding other languages like Rust.
Those are just two of the major ones. There are other major ones like OpenCL that come to mind too, let alone other projects that get code contributions (OBS, Blender, etc…)
They probably don’t contribute as much as they could , in an ideal world, but the foundation of your comment is untrue.
The author didn't mention reaching out to Apple for help? My opinion is if Apple documentation call out the use of curl as part of a workflow or enabling step, or actually as part of a product, then there is a moral duty to help out (not a legal or license duty).
I would prefer to see the big tech companies contribute to a foundation who could redistribute funds to key open source groups. A credit for a cloud instance of the machine would be sufficient in most cases. It would also remove undue influence of one vendor over another. Maybe such a thing exists, and I would welcome if someone pointed it out.
May I invite you to descend off that hobby horse for a moment ?
Let us perhaps replace the word "Apple" with a variable parameter, say, $VERY_LARGE_CORPORATION
Lets say you turn up at said $VERY_LARGE_CORPORATION and start talking to them about something wishy-washy, intangible and wholly subjective such as a "moral duty".
You will be laughed out of the room.
Once a company grows to a size where employees are measured in the tens or hundreds of thousands then it is simply impossible to get involved in edge-cases and subjective matters. Everything has to be defined and measurable because otherwise the whole caboodle descends into utter chaos very quickly.
Take "good PR"; it's readily understandable. Why won't Apple, or any other company that won't mind spending $2k on good PR among the IT crowd, pick up the lead? Any PR department worth their free lattes should.
Let me introduce you to the group of companies that belong to https://www.bcorporation.net/
They very much care about things like "moral duty". And there's more of those that aren't officially certified members.
Whilst I have heard of B Corporation, I will admit I don't know a great deal about it/them.
However a brief look round their website suggests to me that what they do is basically structured ESG (Environmental, Social and Governance).
ESG is somewhat different to what we are talking about here. ESG has become a hot-topic in the corporate world, especially for listed companies where investors can now screen for companies based on ESG metrics.
Over-simplified definition of ESG is treating your staff well (working conditions etc), treating your immediate community well (not polluting rivers etc) and contributing to good causes through corporate philanthropy.
The whole point is ESG is about how corporations interact with the humans in their operational environment.
BUT You cannot extrapolate ESG to open-source software, because ultimately, if you look at it through the corporate eye, software is software. And the corporate eye looks at software through the legal (licensing) lens, because that gives them something tangible and measurable to work with, which is the preferred modus-operandi of the corporate mind.
Therefore if a piece of software is openly published with an extremely permissive license then you cannot blame the corporation for taking the path of least resistance (i.e. using it inline with the written terms of the license, which, depending on the license wording, could be little more than ensuring a credit is written in the correct places).
Again, this comes down to the problem with scale. If you have tens of thousands of employees, software licensing is difficult, and if you had to "do deals" for each and every piece of open-source software then it would very quickly become unmanageable.
Equally, a typical corporation with thousands of employees, could easily use thousands of pieces of open-source software. Because there is no price-tag on open-source software, you would then be in a position of having to engage in lengthly discussions that involve multiple departments in order to agree on what to pay each and every open-source developer. It all becomes very messy very quickly.
Don't get me wrong, I see where you are coming from on this.
But to be perfectly honest, OSS developers are not entirely without blame here.
Sitting there, releasing code on an overly permissive license and then wondering why nobody is paying you .... it's sort of a case of dream life meeting harsh reality, really.
OSS developers live in the wishy-washy world of subjective views and "moral duty".
Corporations need standardisation and clarity because otherwise they descend into chaos.
The easiest way to do that is through less-permissive licenses that have a clear requirement for remuneration.
Otherwise, if you start talking about "donations" or anything else, then you start walking into that subjective ground again. And especially in terms of "donations", every man and his dog thinks corporates are "rich" and can afford to donate to their cause. So your request for "donations" will be lumped in the in-box along with everyone else's.
Thats just the easy obvious answer, there are other scenarios too, like a project changing direction making it less suitable for your use.
Companies need to be active participants in and supporters of the OSS communities for the software they rely on.
Look at the page for curl.se: Fastly is a sponsor. Look at sqlite's page: Bloomberg is a sponsor. Look at openssls page: nginx, Shiguredo, Activision, and Microsoft are sponsors.
Most of those companies sponsor many projects.
> I posit that there either is no moral duty as supposed, or its been fulfilled by the single sponsorship
Maybe but the issue in OP is that Apple has specific hardware requirements and various OSS developers are struggling to support M1 chips without paying out of pocket to support them. Whether it's moral or not, there is definitely a logistics issue between Apple and the very real and very important OSS communities.
Is it really too much to expect that the most senior engineers in a company could carve off a small portion of the marketing budget for "community good-will" and make small donations of platform credits (which cost close to zero) or cash?
If each of the $100B+ tech companies put just $10M into open source with a bit of care, I wager they would make the money back in developer job satisfaction and evangelism alone.
There are global enterprises which (internally) explicitly forbid any empolyees to contribute to open source code used to develop their product (tools, IDEs) and even used in their product (The Linux kernel for example).
That said, Apple providing Daniel / the Curl project with a machine should be a no-brainer.
The problem with that is where do you stop ?
How many OSS projects are there ? Where do you draw the line on who gets a free machine ? Then we get into the question of whether it's just the project maintainers or any contributor ? And if the contributors, then how many LoC do they need to contribute before they get a free machine ? If Bob just submits a PR to bump a version number, does he get a free machine ?
Giving a free machine to any Tom, Dick or Harry who has ever published a piece of OSS code is simply not a serious proposition.
Surely a reasonable minimum is the software that Apple are themselves shipping with macos
Exception policies for shenanigans.
It's not that hard to share a mac among a developer team, so one mac per project is a good start. There's natural downward pressure on the number of upstream projects used, so one for each of those is a good start. If you also make it request based rather than reaching out to send them, the capex won't be too bad; some upstreams won't be interested anyway.
Several possible criteria:
A) OSS which Apple uses internally - e.g. I'd be very surprised if there's no usage of curl out there.
(and / or)
B) Very popular projects.
(and / or)
C) Projects where Apple has a strategic interest in being able to run on M1. e.g. certain developer tooling.
You mean like the Linux Foundation? Or the Cloud Native Computing Foundation? Or the Rust Foundation? Or the Python Software Foundation? Or the Free Software Foundation? Or the Software Freedom Conservancy? Or the Mozilla Corporation?
Do they have an obligation to? Even a moral one? Honestly I don't think (I hope!) Apple actually uses curl in their products (probably a good thing). It's just provided as a convenience to developers who might be used to it.
I got curious about Apple's use of Curl and Googled it. One of the top hits was another blog post complaining about Apple not donating Macs:
https://daniel.haxx.se/blog/2021/09/25/curls-first-twenty-ye...
I think you could replace 99% of the uses of Curl (download one file via HTTPS) with like 100 lines of Python or Rust or Go. It's not critical infrastructure in the same way that OpenSSL or LLVM or WebKit are.
libcurl is for sure
You wouldn't say `ls` is critical infrastructure.
It's like... you might use your phone's timer app all the time. But if it suddenly started nagging you for donations all the time you aren't going to donate; you'd just switch to a different timer app. It isn't uniquely valuable.
That's very naive. It's just not something that hard to imagine to replicate. Why do you think a 26 year old project exists, if it's something that could be replaced by a 100 lines something? Is the author stupid, or does he just include a lot of cruft and boilerplate? Are Apple and all the other users wrong?
How old do you think `cat` is? Does that mean it would be hard to replace?
> Is the author stupid, or does he just include a lot of cruft and boilerplate?
I never suggested that. Curl had it's own HTTP implementation. A replacement would obviously not do that. Also curl supports a gazillion protocols that almost nobody uses.
https://endler.dev/2018/fastcat/
https://ariadne.space/2022/07/17/how-efficient-can-cat1-be/
I think this is one of these "not built here" discussions. It's very easy to underestimate the number of edge-cases to support especially in the case of network transfer. You start implementing your own, then you've got to handle non standard certificates, 300 redirects, temporary failures, backoff strategies, optimizing chunk size, handling proxy calls, etc. Using a standard library makes sense.
(And critical IMO means 'important in fact' not 'not replaceable in theory'.)
It doesn't make sense not to do it, if the corporation is a single rational hive mind. Of course, it isn't. A corporation is a collection of individuals, each operating with different goals and limited information.
What do you think it is used for?
Also smart phones, smart watches and TVs generally don't use libcurl for that. Android, iOS and WebOS have their own HTTP stacks. I don't know about cars or consoles.
Nearly every videogame for instance use libcurl as their HTTP stack. You know why? Because it's the best HTTP library for C/C++.
If you have a C/C++ application doing HTTP traffic it's likely using cURL. Or, I dunno, you could look at this page to find out: https://curl.se/docs/companies.html