Apple releases open source modules of Big Sur | Hacker News Reader

Apple releases open source modules of Big Sur

opensource.apple.com

Apple’s Kernel Engineering Manager posted a walkthrough for building and running a kernel from these sources:

https://kernelshaman.blogspot.com/2021/01/building-xnu-for-m...

> NOTE: due to missing network (Skywalk) and power management (XCPM) functionality, your machine will be missing some features. For example, sleep/wake will not work.

That's a little sad, is this recent? I'm pretty sure that at least as recently as El Capitan, you didn't loose any obvious functionality by recompiling the kernel.

I'm glad they document this, but why is this on blogspot instead of a readme in the source or some other official apple.com page?

I guess they suffer from the same disease as the Android team that writes documentation on Medium, and before that G+, instead of Android official documentation.

This is the kind of stuff Wikis (and wiki-likes) were made for. It's so frustrating when trying to find some references and stumble across the answer on random blog posts.

They self document what they did at work in their own private time. Companies engender this behaviour by calling them "heroes" or "MVP". Particular product/technology gets exposure, free (as in beer) advertising, and similarly as free digestible documentation (project walkthrough) from your staff.

If you forced employee to publish on a wiki, it would feel far too much like work. Having it on a privately owned public forum gives a sense of personal ownership and development.

It also gives them creative control.

It's no fun if you have to go through a multi step technical, managerial, docs team and legal review process where PM makes you include/exclude what they want to turn it into a marketing doc, your manager shuffles your priorities, the docs team makes you dumb it down, blow up the word count and remove external links and convert to .docx that will only be available to registered enterprise customers, and then finally legal tells you you can't write any of this anyway.

All of these concerns are orthogonal to whether or not these employees are expected, tacitly or otherwise, to do this necessary work on their own time. There's no reason Apple and Google, of all the companies in the world, can't set aside time for their employees to write documentation, and there's no reason they can't organize their bureaucracy in such a way as to not inhibit this creativity.

Probably because that document is not endorsed by Apple Corporation.

In other words, corporate bureaucracy getting in the way. Not surprising when it's one of the most secretive companies too.

This is actually a more interesting release. Though it would be more interesting if it were for the M1 kernel.

As per the article:

> It is not currently possible to build open source XNU for Apple Silicon Macs.

If anyone had a doubt about the closed nature of Apple ARM processors, this should dispel it.

Well, it didn't dispel it for me. Especially the "currently" part. What now?

The question is, how much of this can you actually build?

I had a hell of a time a few months ago trying to compile the Security framework from OS X 10.9.5 (I wanted to update the cipher suites!). It needs to link to a bunch of internal, closed Apple code.

I did eventually get something to compile, so there's that. But I had to pull in a bunch of reverse-engineered headers from the Darling project and similar efforts, and whatever it was that xCode finally spit out didn't seem to actually run.

It has been a while since I rebuilt the XNU kernel, a few years perhaps, but it is possible, and it works.

These instructions are a lot better than the ones that I had to follow a few years back - I might try it, just need to find a guinea pig :-)

XNU itself is one of the ones that can definitely be built outside of Apple! Lots of people have made custom kernels over the years.

The only one that I have built (for Linux on arm64) is mDNSresponder. It is a required component for CarPlay. I don't know if other Apple open source code runs on Linux (for real, not for tinkering).

CUPS[0] (print server) is a big one... Made by Apple and pretty much the standard for Linux<>Printer interaction...

[0]https://www.cups.org/

There is a chess engine in there :)

https://opensource.apple.com/source/Chess/Chess-408/

Containing this wonderful comment:

"Paradoxically enough, moving as quickly as possible is not necessarily desirable. Users tend to get frustrated once they realize how little time their Mac really spends to crush them at low levels. In the interest of promoting harmonious Human - Machine relations, we enforce minimum response times."

https://opensource.apple.com/source/Chess/Chess-408/Sources/...

I didn't realize that this would end up the single most cited comment (measured on a pretty small scale) of my career when I wrote it.

But frankly, I wouldn't trade it for "You are not expected to understand this".

What was that on?

https://community.cadence.com/cadence_blogs_8/b/breakfast-by...

It's in the Research UNIX scheduler code,

https://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/sys/ken...

I ran across an extreme example of this recently trying to track down an old acquaintance: the website mylife dot com. There was a street address of the person but if I wanted their email address it was like 10 minutes of progress bars before finally being asked for money. I didn't wait and didn't pay, but was mesmerized and intrigued by how much UI/UX investment had been made and how profitable the endeavor must be to merit such efforts. Also made me very sad knowing how many people must have been scammed out of money. Or maybe the paid reports serve some valuable purpose. I think the former.

Additional fun examples (artificial projections of expected behaviour, for the purpose of human familiarity or trust):

- The "Checking for Audit Risk" progress bar in TurboTax/etc software, which is shown while comparing about a dozen floating point values, perhaps after a division step. This takes imperceptable time, but the appearance of "doing work" projects value to humans.

- Comfort Noise (very low volume pseudo white noise) played into a VOIP phone to replicate the ambient inductive noise in analog lines. This assures humans that their handset is not broken.

- Engine Noise generated by some (nearly silent) electric vehicles. This has a huge safety component, so it's not frivolous at all!

All of those "lookup X person" sites do the same thing. You may assume that you've sunk so much time waiting for them to scan all their data, surely it's worth a few dollars to see the results.

Those sites can be worthwhile and there's nothing necessarily scammy about the UI/UX IMHO.

The scammy part is where they push you into a nearly impossible to cancel subscription with promotional pricing as a way of "saving" vs an expensive one-time payment for one report.

The loading bar UI/UX correctly gives the impression that the report will be thorough. In my experience, they are quite good. So that's just fine.

> The loading bar UI/UX correctly gives the impression that the report will be thorough. In my experience, they are quite good. So that's just fine.

The loading bar is lying, plain and simple. "Giving the impression" of something that very much isn't true is just a polite way of saying "screwing with your user's perception of reality". Developers often complain that users cannot at all understand computers - these kinds of practices are part of the reason that is the case.

It's not that far conceptually from the tech support scammers that show you errors in your system's Event Log to "give the impression" that you're insecure, and then offer to sell you bogus security products at inflated markup. Same kind of ethics underneath.

I don't think it's how smart the computer is that annoys people. It's that people want a moment to see and think about the board after their move, before the opponent's move.

This is basically Skeuomorphism, right?

Not really — Skeuomorphism refers to using existing (usually "obsolete") user interfaces as an analogy to help users understand a new UI. Artificially increasing computation time is about making users think that the computer is doing a lot of work when it's actually not.

The goal of the former (skeuomorphism) is to help users understand; the goal of the latter is to misdirect users, either 1) so they have a better experience (like Apple here) or 2) so they are willing to fork out more money. Users are willing to pay more when they think more work is being done (c.f. TurboTax "finding" the most deductions possible, or a public record website "searching" through "all the databases" for a person).

This texture had me smirk. :)

https://opensource.apple.com/source/Chess/Chess-408/Styles/G...

Wonder what their workplace policy is. It’s pretty much “don’t ask, don’t tell” around here.

That's pretty much what it was like when I was with their AppleCare group. During the hiring process, I was fully expecting a drug test (that never came).

Yes, at the time I was hired at Apple, my Intel colleagues were drug tested at hiring. I was not.

But I never saw any on-premises use of illegal drugs, and there were policies around the serving of alcohol.

Lol "grass". They've applied some heavy transformations to it in-game, so it's not at all obvious what it is. For the sake of whoever did this, I hope Apple has a sense of humour lol

I think they might be okay with it.

https://youtu.be/sDZYTLf8L4c

Some people more than told, offered and used drugs openly or the end at the day while I worked there. Nobody cared.

In my brief stint at Apple, only two things seemed to matter; what you could deliver to your org, and whether your manager liked you (highly correlated with the first).

Edit: I guess there was a third thing, don’t do anything that could screw up the brand at all, but that “mattered” only in that it was table stakes.

Considering it's well known Jobs would ask in interviews how many times a candidate had done LSD, I doubt they're particularly worried about recreational drug use.

The book "The Cult of Mac" has a chapter around it (not necessarily employer oriented)

Can someone explain what this means? Does it benefit those working on Linux support for M1?

It doesn't have much significance. Apple does this for all (major) macOS releases. macOS and iOS are built off of open source components, for which Apple releases the source code now and then. It's a courtesy for the most part - most of these are BSD licensed and therefore don't strictly require source code to be released.

These notably don't include much of mac/iOS proprietary user interface components, drivers, or system frameworks - basically everything that makes macOS/iOS remotely interesting. The source code drops are helpful for the creation of other Unix-like operating systems based on the open source macOS kernel (see also the defunct Open/PureDarwin projects), and the released apps are sometimes ported (e.g. the GnuStep project has some older version of Chess.app and others ported over). There's some off and on efforts in the FreeBSD community to port Apple's open source launchd init system [1]. All in all though, there's really not all that much interest in reusing the code from these code drops by the open source community and any efforts seem to lack staying power.

The released source most practical value is most likely derived by researchers investigating macOS/iOS internals and being able to inspect the source code for some of it. As a curiosity it looks like you're able to build and run your own XNU kernel on macOS [2] (what is this, Linux?), but I can't personally attest to the process.

[1]: e.g. https://github.com/freebsd/openlaunchd

[2]: https://kernelshaman.blogspot.com/2018/12/building-xnu-for-m...

"BSD licensed and therefore don't strictly require source code to be released"

Broken promises of liberal freedom.

What? The BSD license is 30 years old, it has never required the release of source code of derived works.

What kind of promise do you feel you have been given?

Not really, this is done for every OS release. It's a completely routine nothingburger with an iffy HN title.

Yeah, the real title should be “Apple abides to the letter of opensource licenses, like every year, while largely continuing to ignore the spirit of them”.

Apple isn't actually obligated to release any of this code. It's licensed under BSD-like terms.

Big Sur came out several months ago, so if this release contained GPL code Apple would potentially be in trouble. (But it doesn't, so they're not.)

It did come with bash, which is GPL. (sure, zsh is now the default, but bash is there too)

macOS actually comes with Bash 3.2, which is from 2007. it is the last version released before they switched to GPL. this is actually why they switched to zsh.

Well, the fact is, the version available on opensource.apple.com in macOS 11.0.1 is 123.40.1 and the version in 10.15.6 is 118.40.2... and they slightly differ.

Interestingly, the new version removed the dependency on the unavailable rootless.h header, making it (probably) buildable out of the box, now.

I would assume the version difference is from Apple updating it internally. They don't run into GPL issues unless they pull from newer upstream Bash code.

^ We all assume that's why, it should perhaps be said. Although I think the circumstantial evidence is pretty strong. :)

Not really - they are obligated to provide it on request, and allowed to take a reasonable time to do so. If someone requested it, and they later uploaded it here, they'd be fine.

In what way does the BSD license obligate that?

I read that comment as saying that GPL wouldn't require simultaneous release, but would simply obligate them to release on request.

But I may also be misreading it.

Yes, I was referring to the GPL, not BSD licensed software.

Not really. They're not required to share their source or alterations to BSD and MIT works, which most of this is.

There is also some original work from Apple.

Please explain how they are ignoring the spirit of these opensource licenses in respect of this code.

You’re confusing Open Source and copyleft, I think.

Apple had no problem including "copyleft" code in their OS for the longest time... but GPLv3 is a no-go due to the closed nature of their other platforms (iOS). GPLv3 adds the requirement that code for derived works not only be released, but that someone should be able to build their own modifications on that platform for free, won't work with the way the the iOS developer agreement works ($99 a year.)

And it wasn't worth the complexity of having different software for each of their OS's, so they just banned all GPLv3 software from the OS.

Apple's use of open source could have totally worked with "copyleft", but only the GPLv2 usage of the word. GPLv3, not so much.

> GPLv3 adds the requirement that code for derived works not only be released, but that someone should be able to build their own modifications on that platform for free, won't work with the way the the iOS developer agreement works ($99 a year.)

I don't see anything in GPLv3 about people having to be able to build derived works for no cost.

What GPLv3 adds is a requirement that your provide "Installation Information" to allow you to install and run the code once you build it. Installation information is things like keys to sign the code if the platform it is for only runs signed code.

The installation information requirement only applies when the object code was conveyed "as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized)". A "User Product" is "either (1) a “consumer product”, which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling".

GPLv3 object code that is not conveyed as part of a change of possession of a User Product is not subject to the installation information requirement. This requirement was added to GPLv3 specifically to stop what TiVo was doing--shipping GPL code on their boxes that could not be replaced by the user--and it was narrowly drafted to only cover that situation.

Hmm, I guess there's two arguments happening here...

- Apple can't include GPLv3 software in iOS because it only runs signed code, and the keys needed to sign system components are kept secret by Apple. From this perspective, the developer agreement is a red herring.

- App Store developers can't include GPLv3 code in their apps, because then Apple would be breaking the terms of GPLv3 by not providing the "Installation Information" as you put it, which would be required for users to freely modify the app. (This "Installation information" being the iOS SDK and a Developer signing key needed to run your own code on your device.) Keep in mind that when Apple distributes your app via the app store, they take on the responsibilities to adhere to the terms of its license as they pertain to "redistribution", since that's exactly what Apple is doing.

> GPLv3 object code that is not conveyed as part of a change of possession of a User Product is not subject to the installation information requirement. This requirement was added to GPLv3 specifically to stop what TiVo was doing--shipping GPL code on their boxes that could not be replaced by the user--and it was narrowly drafted to only cover that situation.

If Apple were to include GPLv3 code directly as part of iOS, they absolutely would be "shipping GPL code on their boxes that could not be replaced by the user"... something that is OK in GPLv2 (so long as you provide the source like they're doing in TFA), but not OK in GPLv3 (because of the tivoization clauses.)

To tie all this to my original point... macOS ends up having to kill all its uses of GPLv3 (even though you can replace it on device and build your own, on macs), because of the shared architecture and engineering with iOS. It's easier to just ban GPLv3 from the entire company really.

Some of this is Apple Public Source Licensed I think which is at (partial) copyleft - at least accoding to the FSF.

You post this every time this discussion comes up, even though it’s patently false and known to be false and you’ve been told it’s false multiple times here. Amazing.

> PowerManagement-1132.50.3

At least the name sounds promising.

None of this is licensed such that they can use it directly, but having the source available can speed up the reverse engineering process such that they can rewrite their own version of the code that interacts with the hardware.

Note that this requires a clean-room process - reading the original source code as inspiration usually taints any work derived from the observations.

This is why open source developers actively avoid reading leaked sources from proprietary projects for example. "open source" code with too restrictive licenses is no different from closed source code.

The relevant developers are of the opinion (one that I agree with) that it does not require a clean room process. A clean room process is a gold standard for preventing accidental copyright infringment, but is not necessary and is enough work they don't consider it worthwhile. Reading (and even to the extent that it's compatible with the license modifying) the original source code to learn how the underlying hardware works is not copyright infringment and does not make future works you produce derivative works of the original.

Source/explanation in their words instead of mine: https://asahilinux.org/copyright/ (See: "Reverse engineering policy")

The policy of those developers does not mean much, and while the policy of upstream kernel developers that need to accept the code can be interesting, but ultimately only a lawyer's opinion have any meaning here.

They might coincidentally be right of course, but their advice, policy and opinion in this legal matters mean as little as yours and mine.

> None of this is licensed such that they can use it directly

Not sure what you're referring to here?

All the interesting bits are Apple License [1], which isn't GPL compatible [2].

[1] https://opensource.apple.com/source/IOGraphics/IOGraphics-58...

[2] https://asahilinux.org/copyright/ - See "Referencing Other Open Source Code"

Interesting. I'm now doing some more Googling, and I can't seem to find a place that explains why the APSL isn't GPL compatible. (But it's almost certainly not, given that lots of authoritative sources including GNU[1] say as much. They just don't say why.)

1: https://www.gnu.org/philosophy/apsl.en.html

Probably (I haven't studied it) because of what the FSF says here "It is not a true copyleft, because it allows linking with other files which may be entirely proprietary." So it's apparently file-based copyleft, which is in the same vein as CDDL. (Which is also generally considered GPL-incompatible for that reason.)

It means the people who were breathlessly jumping the gun a few weeks back, saying Apple would never release open source Big Sur stuff may have been a bit over-excited.

Just a release of their use of GPL code which the would be required to provide if requested anyways. It does not have any impact on Linux support.

It's interesting that because I said GPL instead of open source, my bad, I'm downvoted to -4. Shit.

Quite a bit of this is sourced from within Apple under Apple's license.

There might be some useful things in there. If they released the code/ patches in an easier to review way. As it is, it's fairly difficult to see if anything

Apple in general doesn’t use gpl code. The vast majority of their work is BSD or BSD-like license.

I like looking through lists like this and discovering new OSS projects.

This one was most interesting as a non-C or OS developer:

http://www.nongnu.org/libunwind/

> The API additionally provides the means to manipulate the preserved (callee-saved) state of each call-frame and to resume execution at any point in the call-chain (non-local goto).

Sounds like fun...

libunwind is magic that lets you implement stack traces.

NTFS, HFS, NFS, MSDOSFS, ... So many sources of file system drivers were release with the exception of APFS.

I mean, I don't think there's anything particularly sinister here?

The released the code they wrote for compatibility with other people's file systems, and a filesystem Apple developed in the 80s. They didn't release the code for their own, proprietary (and non-ancient) filesystems. It would be nice if they did, but you can see why they didn't.

"relases"

Why add the scare quotes? I don't know what you're implying.

The word is spelled incorrectly.

Ah, seems it would have been more helpful if they explicitly said that.

“Open source” doesn’t mean much if it doesn’t come with a usable license.

> Modified Code. You may modify Covered Code and use, reproduce, display, perform, internally distribute within Your organization, and Externally Deploy Your Modifications and Covered Code, for commercial or non-commercial purposes, provided that in each instance You also meet all of these conditions:

https://opensource.apple.com/apsl/

Open source website design looks like it hasn't been given any love since 10-15 years. Apple you can do better?

Good a nice clean easy to use website. I wish more were like this

Yum, this is interesting. One can easily modify the URL to fit older OS releases, and get the analogous set of modules for those (like “…-1014.html” for macOS-10.14).

Or just click a few links and end up here:

https://opensource.apple.com

This isn't a new page.

You can also browse the different versions via links from https://opensource.apple.com/

I guess apple using git is too much to ask..

Apple processes the source code before releasing it as open source, like excluding some source files and even stripping specific #ifdef sections. This can be implied from shell scripts that are part of the tarballs.

Such an approach would be way more complicated if the open source release included version control history. These code drops already always come months after the macOS releases.

https://github.com/apple

There is hardly any overlap between their public repos and these modules.

they do all the time for other open source projects, I assume this is just legacy that plugs in with their in-house system.

They could very well be using Git internally. But they don't want to release the full history of their development process, complete with authors, commit messages, and intermediate commits -- that's likely to include or imply a lot of information that isn't appropriate to release.

Indeed, and this is so much an industry standard practice that it was particularly significant when Apple open sourced Swift in Dec 2015, they didn't just do a tarball dump or promise that development would be in the open going forward, they actually opened up the whole development history right back to the start of the project in 2010: https://github.com/apple/swift/commit/afc81c1855bf711315b8e5...

Not really unprecedented though. I'm pretty sure the whole SVN repositories were published for clang, libc++, etc. when they were open sourced.

See also: Google and the non-kernel part of AOSP (it is just a snapshot of the current source at the time, not the in-between releases).

Most internal commit history is intact in AOSP, including the changes between releases. They just have a complex merging setup that deviates slightly from internal branches.

They do, Apple has an internal gitlab instance which a bunch of teams use.

I don't know why you're getting downvoted.

There's a serious argument that a source dump without commits is not the "preferred form of the work for making modifications".

Well, they could have not released anything instead.

Is not a single bit GPL? It's still worth comparing it to different standards of quality.

Beggars can't be choosers but I'm not begging, I'm evaluating them as a corporate citizen.

Then evaluate them how the people get treated on their factories, not whatever they do with a pity software license.

Reply on news.ycombinator.com