Darwin-xnu: The Darwin Kernel | Hacker News Reader

Darwin-xnu: The Darwin Kernel

github.com

375 pointsadamnemecek83 comments

The big news here is the inclusion of arm and arm64 specific files, which used to be excluded in source drops previously.

So is it a totally new story that iOS kernel is open sourced?

It's a story that (a) the iOS bits are included and (b) it's posted to GitHub instead of dumped at opensource.apple.com. We'll see if Apple accepts pull requests.

I doubt they will. Actual commits aren't going to github, just version snapshots.

That's frustrating.

Github is nice, but there is (and should be) far more to the open source world than just Github.

I am happy they are making sources available. It would be wonderful if Apple started developing their kernel in public (I would like to read those code reviews). What tools they use should be up to the developers.

I think he refers to the fact that we don't get the real version history, with informative commits which may provide context for the code in question, but instead just get snapshots for particular versions.

I'm not saying it's useless, but it's certainly much less useful than a proper version-history, and very much a declaration that it is in no way intended to be a collaborative project.

But I guess that shouldn't surprise anyone either. It's Apple, after all.

Apple's made some pretty big genuine contributions to open-source: WebKit, LLVM[1]/clang, Swift, OpenCL.[2]

[1] Yes, technically "not Apple" in the sense that Node.js isn't Joyent.

[2] Admittedly partial credit at best since they've orphaned it in favor of their proprietary Metal API. I'd guess however that Metal will get open-sourced pretty soon.

webkit was based on khtml (LGPL) - not much of a choice for apple

They could've written a rendering engine from scratch, or (in 2001-2002) acquired Netscape or Opera for a song.

Did Apple have a lot of cash in 2001/2? IIRC that is before the iPod hype.

For a long time, they just did source-code dumps whenever they made a binary release (i.e., did what is required by the LGPL). They actively chose to move it to being run as an open-source project, with public history, public bug tracking, and public contributions.

No, because this thing, darwin-xnu AKA the OSX/iOS microkernel, was already opensourced through https://opensource.apple.com/ for both iOS and OSX (to be precise, the source tarballs can be found here: https://opensource.apple.com/tarballs/xnu/)

What is new however is that Apple is now seemingly actively using GitHub to publish their opensource contributions.

Apple using GitHub isn't new, they've been doing it for at least a couple of years. ResearchKit was their first GitHub repository I think.

Isn’t it Swift?

Minor correction: XNU is not a microkernel.

XNU is a hybrid kernel incorporating large elements of Mach (microkernel) as well as BSD (monolithic kernel).

The term "hybrid" is not a particularly useful concept, since the whole idea of a microkernel is to achieve stability and extensibility by placing only a minimal kernel in kernel space, and have everything else (drivers, file system, etc.) live in userland.

XNU is built on Mach (initially 2.x back when it was NeXTStep/OpenStep, later 3.0 for OS X), but everything runs in a single address space. Mach ports are retained throughout the OS as a communication mechanism, including userland, but all the core stuff is in the kernel, and Apple has transitioned many Mach port IPC responsibilites to more traditional syscalls to work around the fact that ports are comparatively slow. For example, all POSIX operations such as read and write are BSD syscalls that correspond to entrypoints in the kernel; they're not IPC messages.

That‘s true, but it‘s also important to note that most of the BSD subsystem is actually a shim over Mach. So once your BSD syscall has made it to the kernel, chances are you‘ll get a mach IPC call from the BSD shim.

The term "hybrid kernel" means it elegantly combines the disadvantage of a Mach microkernel (slow IPC based interfaces) with the disadvantage of a monolithic kernel (no fault tolerance due to single address space).

> No, because this thing, darwin-xnu AKA the OSX/iOS microkernel, was already opensourced through https://opensource.apple.com/ for both iOS and OSX

This is not true. iOS components were always stripped out until now.

This, while probably seemingly unimportant to a lot of people, is extremely great news for a project I'm a part of!

We're working on a full iPhone emulator based on QEMU, and currently, while we are able to boot into early userspace from an unmodified iOS image, the release of the iOS-specific code will be quite helpful in speeding up bug elimination and the development of the virtual graphics hardware that can get us past a basic graphical framebuffer.

wow, interesting! Where can I find your project?

Probably this:

https://github.com/joshumax/QEMU-s5l89xx-port

Or this:

https://github.com/nvsio/qemu-ios

Found sort of via this:

https://www.reddit.com/r/jailbreak/comments/6crw5t/question_...

I assume they won't mind my sharing. They posted it publicly with the same username and have public Git pages.

You've incredible search skills! Thank you very much =))

Actually, you saved me a post! Thanks!

Although quick note, a lot of work is being done in a private repo until it passes the "clean room" test, after which it will be pushed to the public repos. :)

There seems to be some confusion on whether the iOS kernel was or was not newly open sourced and whether it's the complete iOS kernel or not.

I don't see why there's confusion. AArch64 support in platform export (like hal.dll for XNU) seems to be stubbed out and unimplemented meaning that the ARM support is at least from an ancient iOS kernel.

https://github.com/apple/darwin-xnu/blob/master/pexpert/arm/...

License for those interested:

https://github.com/apple/darwin-xnu/blob/master/LICENSE

TL;DR - GPL-ish with a Facebook-style patent clause.

At the Tencent security conference in August, there was a demo of ARM iOS booting on virtualized iPhone6 hardware on x86 Macbook. Video clip was on Twitter but since removed.

Odd that there wouldn't be a public explanation for this. Poking through board_config I found support for the A10X (T8011), and for the three Apple Watch chips, but not for the new A11 SoC (T8015).

With a blog for Machine Learning where Apple shares some of its practices and now open-sourcing its iOS kernel for the first time, Apple is entering a new era of Post-Jobs where every company has started to believe the only way to attract quality resources is to contribute back to the society - Remember how this community scared FB to rewrite React License. It's a great way forward for FOSS and hopefully one that can set up the right way for Post-Cook Apple.

It's a start.

It would be nice to see more of iOS open sourced. Not everything, but enough to run the GUI and maybe Safari.

I think that could be a huge win for Apple. Lower end, open source iOS compatible phone could attack Android offering, particularly at the lower end. They'd gain control over the market (as it would be derived from their source, and compatible with their ecosystem) without having to do much work.

Honestly, if Apple don't do it, someone should come up with an open source iOS compatible offering, which developers could easily re-target their apps to.

VCs reading this, I'm offering 15% for 3MUSD for the above idea. ;)

> They'd gain control over the market

what you described is a loss of control. business is hard.

I think Apple (and everyone else) is doing pretty well with their closed source software.

Apple doesn't need others to make lower and cheaper devices. They're more than capable of doing it themselves, they just don't want to.

No, they don’t want to. But I think it would be advantageous to them to not have Android around.

Potentially they could convert many Android users to open source iOS. It wouldn’t come with the branding, I think it could be to their advantage.

It’s not likely to happen, but I think an open source iOS clone/fork would be nice and could also have potential benefits for Apple.

> No, they don’t want to. But I think it would be advantageous to them to not have Android around.

In 2016 Apple got 104% of the entire mobile industry's profit. Not revenue. Profit [1]

I hardly see how disappearance of Android would bring Apple any more advantages.

This year Samsung looks like they might beat Apple on profits ... due to large sales of components "to a major smartphone customer" [2]

[1] http://fortune.com/2016/11/04/apple-smartphone-profits/ [2] http://money.cnn.com/2017/07/27/technology/samsung-profit-ap...

Thanks, the 104% profit number doesn’t make sense to me and I couldn’t see it explained the the article.

How do you get >100% of the profits available in a given industry?

The rest of the industry operates at a loss.

On average.

Apparently this is such a bad idea, that it's possibly my most downvoted comment ever. :)

I should definitely try and get this funded. :)

I'm still somewhat curious why a iOS compatible open source clone might be a such a bad idea. Even if it was from a different vendor perhaps.

>Remember how this community scared FB to rewrite React License

HN helped, but I'd credit WordPress.

I understood "this community" as in "the FOSS community", not "HN community".

From the AI perspective the fact that Apple hasn't been publishing has really been a problem for them attracting talent.

I disagree. I think that Apple being as big and as popular as they are will continue to attract top talent for both the salary that they pay and the ability to work on projects that a big company like Apple affords. Open source need not apply here but it is still a good thing.

And what are those projects exactly? They haven't achieved anything technically groundbreaking or even game-impacting for developers for a while now. Swift maybe, but even that...

Come on. They produce multiple entire platforms including hardware (including CPUs and GPUs), OSes (kernel to APIs), apps, a browser, developer tools, services, etc. There are literally thousands of interesting things to work on at Apple.

It was not really my point though. It was originally addressing the fact that to attract top quality engineers with tech, the tech needs to be exciting. And Apple hasn't exactly shined in that regard in the past few years.

Now of course, they're a behemoth company and I'm sure they're tackling complex problems, but they're not rewriting entire operating systems or browsers, they're not strong in the Cloud, they don't have a massive distributed systems offering, they're not creating whole new languages, which I would think is what gets the star engineers going on the middle to long term. The reason for that is that their current pace is more about refining things and not messing with a formula that's been making billions year in and year out for the past, what, 10 years?

You do raise a great point about the hardware part of their business though, I would agree, it is something that sets them apart from companies like Google, Facebook, Amazon or Microsoft and that is a game-defining aspect as far as attracting talent goes, though in a specific niche.

Apple created Swift, a new language that is very good.

Apple is also doing a lot of cool stuff with hardware, AI, and machine learning, including hardware specifically designed for machine-learning products like Face ID.

In terms of distributed systems, the app store and iTunes media services are among the highest-traffic services on the internet.

All of this stuff involves a lot of behind-the-scenes work that is very interesting to engineers.

This is a mere source code dump that they slapped their Apple Public whatever license on (despite the explicit acknowledgement that vast parts of it are based on other, more freely licensed projects).

If anything it's a net negative.

The APSL is a perfectly fine license -- its terms are generally similar to the GPL, and it has been explicitly approved by both the FSF and OSI as a free software license.

To someone not familiar with Darwin internals, are drivers bundled or is it possible to get binary blobs? Does this mean we can build an open source stack that could run on iPhone or iWatch?

Without the ability to sign the kernel, even if you could build one you wouldn't be able to run it.

That's big, what's their purpose for open sourcing if not to allow developers to build on it?

It’s possible that this might help security researchers find/fix bugs.

It may help other developers understand how iOS works.

Overall, I think it’s unlikely to be hugely useful, in terms of actuallly using it for anything.

A significant fraction of iOS/macOS/tvOS is now relegated to kernel extensions and this trend has only been increasing -- HFS+ IIRC is now in a (closed source) kext and APFS has been from the start. So while it's admirable that Apple is more actively using Github, this is by no means demonstrative of Apple open sourcing their crown jewels. Running a custom kernel on anything but macOS would still be a herculean amount of work.

HFS is open-source, both the kext and utilities (fun to see that fsck is using code from the early 90s Disk First Aid). I built the kext for Sierra and it wasn’t too hard.

https://opensource.apple.com/source/hfs/hfs-407.1.3/

Oh nice! I usually read the XNU sources that someone had previously mirrored to Github and had missed this.

The new GitHub home aside, there's some cool stuff in XNU. The memory compressor, process hibernation, activity tracing, DTrace...worth poking through!

Looks like they are working on NVRAM support : https://github.com/apple/darwin-xnu/blob/master/iokit/Kernel...

All Macs have NVRAM for settings like default boot disk, volume and display brightness.

Dang, no APFS. Was hoping to gauge how portable it'd be to *BSD.

You have zfs.

It's still nice to have options.

I love ZFS and use it as much as possible. But we make heavy use of UFS at $work because sendfile and fail in place. I'm in the market for a flash filesystem for this use case. I doubt APFS has that all figured out, but it might be technically interesting enough to do so, and it might integrate move coherently with the pager than ZFS (the ARC makes this quite difficult but not impossible to do sendfile/mmap w/o extra copying)

Have you tried nandfs? I have not, so I don't know if it meets your needs.

Interesting side note, source contains 8701 goto statements -- I like it.

     $ find . -name "*.c" -exec grep goto {} \; -print | wc -l
     8701

Most of it to unify releasing resources on function returns to one place at the end of the function body. This is what makes RAII in C++ so important/convenient.

You are saying it like if 'goto' was a bad thing. This is pretty normal for any kernel/system projects.

And the same command on the Linux source returns 160,818.

It's interesting how Dijkstra's take on the use of goto has blinded a large number of developers on the practicalities of its use.

> This is what makes RAII in C++ so important/convenient.

This reminds me of the C++ developer who, when asked what his favorite snippet of C++ source code was, replied with:

dupe: https://news.ycombinator.com/item?id=15371102 and https://news.ycombinator.com/item?id=15371111

This means that we should have cheap non-apple phones capable of running iOS applications. Why would apple allow that?

Kernel != userspace

macOS kernel has been open source since OSX came out.

It does not mean that at all. This is not iOS itself being open-sourced, only the Darwin kernel is open-source, which is pretty much useless in isolation, certainly for trying to develop a cheap phone to run iOS apps.

Oic ... the framework/middleware is not. I took parallels to Android. Thanks for correcting me.

Nice, but I don't want open software as much as open hardware.

(For software, there are plenty of alternatives).

People are never happy.

It strikes me that the unhappy people are not always the same ones and don't always have the same reasons.

For example I imagine there are a few people long unhappy with Darwin's open/closed hybrid state that are perfectly happy with Linux or *BSD ... And a bunch of those won't necessarily increase their happiness with Apple even if they open up a bit more. So it's correct to say Apple can't or won't please that type.

What makes you say that?

Lack of good options to put a libre OS on phone hardware and build a libre phone.

Why can my top-of-the-line desktop be libre but must my phone be locked down?

Without hardware and userspace this is pretty useless to anyone but security researchers right? That is why he said that I would imagine.

Tabs instead of spaces. Tsk, tsk.

https://i.imgur.com/VE7kEeC.png

"debo" must have been an intern.

Reply on news.ycombinator.com