A Look at PureDarwin – An OS Based on the Open Source Core of macOS | Hacker News Reader

A Look at PureDarwin – An OS Based on the Open Source Core of macOS

jamieweb.net

244 pointsjamieweb104 comments

This was one of my hobbies at University (learning how it works, not contributing to it, to be clear). This SuperUser question was basically the culmination of my effort before I ran out of spare time, found a job, etc.

https://superuser.com/questions/546151/is-it-possible-to-com...

I'm super excited that people are still working on this! I miss having the time to go down rabbit holes, but for those that are there's are people out there watching!

A huge kudos to Jonathan Levin too! It looks like the full set of new *OS Internals books are now available (last I checked only two volumes were available) -

http://newosxbook.com

They are the spiritual successors to Amit Singh's Mac OS X Internals: A Systems Approach (http://osxbook.com).

While we are on the subject of macOS internals.

Do you know where I can read about what it would take to replace WindowManager and the rest of the desktop environment on macOS?

I would like to make my own implementation of the desktop environment, including replacing the top bar with the menus and stuff, and replacing the dock with a dock of my own, as well as replace spotlight search and Finder. Furthermore to also replace the open file dialog that is called to from programs when they open files.

Darwin is just the os kernel not the GUI stack on top. You can, however, take a look into KDE’s codebase to see how they manage panels (docks) and using bonobo for search and dolphin as a finder replacement. Menu has a special Application mode which moves the app menu to a panel (that Is docked to the top of the screen for instance)

> Darwin is just the os kernel not the GUI stack on top.

I know. I was speaking about the broader part of macOS, and of replacing the desktop environment.

Specifically I need to know how to prevent the default desktop environment with all the parts I mentioned from launching on log in.

WindowManager on macOS is started by launchd at least. Don’t know how intertwined the desktop environment as a whole is.

But yeah, how to prevent the stuff I mentioned from running, and then what I need to implement in my own DE so that I can have my own working DE on my MacBook Air where I can still run all of the applications that I have installed such as CLion, Photoshop, etc.

What KDE does is not of interest to me in this context. That being said I run KDE neon on my desktop so it’s not like I have anything against it. In fact I like it a lot. But it’s not relevant to what I am talking about here.

You are asking how to create your own DE for macOS, this is not possible or it would have been done already.

People keep saying that but why wouldn’t it be? It’s not magic, it’s just software.

— some lazy cynic

What a silly attitude.

It is almost certainly /possible/, but most people in a position to do so haven't thought it was worth the amount of effort involved.

> I would like to make my own implementation of the desktop environment, including replacing the top bar with the menus and stuff, and replacing the dock with a dock of my own, as well as replace spotlight search and Finder. Furthermore to also replace the open file dialog that is called to from programs when they open files.

While they may seem like a unified application, these are all separate components. Dock, Spotlight, and Finder are apps, and the open dialog is an XPC service.

How isolated are they from one another? For example, if I manage to replace the Dock first, does the Dock need to provide a lot of interfaces for the other parts of the DE to call into?

For example, when an application is started from the spotlight search, its icon pops up in the dock (if not already there) and becomes active, and when applications exit they become inactive in the dock and optionally disappear.

>also replace the open file dialog that is called to from programs when they open files.

Maybe take a look at Default Folder X or contact the dev to see how they go about this. I think they just inject into the existing implementation, but it might provide a starting point.

Injecting into the existing implementation is one way of achieving some of the stuff I want to for sure but it feels like that would be more fragile (though less work) and also it would limit what I would be able to do quite a bit.

That being said, DFX does look interesting and it looks like it is doing some of the kinds of things that I had in mind for my own imagined DE so I will have a close look at it for sure.

I wish the PureDarwin folks would prioritize clear build instructions. It'd be really great to see something like a "Linux From Scratch" or process to build a bootable disk image using only what's in their git repos. (Maybe it exists but if so it doesn't show up/is linked to from any of the obvious pages of their wiki)

That seems like a necessary first step towards eventually getting a bootable ISO that can install a PureDarwin system on a blank machine, physical or virtual.

Apple's Open Source repository has the current version of Catalina's core. https://opensource.apple.com/release/macos-10145.html

Why is PureDarwin on an older version? Lack of volunteers? Too many hacks required?

10.14.5 is Mojave, and we're still waiting for the latest 10.14.6 Mojave source. Their open source repo seems to be about 6 months behind at any time.

Isn't that a license violation?

It’s a BSD license, so they don’t need to release the source.

Ah, okay, that makes sense. Should have looked up the license before asking. I'll upvote your clarification.

Why would I want to use Darwin?

macOS has some UI advantages, its the largest reason I use it - but without cocoa, and all of the core foundations that make OSX, OSX, I just dont see the point.

Yep might as well just use one of the BSD linuxii

For me: you don't want to. The underlying Darwin/XNU layer is probably the most quirky (or depending on your PoV outright broken) unix-ish system you will see in general use today.

that was the subtext I didn't, I know XNU is not super performant.

z/OS is a much weirder Unix than macOS, although it depends on how you define "general use". But, there are a lot of enterprises out there which still run IBM mainframes, even if they often aren't keen on advertising that fact, and a huge lot of stuff in z/OS now depends on the Unix subsystem, in particular anything written in Java (which IBM has been heavily encouraging as a development language on mainframes, resulting in a lot of hybrid apps which are a mix of newish Java code and ancient COBOL/assembler/etc working together.)

(Some people will question whether z/OS really counts as a "Unix", but like macOS, it is actually a certified Unix, and as such has greater official claim to be Unix than FreeBSD or most Linux does.)

(Addendum: I say "most Linux", because while the vast majority of Linux distributions are not certified Unix, some have been. Right now, Huawei EulerOS is a certified Unix based on a Linux distribution. Inspur K-UX is another Linux distribution which used to have Unix certification, although the certification has expired–I guess they failed to pay the renewal fees.)

I don’t know about the current version, but macOS has been certified as UNIX in the past.

Catalina is a certified UNIX 2003 – https://www.opengroup.org/openbrand/register/brand3653.htm

UNIX 2003 is not the latest version of the certification, the latest UNIX certification is (somewhat confusingly) named UNIX V7 (different thing from the classic Bell Labs V7 UNIX), but thus far only Oracle Solaris is certified to that.

z/OS is still only UNIX 95 – https://www.opengroup.org/openbrand/register/brand3601.htm – maybe one day IBM will upgrade that to the newer UNIX 98, UNIX 03, or UNIX V7 certifications, although I suspect not. IBM has decided to focus on running Linux Docker containers under z/OS instead (which it calls zCX).

Unix is a license and linux is not. In fact the lawsuit Ibm facing is precisely that.

But we usually not used this but use posix. In fact i think that is what linux originally built against.

For others, z/OS has personalities, just like other mainframes and Windows.

What is certified as UNIX is the POSIX personality.

I've never heard the term "personality" used with respect to z/OS before, it certainly isn't standard terminology. (It is standard terminology with respect to the original design of Windows NT, and IBM's abortive Workplace OS, but z/OS has little in common with either.)

And strictly speaking, z/OS as a whole is a certified UNIX. Look at the registration certificate [1], it never mentions UNIX System Services. Obviously without that component z/OS would fail to pass the certification tests, but the certification is for the operating system as a whole not just that component of it.

[1] https://www.opengroup.org/openbrand/certificates/1199p.pdf

Maybe I misunderstood its architecture, but I thought that with the language services and the different kinds of workloads, it also had such clean separation.

Not really. A z/OS program is allowed to call a pretty arbitrary mix of classical MVS APIs and UNIX system calls. There is no strict separation as such between "classic MVS programs" and "UNIX programs". An MVS task isn't classified as a UNIX process until it makes its first UNIX system call (exception: programs launched via UNIX system calls get classified as UNIX processes even before they start executing) – some programs start issuing UNIX system calls as soon as they start, but a program can run for hours or days as an MVS task, and then suddenly out of the blue start making UNIX system calls, and then at that point the MVS task will be "dubbed" as a UNIX process. (To be a UNIX process, the MVS task needs to be assigned a UNIX security identity – UID and GID – which is the job of the security subsystem – RACF, TopSecret or ACF2 – if the security subsystem isn't configured to assign a UID/GID to that MVS task, all UNIX system calls will fail.)

Compare this to Windows NT, where each executable was marked as belonging to a primary subsystem (Win32, POSIX, OS/2, etc) which is fixed from the time of process startup. In contrast, z/OS offers the flexibility that any MVS task can potentially be "dubbed" as a UNIX process at any time. Or similarly, look at WSL1 in Windows 10 – a process is either a Win32 process which is allowed to call the Win32 API, or a picoprocess which can call Linux system calls, you can't mix Win32 calls and Linux calls in the same process. (You can do it via two processes, one Win32 and one WSL, communicating over IPC; by contrast, z/OS allows mixing UNIX and classic MVS APIs in the same process/task running in a single address space). So the separation between "UNIX" and "classic MVS" in z/OS is actually a lot weaker than in Windows NT, which is why the "personality" term which was used in the later isn't really applicable to z/OS.

Thanks for the overview.

Care to elaborate? Which aspects of XNU are you referring to?

Maybe not you personally, but one of the long-term informal goals of PureDarwin is to have an open-source environment you could run XCode CI builds on.

But most Mac apps will link against Cocoa…

You don't need to ship a shared library in order to compile an executable that dynamically links to said shared library. You just need the header files listing the shared library's exported symbols. You can treat Cocoa like any other closed-source blob of a vendor library.

I the context of CI, what’s the point if you can’t run tests?

If the application is well-structured, you can run many types of tests without ever touching the UI. You can also run things like linters and static analysis tools (although those are usually platform-agnostic).

Seems like if linked libraries weren’t present, GUI or no, it would be problematic. Im a hardware guy though so not much experience with that.

This isn't my exact area either, so my terminology is probably off, but one way to do this would be to compile the "backend" code separately and run unit tests on that. Treat it as a library in your application, if you will.

Another approach I've heard used was just faking the library from the headers - as long nothing from it is called in the course of your testing, your program wouldn't notice.

I think the Google vs. Oracle lawsuit might put a wrench into this logic.

If the outcome of that lawsuit goes that way, basically all OSS is gonna become illegal, so I think this might be the least of our worries... :(

> Oracle’s lawyers wrote. "The next Oracle will think twice about investing as heavily in a venture like Java if it knows that any competitor could freely copy its work to compete directly against it."

It would make me terribly happy if Oracle would refrain from doing so. I think this case is ridiculous. The same developer implemented a range check in the same way he did 10 years ago. Yes, copyright for software is broken...

And the lines in question were even removed, which is a tragedy itself, but I cannot see Oracle as anything but hugely detrimental to software. Developers should ignore anything they touch.

If they do indeed rule in favor of Oracle, it wouldn't be undermining OSS, it would undermine the whole judiciary system. It would be scientifically proven bullshit.

  private static void rangeCheck(int arrayLen, int fromIndex, int toIndex {
     if (fromIndex > toIndex)
          throw new IllegalArgumentException("fromIndex(" + fromIndex +
               ") > toIndex(" + toIndex+")");
     if (fromIndex < 0)
          throw new ArrayIndexOutOfBoundsException(fromIndex);
     if (toIndex > arrayLen) 
          throw new ArrayIndexOutOfBoundsException(toIndex);
  }

Suppose it is a programming language runtime, a database, an app server? Many of those get ported to Mac, but very little (or even none) of their code needs to talk to Cocoa, since they are mostly headless server components (or have CLI interfaces only).

Why port that kind of stuff to macOS? Obviously it isn't going to be the deployment platform, but developers will want to run that stuff locally during development. (Less important than it used to be with the popularity of Docker on Mac, although still it can be a bit more developer-friendly to run things directly on macOS instead of under Docker.)

Why is a booting XNU helpful for me to build macOS or iOS applications on?

For the same reason that ReactOS is helpful for you to build Windows applications: you can install the vendor's actual SDK on it (that'd be XCode here), without needing a license for the OS (which, in macOS's case, translates to "without needing to use Apple hardware.")

You know that there's entire data centers just running Mac Minis as VM hosts, in order to comply with Apple's OS licensing requirements? PureDarwin can just run on AWS.

Sure, running the XCode GUI needs Cocoa. But running xcodebuild doesn't, IIRC. It just needs some other, lower-level libs, ones that (reimplementations of) are in scope for PureDarwin.

Exactly. I don't think even Apple realizes what a burden it is that macOS isn't ever licensed to run on non-Apple hardware.

Self-service test VM provisioning, CI build/test environments, QA machine labs -- all of these would benefit immensely from not having to run on a farm of Mac Minis in order to stay compliant, while their Windows and Linux counterparts are able to run on large on-premise racked server environments with high-performance, high-availability storage and fast networking.

They realise that alright, that is why they offer cloud services to do exactly that.

They do? What services do they provide in that regard?

Via https://www.buddybuild.com/ acquisition.

https://docs.buddybuild.com/builds/xcode_versions.html

Buddybuild does not accept new customers.

But then if Apple gets pissed off at that, they just change the XCode EULA to say "no non-Apple hardware, please". Actually, I would not at all be shocked if the XCode EULA already says that.

No idea what Apple is going to get upset at...

But, surely it is in their business interest to make life easy for developers, especially if doing so isn't undercutting their end-user-focused hardware business? A cloud server running PureDarwin+Xcode is not undercutting their end-user-focused hardware business, and while it might mean they sell slightly less hardware units for that cloud use case (not a core market for them), surely the added ease for their developer community (who develop apps which make those end-user-focused hardware units so attractive) more than makes up for that?

Yeah I'm on your side. But Apple is Apple..

They've pretty well demonstrated through questionable deprecations and hoops they make people go through with each release that ease for the developer community is not what they optimize for, but I agree it would lead to better end user experience.

it's possible to build enough of their tooling from open source parts to build Mac and iOS apps on linux, but that's rather on the extreme side.

And yes, that includes notarization support.

Apple could easily require xcodebuild to run on MacOS in xcodebuild's licensing terms.

Darling[1] provides an opensource userspace runtime, which could be combined with this.

It's similar to reactos sharing userspace code with wine.

1: https://darlinghq.org

Darling can actually run parts of the Xcode toolchain!

If you don't feel an urge to try it, then it's not for you.

GNUstep does reimplement alot of Cocoa stuff

Why would I want to eat vanilla ice cream?

chocolate ice cream has some taste advantages, it's the largest reason I eat it - but without cocoa, and all of the core flavor that makes chocolate, chocolate, I just don't see the point. (like your post)

Are there are Darwin hackers lurking on HN? I would pay good money to get a brief summary of the kernel for kernel hackers such as myself who only know how things work in Linux. What are the big architectural differences or cool features not in Linux? Also, how hard is it to upstream changes/get drivers in mainline?

Amit Singh’s excellent book MacOS Internals: A Systems Approach [1] is a bit outdated now but still recommended if you’re interested in the workings of a non-Linux commercial OS.

[1] https://www.amazon.com/Mac-OS-Internals-Approach-paperback/d...

Jonathan Levin's *OS Internals volumes are an updated version of the same info, and wonderful.

Looks interesting and would love to read/review it. But at $88 a piece for 3 volumes ... hmm!

Volume 1 and 3 are easily worth their money, and then some. (Still waiting for my order of Vol. 2 even though it should have been released a few weeks ago.)

Here's a link to the first volume, made available on the book's website:

http://newosxbook.com/MOXiI.pdf

The Design and Implementation of the FreeBSD Operating System, 2nd ed., in the same category.

https://www.amazon.com/dp/0321968972

Not a Darwin hacker, but:

> What are the big architectural differences or cool features not in Linux?

Darwin is based on XNU from NeXT, which itself is comprised of OSFMK Mach microkernel but with many of the things pulled into the kernel itself for performance. A lot of the stuff with Mach tasks/ports/messages is still there.

> Also, how hard is it to upstream changes/get drivers in mainline?

You can write your own using IOKit but they're not going to be in the mainline tree, because Apple.

> Darwin is based on XNU from NeXT, which itself is comprised of OSFMK Mach microkernel but with many of the things pulled into the kernel itself for performance.

And amusingly, the syscall interface from FreeBSD 4 or so.

IOKit is actually a considerable chunk of the kernel, particularly if you count the various "families", some of which remain open source. I'd say it's Apple's main contribution to the kernel at the architectural level. It's implemented unlike anything in BSD or Linux, as it uses C++ on the kernel side. (The user space side, IOKitUser, is written in C.) These days even Apple uses it for non-device-related kernel services - you can see what the iokit registry looks like on your system using the "ioreg" command or the IORegistryExplorer or IOJones GUI tools.

Darwin is actually a BSD variant, so it does tend to absorb some features from the rest of the bsds such as dtrace. Darwin is nigh-impossible to get patches into as an outsider ( I vaguely remember a blog post or maybe a talk about the process). The rest of the BSDs are fairly easy to get patches into upstream/mainline.

FreeBSD got dtrace in 2009 (FreeBSD 7.1). Mac OS X 10.5 already had dtrace in 2007.

That's because dtrace is a Sun thing, and there was a brief period where Apple and Sun were planning on some very serious collaboration. There was also talk of replacing HFS+ with ZFS at the time.

Darwin is not a thing. It's just Apple dumping the code to satisfy the licensing requirements.

Except it's BSD licensed and not necessary to share the code.

True. But Darwin is still not a thing :) There is no real community, bug tracker, etc. My point that it's just an Apple code dump remains valid, I think.

Open source doesn't necessarily mean open development. Companies like Apple embrace the former without the latter. It is perfectly valid to share your source code openly but not allow outsiders to do development. On a lesser scale, SQLite is another example of open source without allowing open development. You can't really contribute code to SQLite even if you wanted to. Do you think SQLite is not a thing? Heck, even RMS used to be super protective of Emacs.

I did not say it was not open source. I said it's just a code dump, and so far you have not really proved me wrong.

There is not even a proper repository with version history, as far as I can tell. Just this stuff: https://github.com/apple/darwin-xnu/commits/master

That's literally a code dump.

Apple has in the past provided a fully bootable Darwin operating system -- no Gui, but the OS worked just fine.

Your argument is pedantic where pedantry isn't warranted or necessary -- Darwin is a BSD and is BSD licensed, Apple doesn't have to share anything, but they do.

I'm not saying they have to do anything. I'm just wondering why people on this thread are pretending it's anything except a code dump. Look at the Github mirror I've given above. All the commit messages are like this:

  Imported from https://opensource.apple.com/tarballs/xnu/xnu-4903.221.2.tar.gz

I mean, this is a response to an article that illustrates the severe limitations of Darwin as an operating system.

In the area of kernel development, maybe, but Apple develops WebKit, Swift and some other projects in the open.

I have to wonder, why does Apple even make Darwin open source?

It doesn’t seem like it’s of much use to Apple—there isn't some large community of outside contributors submitting pull requests. So why do they bother?

It’s quite a good marketing line for the company.

How so? Their code isn’t up-to-date at all.

Marketing rarely relies on anything deeper that a sound bite. “MacOS core is Open Source” can play well in the media.

Licence compliance -- they took the Unix kernel and don't really want to give back but having now taken the kernel it now has obligations under that licence to make it available for others.

Interesting. Which license are you referring to?

Which license? I'm not aware of any obligations to release Darwin's source on Apple's part—hence how they're able to wait half a year before releasing the source of new versions.

One time I used Darwin open source to make a printf-sprinkled version of the dynamic linker to track down a bug that was causing our app to not start on about 1/3 of our install base. There were no error messages or console logs and the logs produced by feeding dyld environment variables were not helpful, so the ability to build a custom dyld was instrumental.

The problem wound up being that we leaked a sticky bit directory into part of our install, the installer inspector hid this from us, and apple changed dyld's behavior in a patch release to silently bail if it saw a directory with sticky bit set.

If you don’t mind the fact that they’re closed source, could the original Apple kexts be used for networking?

You’d be constrained by making sure the specific KPI is compatible and additionally that libsa is properly implemented so that it can patch the vtable to dynamically adapt to ABI differences

No need, you should be able to use Hackintosh networking kexts which are open source.

Oh, good point! Although, that wouldn’t have helped the article author running VMWare, as the specific kext he needed was Apple-provided and closed source.

VMWare wants Intel E1000, which has been supported for a while:

http://osx86drivers.sourceforge.net/ (Older) https://github.com/Mieze/IntelMausiEthernet (Latest)

I'm not really sure how useful it is for most of us -but it doesn't have to be. Not all operating systems are general-purpose.

I'm glad to see it continues to be maintained and published though I share the frustration other's have mentioned about how incomplete it is. There is no "more", no "less", no "halt", no "shutdown" -those are just a few of the "wtf" finds that pop up on casual examination.

I wonder if GNUstep works on this…

It does, insofar as GNUstep really works.

How doesn't it?

It's old, clunky, and neglected. It's still not up to OpenStep levels, which in itself wasn't really polished.

Bingo. And all the old timers refuse to change.

I am glad someone is keeping up with Darwin and making a version one does not need to compile to be able to use.

I wonder if it can run OSX command line programs?

If you can run it, and they don't link to cocoa, it should be able to do that just fine. Apple used to distribute a Darwin iso, I remember running it in the mid-2000s on an older Mac mini.

Can it run nvidia card Sk my old mac can be used; windows can run it but it optimised.

Reply on news.ycombinator.com