Baseline Mac OS X Support merged into FreeBSD package manager | Hacker News Reader

Baseline Mac OS X Support merged into FreeBSD package manager

github.com

229 pointsemaste59 comments

This pull request is a thing of beauty. Each commit is a little slice of (mostly) independent awesome. The project nerd in me just did a little dance.

That guy seriously rules.

I've found this to be true for most of FreeBSD and even other BSD-licensed code.

How the fuck do people do this with git? I loved doing pretty commits with mercurial's patch queue. Git is awesome but it seems like the only option for pretty commits is rebase (ew). All the git patch queue implementations are basically dead or unmaintained. :(

I haven't used it much so please forgive my ignorance, but what is wrong with git rebase?

Rewrites history. Also, Mercurial allows you to version control your patch queue (WIP commits).

How is this different from the information stored in `git reflog`, which allows you to rollback to commits before their rebase? You can just as easily revert back to the old version, since it never goes anywhere.

Besides, the need to perfectly preserve history in most cases is totally overblown AFAICS, especially local history nobody else sees. I don't care if a person who submitted patches to me made 20 separate minor commits to fix minor things in some case like a code review (e.g. "fix spelling", "fix 80 column violations", "rename this thing", "clean up code a bit and make it shorter re: code review"); those are superfluous and add no meaning to the actual work itself and can be rebased/squashed away in almost all cases. If they submit 20 minor commits that are each independent of one another and isolated, that's another story.

The alternative seems to just be 'have an ugly history littered with these commits' if "rewriting history" is so incredibly dangerous/terrible like it is always implied (which it is not, because you can always recover from it with the reflog until you push). But I'd rather keep my project history clean and clear; a tidy history is just as important as tidy code IMO. FWIW, I think the OP's set of patches are clear and do not constitute an ugly history.

The actual way to 'stop rewriting history' is to disable --force pushes, which does unilaterally rewrite history for all downstream consumers. This is also true for Mercurial. Rebase does not do this, or anything close to it.

As someone who reads and writes a lot of patches, this is an exceedingly common workflow. How is Mercurial any better in this situation where I don't want all that useless information?

> How is this different from the information stored in `git reflog`, which allows you to rollback to commits before their rebase? You can just as easily revert back to the old version, since it never goes anywhere.

Patch queues make the distinction between mutable WIP patches and finished commits explicit. Also, versioned patch queues make it safe to share WIP patches.

Just the user interface. It's a lot easier to fuck up during a rebase. MQ is a lot like git stash on steroids. Super easy to flit between multiple patches that I'm working on and put a change in the appropriate one. Wasn't uncommon for me to touch 3-4 patches in a random order over the course of a few minutes.

Let's say I'm working on patches 1, 2, and 3.

Situation: Currently on 3. I write some code and decide it should be in patch 1.

Git: Can't interactively rebase with uncommitted changes. Some sort of stash juggling.

MQ: Go down two patches. Commit the changes to the patch.

Situation: I'd like to introduce a new intermediate patch 1.5.

Git: Make a new commit on top of patch 1, rebase the other patches on top of it. If there's potentially conflicting merge errors they need to be addressed immediately—or I'm stuck leaving an alternate history of commit 1 around until I'm ready to deal with merge issues.

MQ: Go down two patches. Make a new patch. Deal with the merge errors whenever I go up/apply patches 2 and 3.

Yes, one trick I use a lot is interactive rebase. I do a lot of commits with "WORK IN PROGRESS" then git rebase -I HEAD~X to improve the history before pushing.

IMHO mercurial is superior to git in a lot of aspects, but there is main philosophical difference in the usage. Mercurial is more about indelible history while git people love to edit the history of the commits.

> git people love to edit the history of the commits

It varies. I avoid editing the history and make my opinion known whenever the topic comes up. Editing the history now requires making assumptions about what future developers might want or need; making assumptions like this is similar to a form of You Ain't Gonna Need It (YAGNI). It's better to give the future developers everything that we have, no matter how "ugly", and allow them to get what they actually need out of it.

In a previous life, our development workflow required rebasing as one of the steps for permission reasons (only senior devs could merge branches into master, so having everyone rebase first made sure conflicts were resolved by the branch's author)

These are actually the raw commits as I wrote them; I try to structure my work to build on itself in this way.

Interesting! What is the advantage of the FreeBSD package manager over existing OS X package managers like Homebrew or MacPorts?

It's a binary package manager vs homebrew that downloads the source, patches it and compiles it AFAIK.

Homebrew can "bottle" binary packages and "pour" them on install also.

https://github.com/Homebrew/homebrew/blob/master/share/doc/h...

Most of the standard stuff in homebrew is available in bottled form.

Which isn't to say Homebrew doesn't have its problems— setting up your own bottling infrastructure is difficult/impossible, and basic stuff like caching dependencies isn't there (cf. https://github.com/Homebrew/homebrew/issues/27457).

It can be done (I'm doing it in production), but it is not thought out with this situation in mind. I forked the formula repo at some point in the distant past, and untar it onto each machine. Then I manually manage dependencies when I tell it to fetch bottles from my own source.

A lot of the simplicity and 'it just works' nature of Homebrew is because it's meant to be used in one situation and that's it.

You probably want to be using your own tap and the `root_url` in `bottle do` blocks. I'm open to improving your use-case if you can file a feature request about what you're trying to do. Thanks!

Thanks for the reply. My main beef is that I don't terribly want to modify each standard formula (causing more work when I do sync with upstream), but I want the system to be completely self-contained.

Most package managers separate these concerns. There's the package build stuff (debian dir, spec files, etc.), there's the package install stuff (living inside the packages), and there's the repo stuff (apt, yum). Homebrew's inclusion of the install / dep portion inside the formula rather than inside the built bottle make it kind of impossible to fit into the method that I'm accustomed to. Not really sure that such a fundamental change is worthwhile for you to implement, since 99% of your users probably don't want to install their own bottles from a local repo and not connect to the Internet for anything.

If you create a tap rather than using a fork then this problem goes away as you'll never need to sync with upstream and can provide `root_url`s in your tap so your users can download things from e.g. local URLs.

For a long time bottling was just done by me on my local VMs. It's pretty simple really; `brew install --build-bottle wget && brew bottle --root-url youweb.com wget`, upload the outputted bottles somewhere and add the `bottle do` block into the formula file.

You can go a step further with this by using the `brew test-bot.rb` script (https://github.com/Homebrew/homebrew/blob/master/Library/Hom...) that we use for our CI. With that it's `brew test-bot --ci-testing wget && brew test-bot --ci-upload wget` which will write to the formula file, commit and push it.

Obviously there's a lot there that's hardcoded to our use-case but if you can create a feature request issue I'd be happy to work on better docs and more flexibility around this.

Hope that's useful!

That's fair, thanks for the response!

The use case I'm interested in would be running the test-bot automatically in a Jenkins/buildbot/whatever environment, where you point the bot at a repo of Formulae and it dumps bottles to a storage provider like S3. Once the dependency caching issue is resolved, I'm interested in getting this to a point of anyone being able to relatively trivially deploy it.

The other use case would be commercial entities wanting to distribute closed-source libraries or other software via bottle-only. So they'd want to run something as CI that would watch the private repo for a tag, and once tags are created, would automatically bottle those releases and push them out.

I'm not sure what the dependency caching issue has to do with bottling?

The latter use-case I don't have any particular interest in writing code for myself but I'd consider merging PRs for. The prior is possible with brew-test-bot if you do the uploading yourself (although I may be adding S3 support in the coming months). Using Jenkins should make all of this pretty trivial as-is.

The dependency caching is orthogonal to bottling; it's just that there's little incentive for me or anyone else to set up or document the bottling process (for ROS on OS X) until the created bottles are usable, which is not presently the case.

The bottles are usable? Homebrew and some other third parties already use bottles. If they aren't working for you: please create an issue so we can fix them. Thanks!

There's no point in the ROS community bottling ROS packages for Mac-based robotics work until it's possible to install them without a long dependency-resolution step. That's the usability issue.

It's also worth noting that what you've mentioned about "caching dependencies" isn't caching them on disk (we do that already and extensively) but instead being able to cache it in memory to make dependency resolution faster. I don't think that's basic stuff and most users don't seem to have a problem with the speed of such things.

The current arrangement works great for relatively shallow dependency trees; since that's what most homebrew packages have, it's not a big deal.

Having this support in freebsd/pkg may make it a better choice for distributing software on OS X with dependency resolution needs more along the lines of what apt and yum are able to provide.

Both brew and macports provide that nowadays. I still see an advantage in using a package manager that is used for a full distro, allowing for easier management of packages. I am aware that this benefit might never materialize, but it's good some people try.

This might not answer your question completely, but here's a thread from the last time I recall this discussion happening: https://news.ycombinator.com/item?id=8402542

It's been a while since I used MacPorts, but I imagine there is some benefit to using the same packaging system between OSs and not having all package descriptions as executable ruby scripts.

This change was written by Landon Fuller, one of the founders of MacPorts. There's an explanation for it in a comment on the pull request: https://github.com/freebsd/pkg/pull/1113#issuecomment-680639...

This change doesn't mean very much in isolation, but will hopefully be the beginning of some interesting collaboration by packagers on both platforms.

It's good to combine efforts. It means potentially a bigger selection of packages that are available to install and more developers maintaining the project since it now has a wider potential user base.

IMHO, Homebrew offers a lot of advantages over traditional package management tools. It's simple to use, offers a really easy way to add your own packages, and contributing upstream is quite painless. MacPorts has always been reminiscent (to me, anyways) of the FreeBSD Ports Collection...it might even be based on it I'm not sure. So this seems like a prelude to a possible sunsetting of the MacPorts project in favor of simply using FreeBSD's package management tools on OS X. Not sure it's going to draw me away from using Homebrew just yet, but it's nice to know that there's a continuing effort to have OS X be a part of the UNIX-like OS community.

(fake edit: it's been a LONG time since I've used FreeBSD. feel free to chime in about how wrong I am regarding the ease of adding ports on your own private or public server if things have changed)

I don't think MacPorts is a direct derivative of the FreeBSD Ports Collection (someone can correct me if I'm wrong), but it's definitely inspired by FreeBSD Ports-- one of the original MacPorts coauthors (Jordan Hubbard) was the original developer of FreeBSD Ports.

I won't be switching anytime soon, though. I moved to Homebrew a long time ago and haven't looked back-- MacPorts feels opaque and isolated in comparison.

>MacPorts feels opaque and isolated in comparison

What does this even mean? If it means "uncool" then, yeah, I get it.

If you are comfortable with Ruby, then it is easier to build packages for Homebrew, and since it isn't centrally managed like MacPorts, things generally get packaged quicker for Homebrew. On the other hand, MacPorts has a long track record and has generally been bombproof for me across 3 or 4 OS updates now.

As an existing MacPorts user, I looked at it from the standpoint of "what is this doing for me that MacPorts doesn't?" I didn't see anything that seemed compelling enough to switch.

I switched to Homebrew a long time ago, when MacPorts had a large advantage in number of packages, freshness, stability etc.

For me, MacPorts' main mistake was to go all in with the "storage is cheap, CPU is cheap" canard just as everyone was abandoning desktops for laptops. MacPorts decided to build its own copy of almost everything that the OS distribution provides, to avoid having to deal with version differences between OS X releases. That might have been a decent tradeoff on a Mac Pro, but when you have a laptop with a small HD (and, later, an even smaller SSD), it is a real issue to have to permanently waste several GB on duplicate copies of Perl, Python etc. just because you want to build the GIMP; to say nothing of the hours it would take to build them.

Homebrew's big selling point, for me, was that it could just use the libraries that already came with OS X.

> to say nothing of the hours it would take to build them.

FWIW, nowadays (for a few years?) macports has binary packages, so that is not as big an issue as it used to be, i.e. installing ImageMagick would take hours, now it just usually gets all of the packages and dependencies as binaries.

I think used space is not a huge problem either (I have 130 packages and it uses 1GB), the real issue is that it's anti-intuitive that macports does not remove inactive stuff and does not autoclean, so you end up with a lot of wasted space for stuff you had no interest in keeping.

I used MacPorts for years and eventually gave up and abandoned it for Homebrew. It simply became too common that I would come across a project which had directions for setting things up with Homebrew and when I went to try to do it using MacPorts a required package would be missing, outdated, or broken.

I think the world sort of moved on from MacPorts and honestly I think it's a few years too late for big OpenSource package system to come to Mac OS. I think that Apple is on a long term course that excludes (or perhaps confines or quarantines) Open Source. I also think that package management is really behind the curve in a computer science sense and a 'next, next-generation package manager' needs to come about.

I've stayed away from Homebrew because of its insistence on taking over /usr/local. I understand the reasoning, but I use /usr/local for my own installs and Homebrew doesn't play nicely in that kind of mixed environment. I would use it if it defaulted to /opt/homebrew or something similar.

I'm looking forward to this new package manager alternative.

Isn't there a way to have Homebrew monitor a "make install" operation so that Homebrew can manage (or at least be aware of) your own installs?

Possibly. I'll check it out again when I have a chance. Perhaps there have been improvements in the past year.

You can install Homebrew in a different place if you want. I have an install inside my home directory.

How does it work in practice? I tried it about a year ago and it still stomped on /usr/local for some packages. If I remember correctly, that was due in part to the package not making it easy to install elsewhere, but Homebrew didn't handle it well.

It always worked fine for me. /usr/local is not even writable by the user I normally use, so it would have caused an error if Homebrew tried writing there - which it didn't. If you're seeing issues when installing to alternate locations, file a bug report.

It definitely shouldn't do that. Most likely that's a fault with the way the formula or Makefile is written, like hardcoding `/usr/local` instead of using $PREFIX.

Mainly just another sign for Apple that says: "Welcome to the 21st century! In this *nix era we have package managers!"

NetBSD pkgsrc has supported OS X for almost a decade now, including a complete binary build IIRC thanks to Joyent

Naive question: does the pkg(7) format used by FreeBSD have anything to do with the .pkg files Apple distributes? And if not, why not try to be "OSX native" using .pkg files instead of "BSD native" using pkg(7)?

No, Apple's .pkg files are proprietary to Apple's Installer.app. They are XAR archives containing metadata generated by the package creation GUI.

It goes back to the NextStep days too.

FreeBSD's binary packages aren't .pkg, but rather .txz (.tar + .xz).

http://pkg.freebsd.org/freebsd:10:x86:64/latest/All/2bsd-dif...:

  -rw-r--r--  1 root     wheel          626 Dec 31  1969 +COMPACT_MANIFEST
  -rw-r--r--  1 root     wheel          921 Dec 31  1969 +MANIFEST
  tar: Removing leading / from absolute path names in the archive
  -r-xr-xr-x  1 root     wheel        30616 Oct  3 10:49 usr/local/bin/2diff
  -r-xr-xr-x  1 root     wheel        11256 Oct  3 10:49 usr/local/libexec/2diffh
  -r--r--r--  1 root     wheel         2823 Oct  3 10:49 usr/local/man/man1/2diff.1.gz

This is similar to other BSD binary package formats.

This is great news. Pkgng is very easy to use and achieves a very good combination of binary packages and the FreeBSD ports tree when you want to compile stuff yourself. Which happens more often than expected admittedly.

You can already have that with pkgsrc for osx - joyent maintains the osx binary packages see http://pkgsrc.joyent.com/

(pkgsrc is the NetBSD version of FreeBSD ports, but its always been portable to other systems).

Hmm I didn't know that Joyent maintained their own packages for non-SmartOS platforms. The OS X packages seem to be limited to 32-bit versions and even though you didn't mention this I noticed their Linux packages seem to lag behind pkgsrc upstream. What is the advantage of using their packages over the OS X and Linux ones offered by NetBSD? https://www.pkgsrc.org/#platforms

AFAIK they are the only binary build at present for OSX. You can use source for 64 bit OSX of course http://wiki.netbsd.org/pkgsrc/pkgsrc_64bit_osx/

I have generally used source not packages for non NetBSD platforms, other than when I tried out SmartOS and it is all set up. If you have a chunk of disk space, a fast computer and some time you can compile them yourself, and if someone wants to support osx 64 bit that would be great.

I'm building fresh 64bit packages on Mac OS X Yosemite. They will be uploaded in a few days at http://pkgsrc.saveosx.org/Darwin/2014Q4/x86_64/All/. Oh and they are GPG signed. Documentation at http://saveosx.org

I recently installed pkgsrc for osx, and so far so good. (I've never liked the idea of installing software by compiling it.)

However this project could really use some nice webpages and instructions to promote it. It seems like most Mac devs have never heard of it. And much of the info is pretty minimal and seems like it assumes you understand NetBSD packaging.

For an introduction to pkgsrc on Mac OS X, please check : http://www.cambus.net/using-pkgsrc-on-mac-os-x/

BSD ports on Yosemite, what could go wrong LOL!

Reply on news.ycombinator.com