Hacker News Reader

Hacker News

Comment by jcfields | Hacker News Reader

Parent Full thread

jcfields

They've actually been warning users since Catalina (if not longer) when the following are run:

python

irb

php --version

tclsh

For PHP, it also shows up in Apache's server string:

Apache/2.4.46 (Unix) PHP/7.3.24-(to be removed in future macOS) mod_perl/2.0.9 Perl/v5.30.2 on hostname

Personally, I've been hoping they'd do this for a while since it's annoying having two versions of each interpreter installed (the system one that can't be updated and the one I installed myself via Mac Ports). Anyone who uses these languages much probably already has a package manager installed. I think they should leave Perl, though, since it's as much as Unix system utility like awk or sed as a programming language.

View on HN

While the warning shows for python2, it doesn't show for python3, implying that it's here to stay for now:

  % /usr/bin/python

  WARNING: Python 2.7 is not recommended.
  This version is included in macOS for compatibility with legacy software.
  Future versions of macOS will not include Python 2.7.
  Instead, it is recommended that you transition to using 'python3' from within Terminal.

  Python 2.7.16 (default, Mar 25 2021, 03:11:28)
  [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>>

  % /usr/bin/python3
  Python 3.8.2 (default, Dec 21 2020, 15:06:04)
  [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>>

Python is to Unix/Linux/Mac what PowerShell is to Windows. It's here to stay.

No it’s not. And it won’t

I am pretty sure bash is the powershell of Unix/Linux/Mac

bash is more akin to cmd/batch then Powershell.

there is a TON more you can do with powershell than you can with bash. Plus object pipelines are superior to text pipelines

You can also install Powershell on Linux. So I would say Powershell is the Powershell of Linux

you can install bash on windows too.

I don't think powershell is even the powershell of windows. You get some of the better sysadmins using it, but working in the c#/windows world I see developers rarely touching powershell, certainly nowhere near as often as you'll see Linux develops writing a shell script. I wouldn't be surprised if C# was more used for "scripting" purposes than powershell, it usually ends up shorter and more readable anyway.

That's because it's not part of the base OS:

   % /usr/bin/python3
  xcode-select: note: no developer tools were found at '/Applications/Xcode.app', requesting install. Choose an option in the dialog to download the command line developer tools.

Note how despite not coming with the OS, `python3` is still placed in the Apple-only `/usr/bin/` directory. And a binary exists at that path either way, it's just a stub by default. Because none of this is confusing at all.

I don't like how Apple handles this. If you have the developer tools installed, it's not especially obvious what is part of those tools and what is actually bundled with the OS. And so it's not surprising that some scripts I've come across will silently break in semi-odd ways without the dev tools.

Also, why is Python a developer tool, but Perl isn't? Or ZSH, for that matter? Does the distinction actually make any sense?

> Note how despite not coming with the OS, `python3` is still placed in the Apple-only `/usr/bin/` directory. And a binary exists at that path either way, it's just a stub by default. Because none of this is confusing at all.

There's long been problems with this on UNIX systems. For a long time there's generally been three categories of software to track (OS provided, third party managed, manually compiled/installed), and two different locations to put them (/usr, /usr/local. / is different and has/had other constrains and reasons behind usage).

OS provided packages go into /usr, stuff a user manually compiles and installs goes into /usr/local, but where does a third party package manager put stuff? It's not OS provided, but it is packages and maintained, so /usr seems valid. At the same time, it's user installed, so /usr/local seems valid. Both make the other used of those locations more complicated to reason about.

> Also, why is Python a developer tool, but Perl isn't? Or ZSH, for that matter? Does the distinction actually make any sense?

Possibly because some are used by the OS to actually do things for the OS (not sure if Apple finally got rid most those dependencies or not).

The problem of the OS shipping a component for it's own needs that gets used by others is a decades old problem at this point. It's complicated because if it's available users to use, they'll use it, and when the OS wants to update or change it in some way it can break user assumptions. If users update it system wide, they can break OS assumptions.

> but where does a third party package manager put stuff?

/opt

> The use of /opt for add-on software is a well-established practice in the UNIX community. The System V Application Binary Interface [AT&T 1990], based on the System V Interface Definition (Third Edition) and the Intel Binary Compatibility Standard v. 2 (iBCS2) provides for an /opt structure very similar to the one defined here.

* https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.htm...

MacPorts uses /opt/local for example.

Yes, that's where I would suggest putting it, but the reason projects don't is either because they aren't aware of it, feel it's a bit more convoluted to make sure you update where PATH is set, or that /opt is generally used for singular applications, not an extended hierarchy such as /opt/local (which is actually a truly weird name, /opt/macports/ with /opt/macports/bin, /opt/macports/lib, etc would make more sense). It makes sense to use /opt from an "application" point of view, but I think the problem is many of these package managers see themselves as more than that, and as pseudo-system components, which clouds their reasoning.

The LFS spec you linked to is actually ancient at version 0.65, the latest is 3.0 and available here[1]. It's a bit "do what you want", but it really expects the "local" part of /opt/local that MacPorts uses to be some sort of identifier for MacPorts.

That said, it's the Linux Filesystem Hierarchy, so it only applies to Apple stuff inasmuch as it's common practices for UNIXes in general. And it's not like the major Linux distros follow it fully either. They're happy to break from it where it preserves some historical layout their OS has had that they think is easier for their users/admins.

1: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

The reason for zsh is simple enough: Terminal is a standard Mac app, and there are some system maintenance things an ordinary user might do which require the command line.

So there needs to be a shell, and that's zsh now.

Apple switched to zsh because they're allergic to later GNU licenses and the version of bash they were shipping was absurdly out of date.

Yes, that's the practical reason. But what's missing is a conceptual reason, something that makes it easy to remember what is and is not a developer tool.

Possible approaches that would be less confusing than what we have today:

1. Ship Python 3 with macOS.

2. Ship all of the command-line developer tools with macOS, including clang et al.

3. Bring back the Snow-Leopard-era /Developer folder, and put developer tools there.

4. Put developer tools in /usr/developer, or /usr/local/developer, or /usr/local/Apple, to balance both UNIXness and clarity.

5. Don't ship Python at all. The Python foundation already distributes a .pkg installer for Mac. If xCode needs Python, follow your own Human Interface Guidelines and put it in the Application bundle.

All of these approaches have downsides, but I still think they'd be clear improvements over the current situation. Which is to say, I want Apple to figure this out!

It isn't an allergy.

At the time the GPLv3 was written, the GPLv3 was deliberately crafted to put commercial companies into a conflict situation.

(A) Give up selling audio with DRM (DRM did go away) (A) Give up selling video with DRM (A) Give up using HDMI as an output port (A) Give up using digital signatures to secure the integrity of software

-OR-

(B) Give up using GPL v3 software, GCC, Bash

Pretty sure most customers actually want many of the things in (A) and a minority actually care about (B). The items in (B) can be post-installed by those that really want it. Sure those in (B) have a loud voice, but, HN, are they really representative (democratically a majority)?

How would you make the choice between (A) or (B)?

I would bet that if a subeoena for FSF/GNU email was issued there would be a lot of messages related to manipulation and coercion for licensing and re-licensing. The sort of stuff, when associated with companies turns into scandal, lawsuits and monopoly investigations. (GCC almost got to be what could be considered a monopoly compiler)

I do believe that open software has a place and is a really good choice. My opinion is that the GPLv3 is completely toxic, spawned from negativity and is ultimately anti-open source. I will never release any software under the GPLv3.

> Give up selling audio with DRM (DRM did go away) (A) Give up selling video with DRM (A) Give up using HDMI as an output port (A) Give up using digital signatures to secure the integrity of software.

Does GPLv3 really prevent an OS from doing those things? I think it would just prevent it in the same executable as GPLv3 software?

I don’t actually understand Apple’s “allergy”. On the iPhone I’m sure they object to the Tivoization clause (which coincidentally is why I like to use GPLv3), but I’m pretty darn sure macOS doesn’t fall afoul of anything...

Python 3 is required for the ability to install coremltools which makes it a developer tool.

Python 2.7 might be there because old software is there for compatibility reasons for software possibly for mathematical software that never really gets updated. Removing it in future versions makes sense because of the fact that if you haven't updated it won't work on ARM Macs.

The best way to make someone upgrade is to deprecate and then remove access. At minimum the worst case scenario is they roll their own and install Python 2.7.

python3 is a developer tool because it is the extension language for the lldb debugger. For example, I have python3 scripts for formatting my custom C++ data types, and that formatting also works in Xcode's visual debugger (which is presumably using the lldb API).

Previously, lldb's extension language was python2. This changed a release or two of Xcode ago. At that time, to use lldb scripts with that new release of Xcode you either had to run on Catalina (which added python3 to the existing python2) or install your own python3 and point lldb at that. This actually was what pushed me into moving to Catalina on my development system, because the Unicode handling would be much easier in python3.

As I understand it, Monterey and the Xcode that is released for it move the necessary python3 installation from the operating system itself to Xcode. I'm fine with that.

(I would not be fine with Apple telling me I have to go waste my time figuring out how to install a suitable python myself, when I never had to before. And indeed, many of their own developers may be like me and know a little Python but are in trepidation of the env thing.)

The way it was before meant that if they wanted to update python they had to release a new version of the operating system. This is clearly an imposition on the vast bulk of macOS users, who are not developers.

Nit: It’s not that /usr/bin/python3 gets replaced. It’s always a stub, but the stub execs the real python3 if found.

#!/usr/bin/python

How exactly is the above supposed to work when /usr/bin/python doesn't exist. If Apple chose to break the #! contract by doing something else, how would you feel about that?

The scripts should be using #!/usr/bin/env python

This answer is both a great answer, but also a terrible one.

This answer implies all "python" binaries across all operating systems and distributions for all time, are backwards and forwards compatible, no work needed. Guaranteed 100% equivalent.

What about Python2 v.s. Python3?

This isn't true and cannot ever be. This happens for other scripting and interpreted languages.

This approach means that the burden of choice and setup is transferred 100% to the person running the script. If you have 2 scripts that require different dependencies, then you will encounter this problem. I think it is this that encourages folks to include hard coded paths to enumerate explicit dependencies.

Lots of the "package" managers for these scripting languages also don't deal with this very well. They advocate a "do it my way" or "do it yourself". Different languages do it different ways.

Ultimately, the person wanting to run the script, just wants to make it run -- they will follow the instructions to make it work and along the way will make "global" changes -- which will impact what will happen for any other script in the future. This will likely be diverged from anyone else who has a "base install".

The above is mostly about my observations with using Ruby, not Python. However, few attempts in using pre-packaged complex python "recipes" has always resulted in similar conflicts.

> This answer implies all "python" binaries across all operating systems and distributions for all time, are backwards and forwards compatible, no work needed. Guaranteed 100% equivalent.

Practically speaking, how does using env versus the absolute path make this any better or worse? You as the script author don't know what version of Python/Ruby/Bash/etc lives in /usr/bin. Maybe you could do some kind of automagic detection based on the user's OS, but any such assumptions are likely to go stale over time.

All env does is give users a choice about where to put their binaries. The versioning situation is a real problem, but I don't think using env makes it better or worse.

I find the MacPorts approach to be very developer friendly and I wouldn’t really want to use a “naked” macOS install as my work machine. MacPorts makes it look like a very credible BSD workstation.

I really have no issue with Apple trimming down the surface of the OS to a minimum.

Totally agree. I think anyone developing with these tools would too. It's not as annoying as say, RedHat, but out of date "system" versions are just obnoxious.

I don't know if any "native" apps have leveraged them, though. That would be the real risk of disruption.

Also, as someone who never uses PHP, I'd really rather not have it on my machine. Scripting languages are a pretty big attack surface, even if it's just to pivot after initial compromise. I'd rather not have one there that's not being used.

I'll also make the inflammatory statement that people should just use sed and awk in shell scripts and leave Perl for when they really want to write Perl.

It is weird to me the number of times 90's Linux Distros used Perl for things that are easy in awk, awk for things that are easy with cut, and other commands for things that are easy with POSIX shell baked-in facilities. I guess I'm just a shell snob.

cut usually doesn't do what you want it to do. Most of the time you want to split on sequences of whitespace (which may or may not be tabs, it's not easy to tell from the output of the program). cut doesn't let you do this, which is a shame, as it would've been a sensible default. It also does weird things like print out the whole row if there's no instance of the separator on the line.

So because you can't trust cut to do what you want, you'll probably reach for something like awk '{print $3}' instead.

Yeah, after I remembered that `cut` exists shortly before needing to grab some fields of an input, I thought "oh, I'll use `cut` instead of `awk`". And lo and behold, `cut` couldn't do the thing I wanted. Why learn two tools when one will do? The only reason I can think of is the principle of least privilege, but if you can convince me it applies, I'll convince you should write a single tool that adheres to it instead of using countless tools for any given transform/privilege pair.

I tend to start writing in bash/sed and going on to regret it as the script gets more involved and needs a refactor in perl.

Most of the time the script doesn't get more complicated and it's fine with cut/sed/wc etc, but at some point it tips the balance and I grumble to myself.

Perl was invented to replace the shortcomings of hacking together sed + awk + shell, eg. multi-dimensional arrays.

Only if the said script is less than a dozen lines

> since it's as much as Unix system utility like awk or sed as a programming language.

Even those tools suffer from Apple's lack of fucks given. I often have to replace included system commands with the homebrew equivalent because Apple's is missing some common functionality.

I wish Apple would just adopt homebrew as a firstparty package manager.

I'd plump for Macports or pkgsrc long before Homebrew.

Homebrew should first come up with basic functionality... like you know not assuming the OS will only have one user account ever. I was quite shocked when I found this out about the oh so amazing package manager.

Before someone shares some bizarre workaround: no thanks.

The last time I bothered with it, it complained about my use of `sudo` but wouldn't run unless I was logged in as admin - not a temporary bump, just logged in as an admin. Sudo isn't alright but running as admin is…? I really can't work that one out.

It also broke so I uninstalled it. When I need to I go straight to their formulas and install by hand, as a non-admin user.

The issue (in my experience) is that neither Homebrew nor Macports provide a "full fat" package management experience like pacman or even apt. Not only are they missing packages, they're missing features and are loaded with outdated builds. If Apple made a first-party package manager that didn't suck, I'd probably have a Mac on my desk today.

Homebrew is a piece of crap. There's no away to install specific versions of software (unless you dig through the git repo and install using a commit hash). It has a scorched earth policy of wiping out dependencies willy nilly. One time I installed Python 3.8 and it wiped out the version of openssl used with Python 3.7 I used for a client project.

You can actually `brew search python@` to see the available versions (major/minor only, not patch) and then `brew install python@3.7` to install an older version. Caveat is that this depends on someone having made a separate formula, under the correct versioned naming scheme, and uploaded it to homebrew. There doesn't seem to be a python2 formula, for example.

I think it's a difference in philosophy? Brew does a "good enough" job for the most common use case, supplementing Apple's janky *nix environment with commonly-needed tools, without a built-in package manager. It doesn't try to do complex simultaneous versions... in my (admittedly limited) experience, if you need to do that, you either do that in something like a project-based setup (npm/composer) or in a VM/container with its own packages.

In your situation, when you have multiple pythons running in the same environment, how does anyone given script know which one to call? What's a better way of doing this?

MacPorts did receive official resources from Apple for a long time. Then they stopped.

Apple did toss some ARM machines (DTKs I believe) to the Homebrew project.

You could use homebrew to install powershell.

I can’t tell if you’re joking or this is actually possible. Is it?

https://docs.microsoft.com/en-us/powershell/scripting/instal...

From the horse's mouth

You can also install it on Linux if you wish.

Reply on news.ycombinator.com