Hacker News Reader

Comment by oefrha | Hacker News Reader

Related: ongoing compatibility table compiled by the Homebrew team: https://github.com/Homebrew/brew/issues/7857. (Please respect the team and refrain from polluting the thread. If in doubt check the comments marked off topic.)

View on HN

No GCC support if I read that correctly and the effort is expected to take months of full time work. I wonder what that'd break or is Clang significantly more common on OSX now?

Well, for one, this means that Apple Silicon has no Fortran compiler, which breaks a decent chunk of the scientific computing ecosystem.

is there not an flang project? Is it any good?

There's several! We tried all of them without much luck :(. The most complete seems to be the old flang, which was basically an open sourced version of the PGI compiler hacked up to generate LLVM IR by concating strings to generate textual LLVM IR. However, that project is basically abandoned and also depends on LLVM 7, while you probably want at least LLVM 10 to generate good code for Apple Silicon, so somebody would have to do that porting work, which doesn't seem like a worthwhile investment.

Hopefully apple will do something about it as they did advertise their new system would be great for machine learning and all, which seem to be moot without many popular numeric libraries that directly or indirectly use fortran. Would be great if clang/llvm suddenly gain a great open source fortran compiler just because apple moves to arm.

Does anyone in the scientific computing ecosystem use Macs?

I can't think of...any program abyss dynare adios2 eccodes apache-arrow ensmallen apache-arrow-glib fastme arcade-learning-environment fftw armadillo field3d arpack flann arss fobis astrometry-net gdal asymptote gdcm aubio gmic augustus gmsh biosig gmt boost-mpi gmt@5 bsponmpi gnuradio caffe gr-osmosdr calceph grace cdo graph-tool ceres-solver gromacs cgns gwyddion clp hackrf coinutils hdf5 cp2k hdf5-mpi cpl hdf5@1.10 csound hdf5@1.8 dartsim hypre datetime-fortran imake dlib imgproxy dungeon inspectrum dvc ipopt 147 formulae.

Edit: Forgot to libtorrent-rasterbar made front in that sphere that aren't Windows or Linux?

Lots of students have Mac notebooks, but want to run julia/scientific python/etc.

Yes, that would be me, and half if not most of my colleagues.

Very much yes

Sadly, yes. I've clients who still prefer to work on their Mac laptop and desktops. Large runs still go to Linux clusters, but small, debugging runs occur locally.

Per the parent, the lack of Fortran compiler causes an issue because some BLAS/LAPACK libraries align their symbols with whatever a Fortran compiler would do. As such, unless there's been a recent change, things like CMake need the compiler to determine the name for linear algebra routines.

I phrased the above as "sadly, yes" because supporting macos has been a pain and costs a fair amount of labor. In my case, Homebrew is required and there a bunch of specialized quirks in the build and packaging system that I'll contend are more difficult to deal with on macos than on Linux and Windows. Lastly, the requirement that macos be virtualized only on mac hardware complicates automating the build and test system since those processes must be relegated to a macos machine and I'd really rather not do primary development on a mac. Frankly, I'd pay Apple a not small amount of money to virtualize macos legally on non-mac hardware.

BLAS/LAPACK isn't that much of an issue, since Apple probably will need to ship their own implementation of this for Apple Silicon to get reasonable performance, and they can implement this in C, assembly, or whatever they want.

If they don't do that, then you can just use any "generic" C BLAS/LAPACK implementation, performance will be poor, but that's pretty much the only option if your hardware vendor doesn't support BLAS.

I guess people can write their own BLAS as well, but if you want to re-implement BLAS in 2020, then Fortran is probably not the best language to do that.

Apple's copy of LAPACK has not been keeping up with upstream and is missing some of the newer methods and bug fixes. Also while those are certainly the big two dependencies they are by far not the only ones.

Sure, lots.

I see the opposite from you: Mac laptops, crunching under Linux, very rarely see windows in the wild. I know windows sales are still robust so this is what I see but am not sure how far it can be extrapolated.

A huge chunk of scientific and numerical libraries rely on Fortran, for which 64-bit support is seriously lacking on Windows.

Doing this work is significantly easier out-of-the-box on Mac/Linux device.

Data Science is my main job and the first thing I did at my last job was ask if I could wipe windows and put Linux on the desktop they gave me.

> A huge chunk of scientific and numerical libraries rely on Fortran, for which 64-bit support is seriously lacking on Windows.

How so? mingw-64 exists, and so does Intel Fortran (with good integration with the VS IDE as well).

Intel and PGI Fortran work just fine on Windows, with VS integration as pointed by others already.

Yes. Most people in my lab have a MacBook (Pro) where we do most of our coding, and we ssh into a Linux cluster to run most of the compute. We have desktops but these are all Windows/Linux dual boots. We all practically need to use PowerPoint as well, so Mac is nice to have here since it can’t go on Linux laptops, and nobody wants to constantly reboot all the time.

Walking down the hallway in my physics department (pre-covid lockdown), I see almost all Macs. Some of them run windows, but most are apple hardware and software. Windows is quite rare when it comes to professors. It's more common among the grad students though, about tied with MacOS.

It was like this in my previous department as well.

Would it would work under Rosetta 2 (Apple's x64 emulation/translation layer) until the aarch64/darwin ABI can be ported to GCC? Or is there a reason to assume it might not?

Yes, it works mostly fine under Rosetta. However, it is also precisely the area where you don't really want a translation layer, because you're spending a lot of time doing heavy compute.

That makes sense. I'm hoping it's going to be an early adopter problem that'll have been solved by the time the transition is deemed complete.

Many compute heavy tasks are using Metal and those should continue to function at the full speed of the hardware.

I doubt GCC is, or even could be, using Metal.

I am referring to modern HPC like workloads, they will be using the GPU. Not saying GCC is using Metal in any way whatsoever.

HPC on macOS? Using Metal instead of CUDA/OpenCL? Sounds extremely rare.

Not sure if it's changed, but last i checked a few years back, most of the BLAS/LAPACK etc. stuff was f77, which should 'do' with f2c (albeit, perhaps not as fast) and is at least partly why f2c was hosted at netlib..

I had no idea, that explains so much! I was wondering how these open source numerical systems were building fortran code "so well", as in not a huge operational burden. My understanding is that the OSS fortran compilers are used for dev/debug, but when it comes to production, commercial Fortran compilers for your platform vendor offer much better performance.

Won’t this work? https://lfortran.org/

I just did a quick google search for it. I know nothing more than that it is FORTRAN compiler for LLVM. (It claims to be an alpha release)

Perhaps! It's one of the two candidates that might actually work in the near term. It didn't quite seem to work out of the box and given the newness of the project, we are a bit cautious, but there's some folks looking into it, to see if they can work through the issues.

This is incorrect, the LLVM project has an official Fortran compiler in tree called flang: https://github.com/llvm/llvm-project/tree/master/flang

It is quite hard to write portable Fortran code, but code that at least compiles with PGI should compile with that as well.

I wonder why this is allowed to persist? Have there been any studies across benchmark suites relative to compiler and platform? What was the spread? Was this general purpose, it works well enough speed or was this I need to get 1.75% better on task x or my super computer budget is blown?

I find the whole Fortran ecosystem and lineage fascinating for some reason. Like an alternative universe where integers are only used for loop counters.

If you actually read the code, it turns out that said compiler is not yet capable of generating LLVM IR let alone binaries.

Wait what? We were using the old flang project just fine. I thought they just moved it in tree and keept developing it.

There was old flang which was basically just the PGI compiler stripped of its backend and with a piece bolted on that converted the compiler's internal IR to LLVM by basically dumping out strings and then parsing them back in. Many people in the LLVM community thought that this was not a maintainable long term strategy for a Fortran frontend, and after much debate, the flang team decided to start over on Greenfield compiler called f18, which was to be architected more like clang. Then Chris released MLIR and said team decided to target that rather than LLVM directly. The only thing that was kept from old flang were some of the runtime libraries. f18 was recently merged into the llvm-project repository whereupon it was renamed flang. Old flang is dead, but the terminology is a bit confusing now. Additionally there are two external LLVM Fortran compiler efforts: lfortran out of the US national labs (which also paid PGI for flang) and fc, which I don't believe is currently open source, but is being developed by as compiler consulting shop on India called CompilerTree (not sure who's footing the bill for that one). Anyway, long story short, none of them seem to work. Old flang generates binaries, but is abandonware at this point, so not the correct foundation to build on. New flang doesn't even generate LLVM IR yet. FC doesn't seem public yet as I said. Lfortran may work, but the authors say it's very alpha and it certainly didn't work out of the box. I know some folks are looking into it to see what would be required.

Apple sucks for scientific computing. It does not let you even load your own generated shared files, because Apple determines it is too dangerous for the health my own purchased computer. I have never had that problem on Windows or Linux. That said, you can get gfortran by as simple as typing brew install gcc-9, or some similar command (requires brew installation).

By Homebrew policy, anything requiring a C/C++ compiler but doesn't compile with clang is out. So gcc not working shouldn't disrupt the overall Homebrew ecosystem too much. The biggest problem is gfortran, which is what the great majority of gcc dependents use it for (just `rg 'depends_on "gcc"'` in homebrew-core and all the "# for gfortran" comments should make this obvious).

Btw, here's the list turned up by a dependent search in core:

  $ brew uses --recursive gcc itk                             numpy                           qrupdate itpp                            numpy@1.16                      r jags                            nwchem                          ratfor joplin                          octave                          rawtoaces json-fortran                    ola                             raxml-ng kahip                           ompl                            reprepro kallisto                        open-mpi                        root kim-api                         openblas                        scalapack lammps                          opencoarrays                    scipy lapack                          opencv                          shogun libbi                           opencv@2                        simple-tiles libmatio                        opencv@3                        siril libxc                           openfast                        skymaker liquid-dsp                      openkim-models                  sratoolkit luaradio                        osi                             suite-sparse mapnik                          osm2pgrouting                   sundials mapserver                       otf2                            superlu minimodem                       packmol                         synfig mlpack                          pcl                             urh mpdviz                          pdal                            veclibfort mpi4py                          petsc                           vips mpich                           petsc-complex                   virtualpg mvtools                         pgplot                          visp ncmpcpp                         pgrouting                       vtk nco                             plplot                          waon ncview                          pnetcdf                         wxpython nest                            points2grid                     zita-convolver netcdf                          postgis networkit                       proteinortho ngspice                         qd --include-build. That would add 4:
  boost-python3 btfs libtorrent-rasterbar openimageio page today btw...

Weird to see wxPython on the list..

Any numpy (which uses gfortran) dependent is on that list. I suspect some could partially work by sacrificing some functionality, or maybe just run slower (disclaimer: I have no idea of numpy internals).

does numpy leverage blas?

yes

Yes. https://github.com/numpy/numpy/blob/62fa23c44fb49c1d238e1de4...

well, that's probably where the fortran dependency could creep in, no?

Wouldn't that entail swapping out all the underlying Fortran that makes NumPy so lovely?

Interestingly at WWDC Apple said that they are contributing patches to NumPy to get it to work on Apple Silicon

GCC supports ARM targets and the macos platform. It took a couple of hours to add support for the new target triple to Rust, so it should take about the same work to add support to it for each of the GCC frontends (10x couple of hours ~ couple of days, maybe a week).You don't need to implement neither hardware support nor OS support, just combine the two things that are already implemented, and fix a couple of incompatibilities, but not many (some parts of OS support will be x86 only and these would need fixing, but fixing these should be trivial since you can c&p from the arm+iOS code).
Why would it take months to add support for this to GCC? GCC's architecture would need to be really flawed for this to be true, i.e. , requiring you to re-implement a big chunk of hardware support for each OS, or full OS support for each new hardware. Given how many architectures and OSes GCC supports I just can't imagine this being true.
Unless by this work taking months, you mean "not doing anything the first 99% of the days, and doing all the work in the last couple of days". In that case, sure, this can take as long as you want.

Yeah, I was surprised by that statement too, but then again, I know nothing about how GCC is architected. I'm hoping somebody will come along and just fix it in a few days, but who knows.

https://developer.apple.com/library/archive/documentation/Xc...

IMO Apple should be sponsoring that work.

IMO Mac OS X should come with its own integrated package manager and corresponding central repository for distribution of various open source software packages. A package manager that worked well with the App Store and Xcode.

This is their only real chance to keep Devs from migrating to Windows 10 with WSL2 or baremetal Linux. I couldn't believe they didn't announce any thing for Big Sur, I could have sworn they would if they really cared about non-macOS/ipadOS/iOS devs. Especially that they're also dropping Perl/Python/Ruby frameworks.If anything, they could have just low-key embrace brew and take it from there. They didn't, and I think that sends a message: they've given up.

During the ARM announcement they said they would be contributing support to open source projects, and Homebrew was on the slide, along with a dozen or so other major projects (though Homebrew itself represents hundreds). I do not know what form that assistance will come in.

Apple has been dropping lots of balls in this way.They should support open source in an official way.  They should support other languages besides swift.  They should support virtualization/containers.  They should embrace techies, scientists and engineers.  But it seems they're closing down these areas of their ecosystem.

Starting to feel vindicated by my refusal to use Homebrew when I migrated to macOS recently. It felt like bringing a whole lot of baggage along.All I need is working golang and Python 3. The rest I can do on a box in AWS.

It's nice to have a setup that lets you get stuff done with a remote machine without needing a lot of parts strung together - I had this at a previous role. I've also had roles with very little support, and being able to spin up all the utilities I needed on a work computer was helpful.

Reply on news.ycombinator.com