MIT Scheme on Apple Silicon
kennethfriedman.org
kennethfriedman.org
I am SO HAPPY to see this work! Major timesaver for me and anyone looking to run the executable versions of Functional Differential Geometry[1] and Structure and Interpretation of Classical Mechanics[2] in the original language.
[0] https://github.com/sicmutils/sicmutils [1] https://github.com/sicmutils/fdg-book [2] https://github.com/sicmutils/sicm-book
The Clojure port is quite fast, faster than the original for all benchmarks GJS has sent me, and more fleshed out. (That will change, as I've been pushing bugfixes and performance improvements back upstream as I go, as a meager gift to GJS for making this huge, amazing library in the first place.)
I actually wrote to GJS this morning asking for instructions on how to compile the original "scmutils", since I have the same problem. He responded saying he'll get back to me this afternoon, so I'll post here once I have details.
If you are still interested in getting the books going with MIT-Scheme, I put a decent amount of work into the exercises using the original codebase here[2], including a dockerized version of mit-scheme[3] and the scmutils package[4] that might be useful.
- [0] https://github.com/sicmutils/sicmutils/tree/main/test/sicmut...
- [1] https://nextjournal.com/try/samritchie/sicmutils/
- [2] https://github.com/sicmutils/sicm-exercises
The vfork/fork issue and the compiler upgrade issue don't seem to be too problematic to work around, so there must be some kind of ARM limitation that's preventing Scheme from working, but what?
There is an escape hatch for writing JIT compilers (essentially what MIT Scheme is in this case), described here https://developer.apple.com/documentation/apple-silicon/port... although it's fairly cumbersome and would almost certainly require a lot of extra, MacOS specific code. I assume that's why no-one has bothered so far to port it.
I know at least OpenBSD also enforces W^X protection universally, anyone else? I know Linux can with the right SELinux policies, but not sure any distro ships with those by default.
There's a per program exception list to handle legacy programs though.
macOS W^X on Apple Silicon, however bans RWX memory outright, making it impossible to have a page in memory that is simultaneously writable and executable. Instead, if you want to be able to write instructions to a page and later execute them (e.g. for JIT compilation), you have to (1) have a special entitlement (or opt out of the Hardened Runtime), (2) map your memory with a special MAP_JIT flag, and (3) call special mprotect-like functions to toggle the protection between RW and RX every time you want to modify the code.
There does, however, seem to be a bit of a loophole: the JIT protection flags are applied per thread meaning that in principle one thread could have the page RW while another has it RX.
Still isn’t, because the arm64e ABI isn’t stable. As such, any binaries not bundled with the OS, including Apple applications, use the arm64 ABI without pointer authentication.
You can use -arm64e_preview_abi as a boot argument to enable arm64e support for non-OS bundled processes.
Note that however the arm64e binaries that you compile might not work on future macOS releases.
One really fun way to hone your skills is https://microcorruption.com/, a ctf-style simulated hacking game originally made by Square and Matasano.
I suppose the wait is on for someone to rewrite the JIT engine to be compatible with Apple's implementation of ARM.
- Compiled code needs to be allocated separately from Scheme objects. It can still be garbage collected and such - they will probably need to make a separate set of allocation functions for code vs. data. The closure/function objects can be made to point to the code, or, if they don’t need to be written often, simply allocated wholly from the “code” pages. - Before modifying any of the code (e.g. to patch addresses after GC relocation), a system-specific hook function will need to be called to set the permissions to RW. They already call an I-cache flush function after each modification, so this shouldn’t be too bad.
Some of the necessary changes are already sketched out in cmpint.txt. And, sooner or later, they’re going to have to make these changes: OpenBSD already enforces W^X (but provides a workaround), and MIT/GNU Scheme already applies a paxctl workaround to gain W|X on NetBSD.
The main MIT Scheme page say that it's not possible and need significant efforts, so I would be curious to get a description as to why one claim it's impossible while the other show that it compiles and starts.
Are the original authors too much against M1/Apple and justify themselves ? Or does compiling on M1 sort-of works until you hit more complex features that will crash or misbehave ?
Where is this coming from? What about the literal thousands and thousands of programs on your OS right now? Or the thousands of systems that power large corporations that predate JavaScript popularity?
Sure the language is popular right now, but software development has a much longer history than the last 10 years
For Scala, there is, separately, Scala Native. [2]
Both have ahead-of-time (AOT) compilers that compile down to the target architecture.
[1] https://www.graalvm.org/reference-manual/native-image/ [2] https://github.com/scala-native/scala-native
Yes [1].
If you ask any random programmer if "Java runs on x86", 99% of them will say either "yes" or "I don't know what x86 is". Similarly, if you ask them "does Kotlin run on the SPARC architecture", they'll say "I don't know" and, if you give them some time to find [1], they'll amend to "no".
To be precise: the meaning being used by most programmers (and here) is "either the compiled binaries, the virtual machine, or the interpreter runs directly on the given architecture" - which clearly excludes MIT Scheme running on Rosetta, just as (to take a less controversial example) the fact that might be able to run the JVM on qemu on SPARC doesn't mean that the JVM runs on SPARC.
Thinking about the various levels of abstraction of VM's and interpreters is a fun exercise in general, but I don't think it's constructive in this particular situation.
There's a special aarch64 build of the software available, so it clearly runs on ARM. Perhaps there's some kind of issue specifically on macOS that makes the existing ARM port incompatible with Apple's ARM implementation?
I can’t imagine what you mean by this, Rosetta 2 is a binary translation system implemented in software, based on QuickTransit. There are a few features implemented in Apple Silicon to make translation easier and more efficient, such as supporting Intel memory ordering, but thats about it.
I think it’s reasonable to worry about how long rosetta2 will be available. The first version, that allowed Intel Macs to run PowerPC binaries, was available for 5 years. Having said that, there’s no guarantee versions of MacOS beyond 5 years time will run on today’s M1 anyway (though M1 compatible versions will likely still get updates beyond then).
Even the company that made QuickTransit is gone now, having been bought out by IBM a decade ago.
The same backend is used for AoT and JIT compilation. It uses a custom lightweight IR that’s really close to x86 itself, with a big focus towards reducing translation times.
Rosetta 2/Cambria was fully written in-house.
For starters, the Mac became a lot more popular in the Intel era than it ever was while on PPC, so there's a much larger quantity of legacy software that Apple would be cutting off. Secondly, the overall user experience of running apps via Rosetta 2 seems to be a lot better than Rosetta 1. And for Apple, Rosetta 2 was developed in-house and doesn't require continuous licensing fees to keep around (not that I'm particularly sympathetic to Apple's pocketbook.)
I don't think any of those things matter; Apple will stop supporting Rosetta 2 as quickly as they can. They announced the transition to Apple Silicon will be two years and unless something unforeseen happens, that's what it's going to be.
I suspect that Rosetta 2 won't be available for new Apple Silicon Macs running macOS five years from now.
Of course, no matter how many years in advance Apple warns that a particular technology is going to be deprecated, that never stops people from complaining vociferously when it happens.
A great example is 32-bit apps, where Apple gave something like an 8-year heads-up that 32-bit apps were going away, which happened a few years ago but it's not hard to find threads on HN where people are still complaining about it.
But actually, I personally believe that the actual reason Apple killed 32bit support was because they didn't want to build it into Rosetta. (And they didn't want Intel computers to be able to run anything their new Apple Silicon computers could not.)
Before Apple Silicon was on the horizon, it was no problem for Apple to keep 32 bit and carbon libraries around for eight years because they might as well, it's not doing any harm.
(I'm also one of the people who was/is mad about 32 bit support, but I acknowledge that my opinion on the matter has no bearing on what Apple will decide to do.)
I doubt it; that’s not how Apple rolls. They’re not like Microsoft which keeps legacy technologies around for backwards compatibility for several years after a technology is no longer mainstream.
Reasonable people can disagree but Apple is about the present and the future, not the past. Sure, they could have kept Carbon around or pick your favorite framework from the past but that’s generally not their thing.
Occasionally something from their past reappears, like the QuickDraw GX font format from the 90s that became the basis for today’s variable fonts on the web.
Apple has always been fine with some software not making the leap to the next operating system or processor architecture.
We’ve seen this going back to 68K to PowerPC then to Intel and now ARM.
Being able to run x86 operating systems (Windows) natively on Intel Macs was a huge selling point not that long ago and now it’s an afterthought that current buyers (mostly) don’t care about. Microsoft would bring Windows for ARM to Apple Silicon and so far, they haven’t.
And while this is all going on, Macs have never been more popular.
Microsoft want to do that but it will be a huge risk since it will alienate their enterprise consumers.
Rosetta 2 contains functionality to correctly emulate 32-bit Intel code.
32-bit isn’t completely gone, it’s still on watchOS.
I also remember it taking a while to get Gerbil Scheme running on M1.