Why Ruby App Servers Break on MacOS High Sierra
blog.phusion.nl
blog.phusion.nl
I got the gist of his summary but his writing is a bit... awkward. Is English not his first language?
As far as I know it is.
The single thread in the child calls malloc(). But the malloc mutex is already held, because in the parent process, a thread was executing there. Unfortunately, since that thread does not exist in the child, it will never be able to release the mutex. The thread in the child is deadlocked.
https://github.com/ruby/ruby/commit/8b182a7f7d798ab6539518fb...
The only correct fix I can see is to have pre- and post- fork hooks in every library, which make that guarantee.
This kind of talking up of one's own product and saying "we're first; we're best!" rather than working within the ecosystem to help fix a shared issue confirms my feeling that Passenger is a bit of an outlier in the Ruby community.
Well....I don't think I've seen many programmer-to-programmer errors that are less cryptic than the one described in the article. It's actually quite amazing how much explanation you sometimes get from Cocoa!
If I understand correctly, this problem is caused when you call initialize in another thread, and while that is running, you fork.
It affects these app servers because this is triggered at "require" (or "import") time. This is madness, no module should run code just when you import it. OK, I have been guilty of that, too. But if absolutely neccessary, keep it to a minimum. It breaks all kind of things when module imports are not side-effect-free.
Launching a thread and (indirectly) taking a mutex is definitely not something you should do on module import!
One of the rules that Apple defined is that you may not call Foundation class initializers after forking.
Maybe there is an internal `mark_exec_as_called()` function that you could exploit, if you are forking without exec...
> Exception: processes that are single-threaded when fork() is called have no restrictions on +initialize in the child. Examples: sshd and httpd.
¹ https://opensource.apple.com/source/objc4/objc4-723/runtime/...
The breakage, I mean. To clarify a bit, for better or for worse, this is what Microsoft does, totally different psychology: https://blogs.msdn.microsoft.com/oldnewthing/20031223-00/?p=...
EDIT: BTW, it is trivial to install clang without Xcode on a pristine Mac. Just typing cc in Terminal will pop up an installer that will download and install clang (and possibly other tools)
Google has also shown the door to it on their OSes.
Lovers of BSD licenses will miss FSF software in a couple of years.
specially with that username. lol. thanks for the laugh.
Not everyone contributes back to LLVM thanks license.
Just like the *BSD hardly see the same contribution from all those embedded devices, as Linux enjoys.
> Do you prefer that to the BSD-licensed solution that LLVM is, where it's still open source and actively maintained, rather than left by the wayside because Apple just refused to play nice?
I'd rather have the time, money and energy go into a free software commons. Anyone contributing to LLVM is enabling proprietary software. No thanks.
Am I clear in understanding that you're statement is "people contributing to this piece of free software are not contributing to a free software commons"?
Because that makes no sense to me, and it's hard to avoid interpreting the dismissal as pure zealotry.
Being against that, is a dogmatic approach that accepts no contributions to the cause that also help other causes because only the cause may win.
I'm not really for it, but I understand where RMS and his allies are coming from. Proprietary software makers tend to view their competitors, free or otherwise, as an existential threat, so it is "fair" for the free software folk to view them via the same lens.
Why should stable tools which has been working for decades constantly have to change their source-code to keep working on Apple's OS, a piece of software whose primary task is running software the user wants to run no less?
Apple should make sure the user's software runs on its OS, not the other way around. Stop apologizing for what is essentially the world's wealthiest corporation being lazy and putting the maintenance cost of its own OS on everyone else.
Apple has found that they don't need to put effort into api backwards compatibility because people are still freaking upgrading. It costs money to ensure backwards compatibility, so not doing it is desirable. The end user has to put pressure on Apple to do is.
From Hardware i.e. the Mac Pro (Not innovative? How about just make one workstation for Video Production that is valuable) and the Touch Bar. Software, Final Cut Pro the past 4 years and the crashes I see in my apps. Then Mac OS' move to iOS touch gestures on touch pad but I use it in dock mode drive me crazy. Mac OS and OS X thing that was just incredible is the OPTIONAL case sensitivity. If you haven't had to work with Adobe products on Mac this might not be such a big deal but ugh.
My mind blows every time I have to use MacOS for some job. Sure they have a optional case sensitive command line that works almost like a Unix (See optional case sensitive in 2017), but it really just is a mess unless your workflow is like everyone else's. No flexibility if your working with a team.
I would say the primary purpose of macos isn't to cater for everyone, it's to make using a computer a pleasurable experience for the non technical. It's about bringing computing to the masses.
If you're in a mahochist niche, good for you but you're better off on linux.
For me, macOs is pleasurable for the most part, and i'm technical enough to work around any problems I face. On windows/linux I have to do that for features that aren't even nearly advanced.
> features that aren't even nearly advanced
Windows, while not a UNIX basis (which can be an actual problem for technically-inclined users, eg developers) is vastly more advanced that macOS, especially for business users. Generally for maintream users, Windows or macOS are both intuitive after enough time spent using them; what's putting off is to switch from one to the other (eg shortcuts, etc.) Windows management (I mean the app's windows, not the OS) is notoriously bad on OSX these days compared to most other Desktop Environments.
Likewise for Linux, certainly the least user-friendly but second to none to perform advanced stuff (eg Deep Learning, or 'edgy' virtualization for instance involving GPU pass-through).
Don't get this the wrong way, macOS is OK for the masses, but on par with Windows 10 nowadays.
This is a personal opinion based on using all three OS daily, and from observing friends/family using any of them. As long as you make the right choice, there's no 'better' OS, just different pros and cons that suit each user more or less.
Most notably, I now have to troubleshoot my mother's workflow on OSX/iOS (simple stuff, mostly related to printing and sharing pictures/scans), it wasn't so a few years ago. As of 2017 I personally have a much simpler experience out of the box on Android+Windows.
a) show complete ignorance about modern OS design & technology and
b) conflating how “advanced” an OS is with you & your mother’s UI preferences.
a) I am willing to accept your statement (I am no OS expert, I have no formal CS training; I'm just a developer of rather high-level software and I've only been tinkering at home with computers for a short couple of decades); however please note that I was merely opposing the parent post's implication that macOS is the most advanced OS.
I suppose this is perhaps a matter of perspective: I define and judge "advanced" here not from a CS standpoint but rather from a real-world pragmatic standpoint: does it do the job, for whom, and how well? I observe that macOS isn't dominant in business nor in server rooms of any kind, and that Linux is pretty much the only relevant solution for most cutting-edge computing projects. Please help me understand how macOS has more "advanced features", as stated by the poster I was replying to. I sincerely fail to see what macOS has on Linux or Windows nowadays. I, for one, can't do anything better on it.
b) I see your argument as slightly derogatory, but let's move past that. Surely you understood that using anecdotal arguments, implying my mother of all users (!), had the evident purpose of downplaying my opinion to just that: an opinion, not a scientific judgment about the advancement of an OS; thereby implying that the parent post I was replying to had no more grounds than mere subjective opinions to make its statements. At least, none that I could find. There is no conflating of anything, but perhaps that was due to bad wording on my part, in which case I understand the negative reaction (but stand by my opinions, I vastly prefer Windows 10 UX to macOS as of 2017, and I should perhaps add that I was a 100% mac user from 2008 to 2016, at the notable exception of casual gaming which I've quit since then and does not even factor in my current opinion).
I'd gladly hear answers about the respective advancement of each major desktop OS because I'm truly interested in the matter, if only from a dev perspective (and obviously as a consumer/user).
Which is completely subjective and also not what “advanced” is usually used in reference to when it comes to OSes. Perhaps you meant “intuitive”?
Either way it’s subjective so you could have just distilled both your posts (and points) to this:
“Personally, I don’t like it.”
That’s fine. You do you. No harm no foul. Would have saved everyone the essays & you typing them.
Related: you... overwrite. You’re incredible verbose for the amount of data you’re delivering. That can come across as patronizing or condescending. To use a $5 word: you bloviate.
I don’t say this to belittle; it’s merely feedback & trying to help. Tone can be hard in text.
(In this case I wanted to be as formal as possible to convey respect. English isn't my mother tongue so I may tend to overdo it).
Microsoft, and Linux (just the kernel I'm talking about here), have both decided the point of an OS is to run programs, so with each update they both heroic efforts to not break userspace, often adding code just to make sure old programs that did weird things don't break.
Apple have decided to go a different route, and leave a trail of programs just a few years old that are forever unrunnable, as they won't even distribute old copies of their OSes. However, it seems many users are willing to take that choice, as we can see from their success.
They do, actually; if you bought a copy it should be available in the App Store. They even sell physical installers for old operating systems: https://www.apple.com/shop/product/MC573Z/A/mac-os-x-106-sno...
There isn't a great answer to this that satisfies everyone.
If we don't push back against bad practices by developers we never improve the quality of the ecosystem.
In this specific instance, the user has a choice - don't upgrade.
That's how you get everyone stuck on Windows XP. There's no good choice for the user here–either you're on an old OS that's missing features or security updates, or you can't run software that used to work.
I regularly read The Old New Thing and just shake my head in wonder at how much more MS could have accomplished without that baggage.
I would much rather users were occasionally forced to upgrade their buggy software.
Yes, that's the ideal solution, but backwards compatibility overall is a huge can of worms that doesn't really have a good solution. In this case, it might not be possible to upgrade the software (e.g. it's unmaintained, or was contracted out, etc.)
The OS architecture no longer suffers from such things.
It’s better now - because they moved to abstracting a lot of that stuff as you said - but the number of man hours spent on it and the knock on effects of the overall system design is non-trivial.
Plus, if they’d not had that policy they could have been where they are now in the early 2000s
Kind of like OSX was.
Ignoring even the security implications, you can't simply not upgrade either as software doesn't exist in a vacuum. All the other software/hardware involved might need newer versions or bug fixes. Microsoft doesn't even support newer Intel chipsets in versions of Windows older than 10 -- that's only even possible because 10 is highly compatible with older versions of their OS.
However, a user who just upgraded Windows and half their programs stop working will simply not upgrade. In many cases such dodgy applications are also never fixed because the vendor doesn't exist anymore, doesn't care, or the flaw is in some library they use and they don't really want to invest that much time to change it.
Besides, detecting such behaviour is pretty much the same as fixing it with a compat shim, since you have to do more or less the same work. So from MS's perspective the benefit of working around the buggy behaviour is much more prominent than trying to discipline developers by making users suffer.
char *gets(char *at);
x = gets(sbrk(0));brk(x + strlen(x) + 1);
char buf[99];
x = gets(buf);
The API has a choice:
a) They can make it harder to do the wrong thing; make `gets` generate errors, warnings, crash if you use it, etc. This requires people fix their programs. That's what GNU did for `gets` and it's what Apple is doing here.
b) They can change the rules: It's possible to modify the ABI and require markers or a mechanism to detect where the edge of the buffer is. This requires people recompile their programs. I think Zeta-C had "fat pointers" so gets should be safe there[1]
c) They can work around: If you have enough resources, you can look at the programs using this API and figure out what their buffer sizes are and hardcode that into the API provider. Microsoft has famously done this for SimCity[2]
That's it. They can't really do anything else: The function is difficult to use right and programmers do the easiest most-obvious thing they can. oh i need to get a string, so i'll use gets... but i need two strings so....
Anyway, I'm not aware of any good rules for choosing which way to go: Less total effort seems like a good metric, but this is very hard to estimate when you live on an island and don't know how other people use your software.
Memory corruption is serious though: It's often very easy to turn into a security hole, so I generally advocate doing something. All of the people just disabling this security feature make me nervous. I wonder how many of them run websites that I use...
Just as we often grow the stack with page faults, you could grow the heap with page faults: Modern UNIX doesn't because this is another one of those "almost certainly a mistake", but:
signal(SIGSEGV, grow);
void grow(int _) { sbrk(PAGESZ); }
No chances I make it to the top of MS though.
So the obvious question is whether it's fundamental that with kernel threads the fork() system call doesn't clone all the other threads in the process? Yes, that's not how it's done, but could Apple choose to implement a new fork_all() system call? I imagine it wouldn't be easy - you'd need to pause all the running threads while you copied state, but is there a reason it's actually not possible?
Edit: simply signalling all threads with SIGFORK, waiting for a handler to return and freezing a thread at SA_RESTART point feels right way to do it. Why did POSIX decide to fork() only one thread by default at all?
If you're just going fork() && exec() then why would you copy all that state just to run some subprogram?
Is that what you want any of the time?
This prefork implementation is silly. Either do prefork after initialisation so you can take the benefits of COW, or don't bother: Just use SO_REUSEPORT and run 10 copies of your server. This distributes the TCP traffic and provides you an excellent way to upgrade-in-place (just roll up the new version as you roll-down old ones).
Today though, everyone loves threads. It's hard to know precisely what happens in some low-level library, so it's hard to be sure that you don't have any threads left running after initialization. This is the subject of the OP.
My question though is broader - if we chose, could we add a version of fork() that did clone all the threads? I'm not entirely sure what it would be used for, but I'm sure there would be uses. Likely some of those would be for increased security, as processes provide stronger isolation.
I suspect strongly there aren't.
One of the biggest problems you have to contend with is around mutexes (and the like) and their state. If you copy them, we double-book an external-resource (like storage or something), and if you don't we almost certainly deadlock.
Another is any thread waiting on a resource (like reading a file descriptor) or writing to the disk (do you write twice?) and so on.
~ man fork
Here are a plethora of examples from the Windows 95 days: http://ptgmedia.pearsoncmg.com/images/9780321440303/samplech...
You could imagine an implementation of the Ruby interpreter which kept all the expensive and shareable state (bytcode and class metadata, i suppose) in a shared memory segment, and used spawning to create worker processes with their own process heaps, but sharing that segment.
It would be a lot of work to implement. It would probably be easier just to get Ruby to use threads properly.
There's a bit more to it than that, as programs can run code and load data prefork. Pretty important for things like large config files.
Maybe this? Though the resolution was to disable uwsgi proxying globally... https://stackoverflow.com/questions/35650520/uwsgi-segmentat...
The same happens on macOS, it's just that Apple added a contract that you can't initialize ObjC classes after forking.
Apple is probably tired of bug reports to their software that really stem from some other place.
Apple still sucks BTW. Heavy handed nonsense. Let developers deal with the consequences of their actions.
I've never felt the need to install docker on my local dev environment. It's great for production, and I'm sure it's great for people who can afford the disk space. But when space became tight, Docker was the first to get the axe. I haven't missed it yet.
But my question is: is this really that important? I mostly use macOS for development, don't feel that the preforking model has that impact in the development cycle.