I'm not justifying Apple's ever increasing shift towards enacting strict constraints on their platform developers, but the comparison with Linux is pointless.
Any specifics, or are you just trolling?
The company I work for develops high-end desktop software for Linux, Mac and Windows, and Linux generally causes us the least trouble and OS X (thanks mostly to atrocious graphics drivers and OpenGL support in addition to really bad memory allocation/paging issues that have only just been fixed in Mountain Lion) the most trouble. And this is on limited numbers of hardware and software variations for OS X.
But we've got customers running everything from RHEL 4.5 to the latest and greatest Ubuntus, Mints and Arch.
All with the same binary installers.
The only linux-related weirdness I can recall in the last two years distro-wise was Scientific Linux, which seemed to have some weird XOrg config issues.
So you integrate with the user's desktop environment, using standard local widgets, theming, integration with UX guidelines, local library dependencies, etc?
Or are you shipping a Qt app with your own dependencies included? If so, Qt is pretty notoriously buggy on OS X (and rightfully disliked by most users as it stands outside all platform conventions), perhaps explaining some of your complaints.
I don't believe there are general UX guidelines for Linuxes, but for things like notification popups, system tray stuff, yeah, we do all that natively to the distro through Qt (DBus handles all that transparently within Qt very nicely).
We ship Qt as shared libs with the binaries and all dependencies we need system-wise statically - i.e. zlib, libpng, libjpeg are statically linked. For things like embedded Python, we have to do the same on OS X anyway, due to different python and zlib versions per OS X version.
As for Qt and OS X regarding nativeness - I keep hearing this, but I never see any good examples. I agree it's possible to create Qt Apps on OS X that looks crap, but it's also possible to make them look native as far as I'm concerned. The only thing I can remember not being able to easily do in Qt regarding OS X widgets is the horizontal grouping of side-by-side buttons in radio-button fashion. But it's easy enough to knock up a QWidget subclass which replicates this. But admittedly I don't think I've tried to replicate every possible OS X control/widget...
I think what might be the most difficult part of making a Qt app on OS X look native is the layout and spacing stuff, which generally does seem to be a bit crap within Qt on OS X.
I can have the same code to do a side-panel with edit values in a panel, and it looks great on Linux and Windows: http://imgur.com/kUI90ry
on OS X: http://imgur.com/yqigBdw
and the spacing and padding is all over the place - so I have to add a manual style-sheet to get the padding and spacing right, which is crap (image above is without extra stylesheet). So I'm not saying it's perfect or easy, but I think it is possible.
Does QT basically render a bitmap, or does it use the native controls and just has some odd default styling? If it's the first, then I would probably do the UI in native Objective C/Cocoa and keep a nice cross platform base for the VFX core of the app. Not sure if that's feasible for your application of course.
http://imgur.com/68b9eoO - is native Cocoa moc-up in XCode.
I concede the borders are slightly different, but I personally wouldn't have noticed or minded.
The render button is a custom button - the only difference from the native one is the right-hand menu triangle which isn't native.
Qt has styles, and draws controls itself via vector drawing. And you can customise this yourself: So you've got all the power in the world to do what you want, but at the expense of loads of complexity that no-one's really going to go to the trouble of doing. So basically, it tries to emulate the controls on all platforms, and does a pretty poor job on OS X.
I would side with the reader below, I think that going for a completely different look works better than 'almost' native. Ableton Live is a good example I think that has (almost) the same UI on Mac as on Windows which doesn't look bad on either OS.
I am thinking of Pixelmator, or Apple's Final Cut Pro and Motion for comparison to your screenshot. Even modern versions of Photoshop manage to look much nicer than that.
Qt "emulates" native control appearances by drawing them itself. This means it doesn't get it 100% correct (in OS X's case). You can apply stylesheets to configure all aspects of spacing and appearance, so in theory it's possible to write a stylesheet to completely emulate OS X's controls (or at least the majority of them), but that'd be a lot of work which you probably wouldn't want to do.
I could make it any colour I wanted with stylesheets in Qt - this was a personal app, and I'm not too concerned with what it looks like.
on 10.6 (and 10.7), memory allocation/paging is pretty atrocious if you haven't got much free mem left.
Basically, OS X prioritises paging to disk over freeing up Inactive memory (which isn't actually being used for anything at present), which is pretty crap. So it's very easy to make it page when it shouldn't when you allocate loads of memory, and the system will generally just grind to a halt.
Apple fixed this in 10.8, so now it will free up Inactive memory first, before it starts paging.
But neither of these explain any crashes within Nuke.
Google may develop a lot of things behind closed doors but let's give them credit where credit's due.
No wonder you're having issues on OS X.
There are many software categories where visual-design decisions are not the only competition advantage.
Different UI toolkits make doing certain things hard, certain things easy, and some things damn near impossible without large effort on the part of the developer.
So when the parent commenter says "Nobody on OS X wants a Qt app", I'd argue that yes, although you can make a "well designed QT app", it's much harder to make a "well designed OS X app" with QT.
Why will it be less likely to be a well designed OS X app? Because it will be different to a Cocoa app made with interface builder, in subtle and sometimes not so subtle ways. Sure you can code around these and make adjustments, but the level of effort and investment to get to the stage of if you'd just built it with a more suited tool for the job (Cocoa / Interface Builder on OS X) is quite high.
This is why there is some truth the the admittedly generalised statement made above that "Nobody on OS X wants a Qt app".
There aren't any competitors, however, so people use Mathematica.
I use high-end expensive that are Qt-only too, but I would gladly switch if there were other options, because Qt is buggy, uses non-intuitive non-standard UI, exhibits poor performance due to impedance mismatches between the Mac and Qt event/threading models, and ultimately decreases overall utility of the UX.
This may well be true, but it's quite possible it's just the developer was lazy and didn't do all the effort to make it look native.
Edit: in fact, I'll give an example: VirtualBox - it does look crap on OS X, but that's because the developers obviously didn't make any effort to make it look native. They've got icons in tabs pane buttons and huge toolbar buttons along the top (which isn't a unified toolbar btw - which would make it look more native), which goes completely against the way OS X apps tend to look.
I've literally never been fooled by a Qt app -- a Mac user can spot that hot mess from a mile away. It's not just a matter of visual layout and UX, although that matters and always falls somewhere between subtly or completely wrong.
The controls also tend to lag at weird times, behave slightly strangely, demonstrate unusual font metrics, etc. The apps tend to exhibit bugs related to menu bar event handling, window management, on and on, and crash more than any other apps I use daily.
This isn't a huge surprise given that Cocoa is designed to be the first and only gatekeeper between events, the OS, UI controls, and Cocoa's main event loop, and most of what makes Mac apps a Mac app has to do with the APIs and integration that you lose first-class access to once you deploy Qt -- that includes 'simple' things like UI layout metrics, which Qt has to re-implement.
I abandoned almost all those Qt apps on my list above. I even dropped EAGLE, despite being "native", in favor of Altera on Windows, which I paid a hefty premium for in no small part because it wasn't a pile of broken UX, Windows or not.
I still use IDA Pro, but the minute http://www.hopperapp.com/ can replace my IDA Pro usage, I'll switch in a heartbeat.
Although having looked at hopper, I must say I'm not convinced it seems any more native than a Qt app on OS X could look if work was concentrated with this goal in mind. So as I said in my VirtualBox comment, I think that while indeed it is a Qt deficiency that apps don't look native straight from the compiler, more work could be done by the developer to make it more native-like.
I've seen controls acting weirdly but only QPushButtons - that seems to be regarding the hit-test area - we've had to sub-class a lot of them on OS X to fix this by making the hit test area bigger which is annoying. There's also a pretty bad spacing/padding/layout issue as I alluded to in another comment.
I agree with the event stuff as well - definitely one of the biggest things which affects us is that sometimes mouse/keyboard events don't get sent to the app if the app isn't active or the mouse isn't over a particular window. Debugging through Qt, it seems that Qt never gets the event from the OS, so we have to install event filters for all windows and intercept them which is crap. We've also seen weird stuff like key release events being received before key down events - again debugging through Qt, it looks like an OS X issue in that that's the order they come in from the OS, but again, without a native version to compare to, it's difficult to tell or test.
Generally the crashing of our apps is due to graphics drivers or memory allocation issues (we deal with huge amounts of memory), and I'm not aware of Qt itself being the result of any crashes on OS X for our apps any more than other platforms, but I guess without a native version to compare to, it's difficult to say - but Qt is very rarely the cause of crashes in my experience and I generally develop for Linux and OS X.
It's heavily Qt-based, and doesn't really follow any Apple design guidelines. But it's the best 3D texture painting app in the world for high-end work.
And if you're really concerned, you can just use the JVM.