Mail Data Loss in macOS 10.15
mjtsai.com
mjtsai.com
And, just to add, I am tired of OS updates updating and/or releasing userspace things like "Music" and saying:
> iTunes forever changed the way people experienced music, movies, TV shows, and podcasts. It all changes again with three all-new, dedicated apps — Apple Music, Apple TV, and Apple Podcasts — each designed from the ground up to be the best way to enjoy entertainment on your Mac. [0]
It is literally the FIRST thing they advertise. What does that have to do with the OS? Nothing! They should release apps separately from major OS versions.
Right? Are there not automated tests for this stuff? I would expect that if you commit something that causes this, the next day you get an email that says, "your name is on commit $FOO, which caused regression tests to fail in tests/firstparty/apps/mail; please fix it (sincerely, Jenkins)" or so. Heck, it's more egregious here because it's a first party application, but it seems like it would be in their interest to keep copies of the most popular third-party applications and include those in their test suites as well. ("Hi, Adobe? Hey, it's Apple. Hey, so this is a heads up that Photoshop doesn't work with the new OS version; we're sending you a RCA so you can fix it before it hits end users.")
That's also why I think other automated tools and "code quality" metrics don't help so much; tests are more of a sanity/last-chance check and shouldn't be relied upon to catch everything. Correctness should be assured through design and code reviews, and for that nothing beats the power of intense mental deliberation. To paraphrase a popular saying, "there is no replacement for intelligence."
What they need to do when it's potentially about permanent loss of user data is to prove that a postcondition is met (i.e. all messages were successfully converted) and roll back if it isn't.
Instead they apparently sync the faulty data with the IMAP server and destroy what most users consider their email backup. This is breathtaking incompetence and negligence if it's actually true.
The worst thing is that it is near impossible to find out whether or not you're affected by this.
I'm so fed up with Apple and its second rate software. I have lost music playlists in this ill fated Catalina "upgrade" as well. I've been wanting to move on to a more flexible and less oppressive environment for a while. At least this finally breaks my inertia.
Try Linux or Windows! Their first rate apps will keep your email safe and secure!
https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/372...
https://forums.linuxmint.com/viewtopic.php?t=88648
https://blog.lightlyseared.online/2010/04/06/fixing-corrupt-...
https://windowsreport.com/outlook-emails-disappeared/
https://thenextweb.com/microsoft/2013/06/20/outlook-coms-mis...
https://www.pcworld.com/article/237103/five_outlook_nightmar...
Do you really believe they aren't running automated tests at Apple?
It's more likely than not, that this was not caught by one of the tests...
In that many people consider the Linux kernel and basic userland as the "OS", and the applications, packaging, etc as the "distro".
Even casually, when we mention OSes, we say "Windows, macOS, Linux", not "Windows, macOS, Ubuntu, Debian, Centos, etc etc...".
I had some apps break and need update after an iOS update, but not layout problems in apps... at least not in the 50+ non-Apple apps I use...
More annoyingly, it seems the changes to how multiple windows are handled has made it so any visible notification will prevent me from interacting with anything except the notification, so I can't ignore them anymore - I have to dismiss them to carry on using the phone.
It's doesn't "suddenly" cause the existing Mail app to break.
It's the new version of the Mail app that breaks -- for reasons not having to do with the rest of the OS.
macOS, as an OS, is not just the kernel, SDK, and GUI libs, and tooling: it's rather a distro that includes regular apps like Mail, Safari, Notes, etc.
If it was OS caused, it would affect others apps too.
It seems that by locking their userland application release cadences to that of the rest of the distribution they lower the number of unique configurations of userland applications and OS components they have to support in the wild. I’d imagine this has implications with regards to the cost of support.
- Peaked in terms of brand perception rather than market cap or market share.
Not an apologist, but showstopper bugs were a thing even under Jobs.
Jony Ive made a good effort to keep pushing hardware design forward, but without Jobs to curb his worst impulses (in my opinion), the end was result was "thinner above all else" (as you mention).
If someone can come up with a corporate structure that allows for the dynamism provided by having a visionary dictator at the helm, without actually needing a visionary dictator, that would be a corporate structure that would take the world by storm.
Almost antonyms, which is perhaps why it's so hard to do without that visionary leader. A philosopher king if you will. Or benevolent dictator for life.
Oh stop. Apple's effectiveness was entirely a product of its time. You had designers building the Zune and calling it competitive but really there was no competition for the products Apple was building for a market that was screaming for a better balance of form vs function.
This picture of Apple's innovation includes no part of OSX, which was always an OS that was technically capable but superbly messy and extremely behind on the times.
Innovate effectively? For a long time of Jobs' tenure, Cocoa was being supported on BOTH Objective-C and Java, and when they ripped it out, they actually chose Objective-C, not Java, as their platform of choice (in retrospect: wow), only to have to build yet another platform (Swift) just a few years later (Swift of course, based off of MacRuby, which they initially tried to build Cocoa on top of internally, so I hear).
Innovate maybe, but effectively no. There is a whole host of broken, abandoned, and outright bad decisionmaking in the OS layer at Apple. Apple's success has everything to do with their success with industrial design, UX design & marketing, and just a bit of being at the right place at the right time.
IMO this interpretation fits the timeline much more cleanly: Apple was churning out the same iPhones and Macbooks long before Jobs left, just as they had been before with the iPod. The butterfly switches, the thin above all else, that's all part of Apple's MO dating back years. Lack of FM radio on the iPods, lack of IR sensors on their early phones, removal of removable batteries, headphone jacks, USB-A, etc, is all part of the DNA. A $5000 screen is not surprising to anyone who saw the Powermacs of the last generation. Butterfly switches look an awful lot like bendy iPhones, which look an awful lot like DOA Powerbooks.
I'm really not sure what part of this is new post-Jobs, but I have a good feeling this is a great case study of confirmation bias. If you believe Jobs was a one-of-a-kind irreplaceable visionary who was single handedly responsible for Apple's success, then you have no choice but to interpret any action Apple makes post-visionary as a failure, otherwise you were wrong about the one-of-a-kind irreplaceable visionary.
And it would be a pretty big blow to most to be wrong about Steve Jobs.
The other interpretation, of course, is that Jobs was not a magical visionary, just a smart guy who made a couple of right decisions, got lucky on lots of others, while still getting it wrong on plenty of occasionss (Macintosh TV cough Apple TV cough Apple TV2), just like most fallible humans.
When they saw the community had no issue embracing Objective-C that is when they dropped the Java bridge, QT for Java and eventually their own JVM.
Chris Lattner never speaks about MacRuby on his interviews, rather how, like clang before it, Swift started as a side project before being shown to upper management.
According to his interviews, many of the Objective-C 2.0 and later improvements were already a kind of slow roadmap into Swift.
Similarly 2008-2012 or so AFAIK, maybe I'm mis-informed, Macoobk Pro's were top of the line notbooks, no compomises. I've from now through then and am typing on one now. But, now they aren't the top. Low powered GPU means all my graphics friends have had to switch. Some of them had previously bought Macbook Pros and installed Windows but they can't even do that now as the GPUs just aren't there. eGPUS don't count as they aren't portable by any reasonable defintion of portable. I want Apple to reclaim the throne here too. What's is there to downvote about that?
QA at most so-called "modern" companies these days has turned into overly fetishising automated testing. Automation is valuable, but crucially also leads to situations where the least-common-denominator situations get tested since it's easier to automate the conditions for testing them. So for example you might write an automated test that verifies that the mail index is correctly upgraded from (say) El Capitan to Mojave to Catalina; but that doesn't show the whole picture. There could be 20 other complex factors that affect it when it's used as part of an OS upgrade, and none of those get tested.
Project managers, line managers, and other suits also seem to love automated testing, because you can show off some kind of real number to someone in an "Oh we wrote 200 tests this cycle, isn't it great that we can run them everyday" way, even if there are 0 issues found. Manual testing in the same case leaves you with no output that can be run in an automated fashion. Also, no one wants to argue against automation (even if it's taking away manual testing resources) because that can make you seem old-school and out of touch. And once this attitude gains steam, no one on the QA team will want to go against the grain and say "You know what, we should spend more time not writing automated tests".
All of this is to say -- sometimes during testing, you simply have to put yourself in a user's place (for as many different kinds of user you can imagine) and actually use the thing you're supposed to be testing. I strongly suspect that Apple is not investing enough, or at all in this kind of testing. Instead, they are pushing it off to those enthusiasts who install and use beta or newly released versions of their software. So far, this strategy seems to have paid off for them, but perhaps people will wise up and stop testing their stuff for free. I am certainly not upgrading my machines to Catalina any time soon.
Such as load testing, end-to-end testing, fuzzing. All of which can be automated to a large degree.
As someone who is "old-school" and has argued against automation countless times, it's more that your arguments against it will likely be dismissed because the management just wants to see the numbers on their pretty "quality metrics" dashboards going up; regardless of what it means for the actual quality of the product. They'll even ask you to "fix" automated tests by forcing them to pass, if they are failing for some reason you don't know yet. Truly a case of "you get what you measure". It's only when you start showing them the bugs that customers come back with and why they couldn't possibly be discovered in automated testing (at least without an impossibly large and nearly-infinite set of test cases on a similarly large set of hardware and software) that they may start to listen to you. Sometimes.
Yeah, I think so.
Apple's apps have always been terrible, and it seems weird because their Mac OS (though of course buggy in absolute terms) has, for almost 20 years, far outshined any other desktop computing OS of similar complexity (meaning, basically, Windows and Ubuntu and equivalent Linux-plus-all-basic-apps type distros).
I think this is a side effect of their silo culture. App teams are tiny, often one person, and they work in permanent stealth mode. Then they finally release a developer preview, which includes the core OS plus like a hundred app releases.
Suddenly, they have thousands of beta testers: Mac developers paying Apple for the privilege. I've been one for the past 20 years or so. But we don't have time to file bugs (especially through their shitty, rude Radar system, although it's recently finally gotten... um, marginally less shitty). We spend our bug-filing time on the OS deficiencies that are fucking up our own applications.
So the OS itself gets stress tested by thousands of developers, whereas none of us really gives a shit about the app collection that will ship alongside the OS.
E.g., relevant to this issue: thousands and thousands of developers install every OS release and beat on it, but almost none of us would expose our actual email accounts to some developer preview release of Mail.app. We'd just use a different machine, or use webmail for a while while working with the new OS.
So the apps themselves actually get released with pretty minimal real user testing.
I lost 400GB of data when the installer repartitioned the hard drive and the files that were in folders in root went bye bye.
I know I'm not supposed to put stuff there, but it was a family video project I was putting together for my wife for Christmas, and she would have seen it anywhere else.
There should have at least been a notice or a warning or some kind of confirmation. But there was nothing.
Nothing in the "Relocated" folder, either.
Microsoft suspended an update after reports of data loss, and was still severely chatised for doing it (explicitly deleting files it did not create without asking permission first): https://news.ycombinator.com/item?id=18189139
I'm a long-time Windows user, have put files in the roots of drives (maybe this is not such a common thing for Mac users to do, but nonetheless the OS should not assume...), and have never had any OS upgrade remove them. It would be interesting to see what, if anything, Apple will do in response.
1. you know not to put stuff in root. you must have disabled SIP to do this? WHY?
2. ever heard of backups? it is SO easy in mac os.
Giving you the other side of this: Podcasts was essentially a disk-space-leak made app, if you subscribed to any podcasts that you never actually listened to (but still listened to others, and therefore regularly opened the app.) It would just pile up new unplayed episodes forever, eventually taking up tens of GBs of space, on a device that doesn't have all that much space to spare.
At least with Windows, Microsoft will never intentionally delete or cause user data to disappear. Bugs have happened (the one where files in the original Documents folder after relocating to a new location would be deleted was pretty bad), but it usually results in an apology, delaying further rollout of the patch and a fix for the issue. Instead of pretending "it's best for our users, they never should've put stuff there in the first place".
Option 2: BURN IT ALL.
Guess which one I don't want my apps to ever do.
AKA exactly the desired behavior.
For example, Apple themselves have a stack of playlists for different genres that are updated on a weekly basis, so after a few months you end up with a ton of downloaded files for songs that are no longer in your library (because they've been rotated out of the playlists).
Problem is, because they're not in your library there's nothing surfaced in the UI, so there's actually no way to know that it doesn't clean up after itself until you go to investigate why you have no disk space.
No idea if this behaviour has survived the transition to Music.app, but considering it seems to just be iTunes with a fresh coat of paint, it wouldn't surprise me.
Apple seems to focus too much on reinventing things, as if every app team gets a new (and inexperienced) set of developers each year and they see whatever is there as bad, incomprehensible code...so they write things from scratch, introducing tons of bugs and removing backward compatibility. Look at its app history and you’ll see a lot of work just thrown away every few years, only to be replaced with crippled applications that would take long to mature but aren’t allowed to.
Mozilla Thunderbird is “boring software”, and hasn’t made huge strides in UX in the last several years, but I trust it more than I would ever trust Apple Mail.
If I want “office software”, I have LibreOffice, which has its own issues in UX, but I can trust that it won’t drop backward compatibility at the drop of a hat.
If I want a browser, there’s Firefox. Safari 13 has become just ok now (from decent) because of the new extensions system that has junked all old extensions. It’s fine for firing up an ephemeral window, and it still provides the best battery life.
I feel the days of power users on the platform are numbered. I doubt if AppleScript and Automator will survive and be around in the next two years.
Is there any Apple app on Mac that’s been great and really improved in a stable manner?
To be fair Firefox did that recently with their transition to WebExtensions too, and Chrome will soon do something similar when they move to Manifest v3. Sadly all mainstream browsers have become "move fast and break stuff", mainly in their desperate bid to keep pace with the de facto standard that is Chrome. But most of them don't have browser dev teams the size of Chrome's...
While I get the privacy benefits with block lists that have been/are being/will be implemented in these browsers, the overall extensions scene doesn’t look very good. Hopefully Firefox will delay or not go with everything that Manifest V3 proposes, whenever that becomes standard in Chrome.
If Apple made shit software this story would be background noise, but it is surprising to see a company that usually delivers relatively polished experiences do this.
Android, Linux, Windows, macOS and iOS are all broken, just in different ways. This is the quality that can be reached for mass-market consumer products with the standard development processes.
Then I compared iPhone and android the ux and form innovations start with Apple and are borrowed by android. Same for OS X and windows, specially the gestures and form it’s flowing from Apple.
Also, I guess, they produce good design. Their products look good, that's true. Too bad, that I don't care about looks at all.
I personally think it’s due to Apple engineers trying to do so much; I remember being impressed by the number of new features and changes introduced this year while watching the WWDC, but it looks like Apple couldn’t keep up its own enthusiasm.
Project Catalyst was poorly executed; I was disappointed that Catalyst doesn’t default to ‘HIG-respecting’ AppKit controls that correspond well with the UIKit equivalents (so that modal form sheets in UIKit are translated into floating windows, e.g.), and the inability to mix in AppKit controls and UIKit ones into the same view hierarchy (NSColorWell or NSPopUpButton doesn’t have UIKit equivalents), e.g.
SwiftUI is great in theory, but the tooling is immature, it’s not available to write any decently complex apps with them.
The iCloud changes were reverted right before Catalina’s release; looks like the iCloud team didn’t have enough manpower to fix all the bugs until the release.
It’s concerning, as one of the premise of macOS was that it works as expected, though that now the newly introduced parts doesn’t. I’m expecting a stabilizing release for macOS 10.16 akin to the stone-stable 10.14 Mojave, but I don’t remember 10.13 High Sierra to be unstable like this one.
[0]: For Linux users who don’t agree with this: I’ve never set up a Linux desktop system with all the hardware working without doing any ‘geek’ stuff like running some shell commands. And no, the fact that Apple only allows to use macOS only on its own computers isn’t an excuse. And no, even on well supported laptops like the ThinkPad, there are a furious amount of bugs that need ‘configuring’
I think the prior perceived quality of Apple software was propped up by a lot of people internally grinding very hard on a far smaller set of products than the company ships today. That might work when you are at 10k people, not 120k.
It’s really time for Apple to make a good faith effort to improve its software development process. Honestly it seems more like an executive leadership issue, incessant and hasty product changes come at the cost of stability.
My Mac is, and has been for many years, a 100% work machine. 'Leisure' computing for me is done on my phone or iPad, and I think the same likely applies for a lot of Mac users.
The yearly cycle mostly makes sense for iOS because they want flashy new features to go with their flashy new hardware, but the Mac has basically no schedule for hardware updates. Even when they happen, they're usually spec bumps and not the sort of update that requires new user-facing OS features.
Of course I want the Mac and macOS to continue to steadily improve. I love the platform. I am utterly apathetic towards Windows, and using Linux for 8 hours a day at my day job has confirmed that I could not possibly stand to run it full-time on my personal machine. I don't want the Mac to die off. But for the love of god I wish they'd just wait until the software is ready before releasing it.
In the end I had to start with a fresh iOS install, which was very disappointing, as my years of accumulated app data were lost. I manually backed up what I could, but some apps had their data lost forever, and I wasted many hours of time.
Very disappointing.
I feel your pain. I hope you remember in the future:
Unless a backup can be verified, it is not a backup.
Backing up to a complex or binary format is trouble.That’s why I’m extra disappointed in Apple. I did everything by the book, including testing my backups, and my data was still lost.
You can also learn to not trust software which promiess you the world, but doesn't let you see its internals.
if this is true how are we supposed to trust apple with our information?
BTW since updating my iPhone & iPad to iOS 13, and my desktop to MacOS 10.15, the Notes app does not propagate changes unless I force-quit the app and then restart it.
Entering a new note, and quitting the app, the new note does not propagate to my other devices. If I re-open the Notes app then the update happens. WTF
A walled garden ecosystem’s main raison d’etre is for everything to work together. If it doesn’t, it defeats the purpose.
I’ve been an Apple advocate for a long time but things are getting dicey. Perhaps my macs will turn into defacto expensive chromebooks. In other words use them to run chrome and in turn gsuite everything, and nothing else.
God awful
Other people are also having issues with mail on iOS: https://twitter.com/CliffordAsness/status/118240716828649472...
It’s a mess and I can’t trust my phone to notify me. Now I have to constantly be checking it
Either way, disaster of a phone, and disaster of an OS.
http://mike.laiosa.org/2009/03/01/emlx.html
https://wiki.fileformat.com/email/emlx/
It doesn't seem that hard to write some more tools to manipulate them.
https://taoofmac.com/space/blog/2008/03/03/2211
The thing here is that
a) there are also a number of SQLite databases that are used by Mail to store metadata and sundry
b) Mail's IMAP handling has been notoriously buggy at times
My guess is that this particular data loss bug in Catalina stems from a combination of the IMAP implementation and the metadata storage, but since they keep playing whack-a-mole inside ~/Library/Mail, I can't offer more details.
Actually this was already a problem on Mohave. I lost all my mail from one mailbox when moving messages from one folder to another (gmail)
I also just use macOS/iOS Mail because FastMail can push notify to the clients, and FastMail's iOS app isn't ideal.
There are a bunch of other 'efficient' mail apps, but if you just need basic features and don't go for some exotic or trendy workflow, Mail works fine.
I don't upgrade to the latest OS for at least a few weeks, though, because if something like this is systemic, it will be fixed in the .1 or .2 release.
Also, always keep at least one local backup (I have Time Machine for that), and at least one remote backup.
When I need a more featured alternative to something I'll use one (Things instead of Reminders, OneNote instead of Notes, VLC instead of QuickTime, etc) but email hasn't been one of those things.
Is there a reason that I should be looking at Mail alternatives? What important features is it lacking?
And there’s nothing particularly wrong or bad about it (until now, maybe).
It's really hard to find apps like that, and I often feel like it's getting harder as time goes on.
2. It allows me to have 7 inboxes routed to one view
3. It's pretty fast.
4. While "boring", It usually" just works.
I’m not opposed to switching but everything looks pretty lousy, in terms of workflow and integration, and if I have to live with “pretty lousy”, then “zero cost” and “mainstream first-party offering” seem to be the way to go.
If I was someone continually running into issues as some seem to continually have happen I might not feel that way, but I've been on the Catalina and iOS 13 betas and thankfully even with the betas didn't have issues.
EDIT: I did have earlier versions of the iOS betas duplicate draft messages. Which lead to confusion and double sending since at first glance it looked like like I hadn't sent a message in drafts; once I figured out what was going on I would just check my sent to make sure I sent it before deleting the dupe in drafts. At least it wasn't a data loss bug.
I have Thunderbird installed for accessing a large e-mail archive on my NAS (simply because it has more control over local copies of messages) and find it horribly clunky and un-Mac-like even today.
And none of the trendy, flashy "new inbox philosophy" apps that have come and gone over the years were any sort of an improvement.
Even though I work at Microsoft today and acknowledge that the Entourage dumpster fire was finally fixed in the modern versions of Outlook for the Mac, I still prefer Mail.app because of its simplicity and straightforwardness, and refuse to use anything else for my personal e-mail.
Which is why I won't be upgrading to Catalina until this data loss bug is fixed...
It even says,
> I don’t know whether these are due to Mail bugs or to other factors such as problems on the Mac or with the mail server.