It's virtually impossible to read old iMessages and they take up tons of storage
keydiscussions.com
keydiscussions.com
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.
I've worked with 10+ messaging platforms so far and Apple's protocol is the most obfuscated and complicated to reverse engineer. Apple has invested millions of dollars to make the iMessage protocol super hard to reverse engineer. It's how they sell tons of iPhones after all:
> However, Craig Federighi, Apple’s Senior Vice President of Software Engineering and the executive in charge of iOS, feared that “iMessage on Android would simply serve to remove [an] obstacle to iPhone families giving their kids Android phones”. (https://www.androidpolice.com/2021/04/28/apple-admits-that-i...)
That’s surprising. How come?
To access iCloud and or services you also need to have a device security key that is tied to genuine hardware.
This bit did make me wonder though:
>We plan to make it work without the catch.
Out of curiosity, do really think that's realistic, or even desirable long term? iMessage is ultimately an Apple service that runs heavily on Apple's infrastructure, and is directly subsidized by sales of their highly vertically integrated hardware platforms. If it turns into a cat-and-mouse fight it seems like they're always going to have the eternal upper hand, which in turn seems like it'd make for a subpar user experience (ie., breaks randomly which for an instant messaging service would be pretty bad). Also seems like they might actually be motivated to respond rather than ignore it since it'd actually be directly leeching their infra if it will work on PCs/Android without a Mac/iDevice purchase in the equation (unlike hackintoshes for example, where whatever debate there is to be had about probably very low "lost sales" it doesn't actually cost them anything).
Obviously you've probably thought this all through, but seems like just requiring an old cheap Mac or old jb'd idevice and thus avoiding Apple might be an easier path. Or alternately just stick to offering a flat out better client with the iMessage bit being Mac-only. Will be interested to see how it goes though!
The Messages app on Mac was rewritten for Big Sur (last year). It gained some features but also some bugs in the process.
Interesting, how do you plan to do that?
A little feedback on the landing page: Almost turned away because I thought „Sign up with Google“ was the only choice and I missed the little „Prefer entering email instead?“ below. I just prefer not having a Google account ;)
Maybe the styling of that link could be adjusted so that it isn’t glanced over that easily.
Does the texts.com client solve the original problem? Does it make it easy to go back and view all those old text messages? I know the website says "from forever ago easily". But you didn't say that explicitly in your post.
I assume you're being modest and not "pushing" your solution. But if it simply solves the problem, maybe you should be shouting this out loud!
Also, a "jump to date" feature (like Telegram) is planned (and super easy to build.)
You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
We recommend using a simple MITM proxy or a firewall app like Little Snitch / LuLu to inspect the network traffic.
> You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
Yes – there's a peer-to-peer connection between the Windows and Mac devices.
https://github.com/CamHenlin/iMessageWebClient
I also have some other iMessage related git repos on my GitHub. Interesting stuff!
EDIT: No, apparently not. Weird; we all thought that was going to make him a multi-millionaire.
That said, this is it: https://gist.github.com/brianv0/35f36a32366a2c34be8d
(You need an unencrypted backup directory to run it over)
Note: It looks like the flat directory structure of the backup is now 256 dirs prefixed by first two characters of file name. I'm not sure, but a small change to the name "sha_file" may be needed to just path.join that with the first two chars of the sha.
Python 2 but can be ported forward without much effort. Manifest.mbdb is now Manifest.plist (there's a sqlite thing now too). timestamp sql might not work so great, would need to check query.
I've been thinking about using the accessibility I framework to ETL the messages into Kinesis to facilitate event driven architecture. 'archived' Messages and attachments are stored as bsob.
Hell, I've even experimented with some goofy stuff, like FaceTime to VOIP- asterisk is just so flexible
https://github.com/michael-danello/iMessageWrapped
EDIT: It is absurd Apple doesn't provide better message organization and UX/UI.
It's impressively insecure.
*For some definitions of "engineered", apparently.
Usually once I scroll to ~5 months back with people I text regularly the iMessage app just straight crashes ... it's a little absurd. To top it off, every couple pages it gives a loading icon and introduces severe latency for whatever reason. There don't seem to be any apps in their App Store that let you search and lookup old messages either.
This is actively biting me right now. Are there any solutions for accessing the iMessage/chat.db database on my phone or via iCloud from a Linux system (Ubuntu in my case)?
This is basically one taken out of microsoft's .doc format playbook.
At first I thought Apple was slow and just didn't have time to get to some sort of export for iMessage chats. Mail had imap, calendar had caldav, address book had carddav. The original mac stuff did other standards like AIM and jabber.
But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
What's annoying is that I couldn't save my messages, I could only upgrade to a bigger iphone.
What's downright evil is all the people - I know several - who have lost messages + embedded photos/videos from loved ones who have died because there was no export.
In an SQLiteDB any programmer can learn to navigate, and any third party can build a tool to export them?
Sorry but the "internal emails" prove nothing about Messages. It's just a quick and dirty implementation that "works good enough" for reading the latest, but not as an archive.
Any source for that? I couldn't find any direct evidence for your claim.
https://texasnewsexpress.com/index.php/04162021-1249/apple-r...
- The FAQ says you'll eventually open source your SDK. I admire that as a software engineer, but from a business perspective, isn't this risky? It seems like--if you don't pass messages through an intermediary service--your value-add is mostly in this SDK + your design. All of these elements are copy-able. - The jobs page includes a backend engineer, so what will go through your backend, if not any message data?
- Are there legal concerns about reverse-engineering these APIs and using them? Some of your supported chat services have open APIs, but others seem intentionally closed. (And a couple parent companies seem downright hostile to 3rd party use.)
This aspect of the iMessage user experience has remained untouched for a decade, despite everyone's chats growing longer. Searching for particular iMessages sucks because the results miss the broader context of the conversation (a single message is not useful when you want to reread a conversation).
(Also just noting, as the post author I created a title better-suited to HN's 80 character limit.)
If you do, tell us about it here on HN.
If it just pinned the message thread to the top and didn’t change the icon to a giant circle that sometimes doesn’t show notifications, it’d be fine. I pinned my GF’s message thread and I ended up missing lots of messages and notifications from her and ended up unpinning her so the notifications would show up.
Not to mention the comically huge circle icon that is centered when everything else is left aligned.
It only made things worse :(
It defeats the entire purpose of pinning.
I had a problem that appeared to be a corner case with Apple Pay. I wrote a paper letter to Tim Cook about it, and within a week got contacted by a staff person in his office. He worked exclusively with me to identify issues I had not considered, and followed up on all of the direct effects and side effects until they were resolved.
[0] - https://www.inc.com/business-insider/tim-cook-wakes-up-at-4-...
Also, be sure to file a bug report / enhancement at https://www.apple.com/feedback/iphone.html
- I’m not wasting my time emailing a top level company executive to inform of decades old most basic of bugs because if they don’t know of it then that’s even a bigger problem.
However, deleting through this interface would be a pain. You select "Edit" and then each attachment is selected individually.
Really wish there was a "delete all" or "select all".
None of the problem listed in the blog post or what you point out are new. They have been there since day one. And it is not that they neglect it either like iTunes or Aperture. They added Memoji!
Which is why I am surprised a the popularity of iMessages in US ( and France ). I think it is mostly due to SMS usage before iMessages was a thing.
https://spin.atomicobject.com/2020/05/22/search-imessage-sql...
Why would you not recommend it when there's no alternative?
Lighten up a little, GP.
And, yeah, it’s a total mess.
When I went to open the email on my new machine, the loading indicator just spun and spun until finally my browser crashed. When viewing an email conversation, Gmail will display previews of all the emails in the chain, including their attachments, which was causing my browser to crash. Next I tried connecting my Google account to Apple Mail, but Gmail's SMTP server would time out before the attachments were able to download.
After a while I forgot about the photos all together, until I randomly stumbled across them while searching my Gmail account for some other files I had sent myself around that time. At this point it was 2014 or so, and my browser was very easily able to open the messages so I could download the photos, which of course at that point I had zero interest in.
I can only conclude that they have made it deliberately very slow and painful to clear space on iCloud. Because they'd rather you paid for more space ...
Why wouldn’t it sync? If I did this right now, my expectation is that it would sync. Is there a limitation in how this works?
Take a photo on you iPhone and it take days for i to show up in the photos app, it instantly shows up in iCloud accessed through the browser.
Same thing in various other directions, delete on web, and it takes days to disappear from other places. Makes zero sense.
Often if i just took some pictures i have to send them via airdrop instead of just opening Photos. Come on Apple!
I am failing to see the additional value of combining them. If Google bought WhatsApp and combined it with the stock SMS app of Android, I don't think I would prefer that.
A system where you can’t message half your friends because they bought their phone from a different manufacturer would have had more trouble getting traction.
Well, it's not useful now, but imagine in the 00s when a minority of users had iPhones. Having two clients, one for SMS and one for iMessages, would have gone against the simplicity Apple was aiming for with respect to user experience. Combining it meant that users could get a better experience (than SMS/MMS at least) when communicating with other iMessage peoples, but not have to swap out clients to chat with others.
Cynically, they intercepted millions of people's comms with hardly a murmer. Charitably they switched millions of people from country-dependent, pricey and limited SMS to internet messaging with read receipts, pictures, video, emoji, audio recordings without any hassle of logging in, changing apps, choosing clients, etc.
This is all controlled via a setting in iOS iMessage settings “Send as SMS”, which I believe is turned on by default but I’m not sure. So your mileage may vary.
Most of my friends do not have whatsapp, and we are constantly sharing photos and video. Regular SMS can barely send pictures let alone videos, and I don't want to go through other means to simply shoot my friend a quick video of something.
That tells me whether the person is replying right now, so it's worth keeping my phone out, or busy and therefore won't be getting back to me for a while.
The time savings add up.
I don't think most people actually give a shit about the bubble colour.
iMessage is a messaging app that uses internet. It should be compared to WhatsApp, Telegram, Signal, WeChat, FB Messenger.
It doesn't make sense to compare iMessage to SMS. Because nobody uses SMS as their main communication method. The only people who ever use SMS are iPhone users when they text Android users.
All this confusion comes from the fact that Apple merged their messaging app with the SMS app. This creates the illusion that iMessaging isn't just a messaging app but it is actually something built on top of SMS. It's just a messaging app that is not cross-platform.
> I don't want to go through other means to simply shoot my friend a quick video of something.
By "other means", do you mean installing a new messaging app? I'm guessing this isn't about the effort of downloading an app but rather it's about getting used to a new app and convincing your friends to switch to it.
One of my friends had her H1B cancelled and was given a 10 year ban from entering the US because of her text messages. When she crossed the border back into the US, the border guards requested her phone, and searched it and found messages to her ex-bf about marrying him for a green card. She was joking but it didn't matter and she had to fly back to her country of origin that she didn't live in since she was a kid. Since then, even though I don't have anything to worry about, as a habit I delete all my apps except iMessage and delete all my conversations there.
https://appletoolbox.com/qa-where-are-imessage-files-stored/
It does have some important limitations.
1) I do clean installs when I update macOS, so the Library folder is created new each time. Which means that archived messages only go back to that point. (I browsed via Terminal and used the command line 'open' command to view them.)
I do retain old ~/Library folders and it's possible to use the same technique in finding messages there.
2) But, and here's the big limitation: at a quick glance, they're simply individual messages, by date. There's no way to follow entire threads. I suppose it would be possible to write your own program to organize these. But the point of the article is that Apple should make it easy. Your method isn't easy.
Edit: response to sibling comment. I'm still on Catalina and the above works. I have no idea about Big Sur.
How does it not realize that it should first search recipient names and present these results first? A search for "Delilah" should not show a bunch of texts about Delilah, obscuring the one thread with my friend Delilah.
Not saying having it also work from search wouldn’t be good — but having this flow available is probably why the other issue hasn’t been fixed sooner.
I think this is how MS teams chat works too. It kind of makes sense, because it has a workflow for solving the "do you want your existing chat with Alice, or are you trying to start a new group chat with Alice & Bob" .
Messages on Big Sur is (basically) the iOS Messages app ported over with Catalyst, so it has a lot of improvements compared to the highly-stale app it replaced.
The scroll-load-scroll-load cycle does suck, though.
With every iOS and Mac OS release I’ve been hoping they’d fix iMessage, iCloud, App Store search, app discovery, extension ecosystem for browsers, or my iPhone taking those extra 7-8 phantom GBs (..iPhone Storage/Others) that everybody guesses could be some cache or logs or something magically else and speculates maybe one day it’ll go away but nobody knows for sure etc.
But no! It seems Apple has been saving these things so that after 10 years they get to say on stage “Now you can search apps like never before. We have named this patented revolutionary app discovery as iDiscover™. As simple as searching for apps!“
Or after another 5 years something like “Now you can read your iMessages in a more readable manner and it will truly sync now. Meet TrueSync™“ when their engineers will have a breakthrough after 17 years of backbreaking research.
I have always wondered how the engineers look in the mirror after hearing those pathetic stage ridiculousness.
For iMessages every photo or video you sent seems to take up separate storage in the iMessage portion of iCloud, and it's difficult to delete because you have to scroll through and select them.
2 apps I use for similar non iMessage archival are - x1 - for file and email search https://www.x1.com/products/x1-search/
- mailstore https://www.mailstore.com/en/products/mailstore-home/
No one asked for managing texts like we manage files.
Maybe not managed like files, maybe you’re right, but it seems like if we can make trillion dollar tech companies we should be able to completely solve this particular issue.
I’d frame texts as one of the most intimate expressions of dialog and journaling between people and families and friends. Just seem odd that this isn’t better.
I'd assume this is why they discontinued it, rather than adapt it to the new from-iOS Messages in Big Sur.
I also have a similar gripe with the browser. How can I sync my Safari tabs that are open on my iPhone with my PC?
I would recommend switching iOS browser to be able to sync your browsing across vendors
Having such a huge trove of history of all of your communications saved that can be accessed by the police or a rogue Apple sysadmin at any time is something that I feel should even make the law-abiding nervous.
If MIC is enabled, iCloud Backup backs up the key, breaking the e2e. If MIC is disabled, iCloud Backup backs up the plaintext message history, breaking the e2e. This is well documented in the Apple KB article 202303 on the topic.
It’s no longer an accident at this point.
https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
https://support.apple.com/en-us/HT202303
> Messages in iCloud also uses end-to-end encryption. If you have iCloud Backup turned on, your backup includes a copy of the key protecting your Messages. This ensures you can recover your Messages if you lose access to iCloud Keychain and your trusted devices. When you turn off iCloud Backup, a new key is generated on your device to protect future messages and isn't stored by Apple.
It’s written in a misleading style, but if you read it carefully it is apparent what is happening.
Turning off iCloud Backup is insufficient, as each conversation is backed up twice: once from each side. Your conversation partners’ devices default iCloud Backup to on.
Apple has the ability to read almost every single iMessage sent or received, on a 12-24 hour lag (iCloud Backups happen overnight), from data already on their servers, without ever touching a customer phone. All of the encryption is a handwavey distraction.
1. Use macbook.
2. Open imessage and scroll 1 year back.
3. Export to pdf.
4. Ensure the pdf contains 1 year of chatting.
5. Delete all the messages that have just been exported.
6. Repeat above for another year.
In the end you will have a pretty looking archive of imessage.
Even an old 2013 macbook with 8gb RAM is capable of accomplishing this task.
It is indexed in a relational database, and it is presented in a Web 1.1 frontend, but it all flows back through the text files, and they are always available for download, optionally in zip or tar.gz archives.
I don't think it works any longer on the latest iOS, but I open sourced the code if anyone wants to pick it up:
I started migrating people to Signal when that whole WhatsApp ToS hullaballoo happened, but now I need to change my phone number and apparently this is not supported in Signal, which is just crazy to me.
With photos, I guess this could end up being around 1GB. Maybe this will push you off the free plan to the $1/mo plan, but I don't see it being that big of a needle-mover for services revenue.
That's a nice bonus, no matter how much the needle moves.
https://appleinsider.com/articles/21/01/27/there-are-1b-ipho...
I definitely prefer my current phone that lets me just rsync ~/Pictures.
You can set it to sync whenever your phone is inside a particular location boundary (e.g. at home, where there is wifi), so it works transparently with zero ports open to the internet, and no firewall-bypassing cloud services.
iMessage never worked since the upgrade. I can only send messages, I see them once, but once I close the app and open it again, I don't see anything else. There's absolutely no way to reset the db of an Apple-included app unless I do a full format of the device, which after my full custom-configuration, I don't want to do. It sucks.
This is probably due to the fact that in my country the majority of people has Android phones, and thus even who has an iPhone uses the same app that uses with people with Android to chat with other Apple users.
Other reason may be that it is used by services - they want as long history as possible to find out what kind of person you are in case you are being investigated.
Anyway these artificial obstacles most often have sinister motives.
Ten years down the road, "oh, looks like you need to start paying us monthly for more cloud storage!"
I kinda wonder how many Gmail users now pay Google for more storage space just to cover their email archive... And how many will in another ten years.
Gmail’s storage space has consistently grown over the years to keep up with the average user’s demand.
Google don’t want to do anything — like charging for email storage — that would disincentivize their mass of sensors^H users from feeding them ever more data.
(If you’re an outlier with way more storage than usual, they’ll happily charge you, knowing that the data point you represent is mostly going to get pruned out of their models anyway.)
Also, now they're actually effectively decreasing everyone's storage. Previously, Google Drive document types and Google Photos didn't count towards that 15 GB limit. They will as of June 1st, 2021: https://blog.google/products/photos/storage-policy-update/
So, while cloud storage usage has exploded over the past eight years, Google's storage limits have not only not expanded to meet it, they've actually begun to tighten the belt.
I use Gimp and generally obtain the same quality with a 450KB JPEG, sometimes 50KB depending on the contents. Images can be remarkably small. They should try increasing the compression ratio until a change is noticeable, but do they care?
That fine for final/production images but it's a terrible idea for your originals, or really anything you might want to edit beyond cropping).
The restore daemon actually downloads every single image and attachment you’ve ever sent or received one file at a time.
Not that I can scroll back that far, but you never know when you need to search for something.
It may be a problem with PhoneView, but I'm not convinced that it is. Also, due to the size of the backup, I do have to do it to an external drive.
I have 75GB of messages in iCloud, I see 25GB on my 500GB Mac and 11GB on my 64GB iPhone.
The Bible contains about 3.5 million characters and an average Bible weighs about 500 grams. 11 GB of conversations would then be about 1.5 tons of Bibles.
Any plans to include phone calls, voice mails, and email too?
Future perfect:
RSS reader-like UI. Top level is Contacts. Including groups (w/ aliases). Drill down to see missives. Show read, unread, return receipt, etc. Sparklines representing activity.
iMsgs, esp vids --> GB --> new phone + icloud $$
But, I'll be honest; if you start to see all of Apple's decisions through the lens of "business decisions", you'll tend to get pretty disenchanted and crestfallen (perhaps even resentful)i encourage othes to consider doing the same. its been great for me.
just in case.
i hope we all learn what good digital hygiene is.