OpenPush: A Free, Decentralized Push Messaging Framework for Android
f-droid.org
f-droid.org
Our product is a XMPP client, which can't work if it is not actively running in the background, so we were working with making apps run in background on Android for over 10 years, since version 2.0.
The trend is clear: with every update, we get more restrictions. Not only bg services are killed off from memory by aggressive memory managers of crappy phone vendors, Google themselves push developers to rely on proprietary FCM services. The writing is on the wall, and one day background services in Android will simply cease to work, all in the name of better battery life and customer care. So this OpenPush service will be short lived, it'll likely be disallowed to work in this form in Android 12 or 13.
The only proper long-term solution here is an anti-monopoly legislation, that would make Google allow Android users to subscribe to a third-party push notification service. Such service is, actually, a relatively simple: current FCM is just a modified XMPP connection that a phone keeps, and it will be easy to replicate and deploy.
Apple is guilty of this, too. A proper anti-monopoly legislations should target them as well, and not only make them allow third-party app stores, but third party push notifications as well.
This control of push notification services and restriction creep that occurs on both platforms makes Google's and Apple's grip on their respective OSs much tighter than Microsoft's grip on Windows ever was. This tyranny must be stopped, and I don't believe it can be done without legal action, by technical means only.
In that instance, having an app which centralizes the push service needs of other apps would be highly useful, as it could then be installed as a system app with the same kind of access as the FCM service. For many phones, custom ROMs are available. Average people probably won't have the skills to install custom ROMs. But the goal of OpenPush is not to replace FCM from all phones everywhere. It's to meet the needs of the subcommunity of people who want to use degooglified Android, maybe also later on integrate with non-Android mobile OSs. In that subcommunity, people likely already use custom ROMs, and among those who don't, the readiness is higher to install one.
They caved in for the current release, but they are going to force it in android 11.
On the bright side I think this will cause android engineers to become more of a specialty as the platform migrates more and more from just another java implementation to a whole new world of apis!
It is a tough spot. How do you have open app stores and not have them filled with malware? Android hasn't figured this out yet. Users are dumb and will just click on anything. They need to be protected from themselves sometimes.
But Google decided that it will be our nanny and will guard all users against misbehaving apps, whether the users asked about it, or not, and you can't refuse this care. That is tyranny and should be fought against.
Now, I don't think that push notifications are bad. They are rather good idea and let them be. It is the monopoly on push notifications which is bad.
> How do you have open app stores and not have them filled with malware?
Long before the 'app store' term was coined, Linux had software repositories, which worked like magic, and without issues:
sudo aptitude install libreoffice
And comparing software repos with app stores is just as bad.
I'm not even quite sure how to address what you're saying, because it's just so completely out of touch with how smartphones are used.
Play Store would be one among many.
I think then there would soon be curated repos for well behaved apps that don't abuse push notifications and background abilities.
Strange idea. How is comparing two types of consumer-market computers and ecosystems is "insane"?
The parent comment is pretty legit IMHO.
My mom and my wife parents are extremely happy with Ubuntu computers. They do browsing, email, Skype, print stuff and whatever else they need just fine. That are way more happy with Ubuntu than they were with Windows, and I am, too, cause in the last three years I have spent exactly zero minutes maintaining their computers. If that's not a consumer-market OS, I don't know, what is? Windows? When mom used it, she always had one problem after another with it. I'm truly sorry for anyone who has to deal with it every day.
On the other hand, we invented inetd so that we didn't have to keep server processes running all the time when they are only lightly/sparsely used.
This is not the case on Android. Looks like every app wants to run in the background and drain my battery. Even seemingly foreground-only things like maps apps install background services.
Ideally, I'd prefer to have an explicit list of apps which are allowed in background.. but missing that, I guess I will have to settle on automatic guarding.
What are Google's motives? Do they really care about their users, or do they simply want more control over user devices?
Users will also be prompted to kill an app if it's detected running in the background a lot. That's perfect because there are many cases where only the user knows whether the app updating in the background is important to them or not.
There’s no extra control outside of the user and the services they want to talk to (I think you can even host your own on Firefox), no vendor lockin, and the power and bandwidth usage are all great. There’s too much money to be made breaking useful push notifications and charging for ads though so I’ll be shocked if it ever gets done right on popular consumer devices.
Seems to work nicely on my Android. Which would imply it's already done. (I'm sure greed will ruin it at some point, somehow.)
I'm curious to know if anyone is using this framework in the wild.
You can read more about it here in case you are curious: https://hackernoon.com/notifications-in-android-are-horribly...
If you're not whitelisted, there's very little you can do except for ask users to take various manual steps to change operating systems.
This site ranks Android OEMs that kill apps in the name of battery optimization: https://dontkillmyapp.com/
I believe this aggressive behaviour tends to come from user-feedback, given users care a lot about battery endurance and heat dissipation.
Absolutely. I don't want my phone to run hot or low on battery when I'm not using it. If there was a way to shut down all background tasks when the screen is off, I'd gladly do that (second device, no sim, WiFi only use case).
If I want my email app to download my emails as they come in because I'm on a slow network connection, I want it to do that even if I get a lot of emails. Especially then. Even if that means I have to charge my phone every day instead of every week.
What they should be doing is using a blacklist instead of a whitelist. Not because they're actually going to be able to blacklist every misbehaving app (though they could certainly catch the most popular offenders), but because that's how you get developers to change their behavior.
If everybody is restricted by default then there is no incentive for bad developers to do better. They get restricted, they weren't going to get whitelisted either way, so they go on making a wasteful app. By contrast, if making a wasteful app could get you blacklisted and restricted when you wouldn't have been otherwise, well now you've got a deterrent. Meanwhile they wouldn't be defaulting to breaking well-behaved apps that are designed to efficiently run all the time.
I'm happy that I'm using LineageOS. My phone is turning 7 years old, and it's running Android 9, and I've no reason to switch so far.
It is the only way I can get my K-9 mail app to stay alive to sync my email.
The solution to the background limitations for now is running as a foreground service. This shows a persistent notification (which can be hidden by the user) but can then run indefinitely. The other option is to integrate this at the system level (via a custom rom, root, magisk, etc.).
> I'm curious to know if anyone is using this framework in the wild.
The client side implementation of this is still WiP, so it's not used yet in the wild. I hope to be at a point where I can work with some apps on integrating it there in the next few months.
Just bought a Xiaomi Redmi Note 7 and it kills everything including foreground services with a notification.
A simple user still can allow it but it will require some tech knowledge to follow instruction that are described here: https://www.reddit.com/r/Xiaomi/comments/amcck5/miui_10_keep...
You don't always have that luxury.
If a OS is actively sabotaging you, then yes, the only real solution is to replace it with something else. That's the simple and obvious solution.
The simplest way to do this is to not buy a phone with shitty OS in the first place. As in "Don't buy a phone from Xiaomi".
And when users complain about it you tell them that the problem is their phone manufacturer. If you can, of course, give them a work around so the app will work. But eventually they'll plug all the holes that allow for a functional phone. So it may not always be a option.
Despite this being true many will see is as an excuse, or worse just a plain lie, and leave you a 1* (or 0* if possible) review ranting about your attitude and bad coding. How dare you criticize their choice of device?!
Should you simply drop support for IE and punish people relying on it and hope it dies quickly?
Or should you maintain legacy code, and actually helping IE to survive by supporting it?
I find that issue to be quite complicated and I'm still unsure if there is a right answer to that.
As far as I know, the only truly 100% reliable way to never be killed is to run as an accessibility service. I haven't implemented this in Red Moon but will likely add it as an option. I believe it has a large impact on bettery life.
You can also set alarms using AlarmManager to restart the service if it has been killed, and optionally wake up the device. This is what DNS66 does.
I'd look into that before starting that implementation.
[1] https://blog.lastpass.com/2017/11/lastpass-android-accessibi...
https://github.com/LibreShift/red-moon/issues/253#issuecomme...
I wish this was more obvious to people. Had to use my retire parent's phone to long into Netflix for them and it was overflowing with of persistent notifications.
The problem is harder than it seems on the surface as idle connections can be terminated by the network operator. There's a paper about this: https://ieeexplore.ieee.org/document/8214213
I guess ultimately one should build a database about phone networks and store measurements of their maximum heartbeat intervals in there so that new connections can use those heartbeat intervals to save battery.
Also ideally the app should send its data via WiFi if available. So it has to detect when a WiFi connection is established and open a connection wia WiFi. Ideally the mobile connection is kept alive though for the case the WiFi proves unreliable. E.g. when I leave my home on foot I get further and further away from the WiFi and there's a short time span where my phone keeps trying to use the WiFi connection even though nothing gets through any more.
Detecting all these edge cases can be tricky. Fortunately the design makes the protocol between the push server and the on-phone app open to iteration and innovation. It's a good design!
IIRC they use the xmpp protocol, so standard TCP over IP. In threads about this topic you often hear claims that Google has special deals with carriers. Whether this is true or not, no idea. I haven't seen any confirmation of this. It would be interesting to conduct measurements whether FCM sends less frequent heartbeat messages than the timeout is for "normal" long lived tcp connections.
In any case, deals or not, I guess Google has built a database as I've mentioned. But I'm only guessing here.
Start at a 1 second ping, and after each successful ping, double the interval until you get to over a minute. Then add a minute between each ping until you get to 10 minutes, then add ten seconds between each ping. If a connection drops, use the last successful interval for tenish pings, then probe with an additional one second. If the connection drops again while probing, reconnect and wait longer to reprobe. If the first ping failed, then assume the network changed and reprobe from the top.
And yes, you do need to start at one second; I've seen networks with a ten second NAT timeout, which was amazingly disruptive, and thankfully temporary.
Do you have any references for your assertion?
However a smart application/protocol writer could make make a ping and a message delivery take the same number of bytes. Which of course means some wasted payload on pings.
Additionally if trying to track peers you could every N seconds send a ping OR a message. After all if you succeed in sending a message you know the peer is reachable.
So it's possible, but no idea how careful OpenPush is though.
Likely not a good fit for TCP streams (like tor), but could easily hide enough bandwidth for instant messaging (like signal) or email, er email like service. Could hide store/forwarding, include onion routing (bouncing through a few nodes), and be quite resistant to traffic analysis.
If each node send X packets of Y bytes to Z nodes per minute then it would be quite hard indeed to tell who you were talking to.
Maybe give peers a few choices in high bandwidth, medium bandwidth, and low bandwidth configurations. All would just vary in the number of nodes they track.
An open push framework only makes sense (for battery optimization at least) if it's centralized, and all applications make use of it.
Sure, from the perspective of a single device. But that doesn't mean all devices have to use the same push server.
There are no strings attached, besides providing reports on scientific contributions achieved throughout the project.
It's not like the BND will come knocking on your door saying make algorithm X weak or you will not receive funding (or worse).
Let's consider a situation in which this might be useful: You have a 15 to 30 minute task running, such as downloading a large file (e.g. Linux ISOs) from a remote server or compiling a small C++ project. Instead of sitting around waiting you could go outside for a short walk in order to get a bit of exercise and sunlight, which helps you get to the recommended minimum of 10k daily steps. Once the task is finished you receive a text message which prompts you to head back inside.
If you just want to send yourself notifications then you can actually get pretty far only using email and text messaging. However, I'm aware that OpenPush solves a few different problems, and I love that we're finally getting an open-source self-hosted alternative to GCM.
[0] https://support.google.com/fi/answer/6356597?hl=en&ref_topic...
I still think it's worth it, to be clear, and for people who want a fully de-Googled device, hopefully eventually all their apps will use OpenPush and the battery life situation is the same as a device where all the apps use FCM. (Assuming OpenPush implements similar coalescing.)
I'd be the richest guy if I'd get a penny every time I've seen an Android app do background polling every 5 minutes (!) instead of using push notifications or even long polling. The usual response to recommendation to improve the design was "Why? It works and it's a lot of work to run a push server!".
So as much as I dislike restrictions on my operating systems, Google forcing developers to push notifications has massively improved battery life across the ecosystem.
I don't know of it's an obligation to use google's firebase pish notifications though. I might be mistaken, but I think that Conversations use its own push notifications already: https://play.google.com/store/apps/details?id=eu.siacs.conve...
Tutanota published a blog post explaining how they deliver push notifications to their Android app with SSE instead of FCM:
https://f-droid.org/en/2018/09/03/replacing-gcm-in-tutanota....
OpenPush would reduce the amount of work needed to implement push notifications compared to the made-from-scratch implementations developed by Signal and Tutanota.
I think I have about 30 false start projects with names like PushMux and rough outlines on potential ways to accomplish it.
Tangentially, I also dream of having true one-way push notifications. Send a minimal notification to the target device via FLEX/POCSAG or even one-way Iridium, and allow the device to connect to the internet as needed to fetch the rest of the notification.
For all the edge cases out there, OpenPush is great news.
However, this is a huge no-no for me as it means that I would need to actively manage the device to make sure that things do work as intended.
Not happy with Google "owning" the push notifications? Then maybe the solution should be a non-profit independent corporation running it.
Having a service-per-app app cannot be as efficient as having a single service serving them all. Both in terms of device resources and user time & experience.
Surely having an alternative is a good thing but the issue seems to be institutional(You don't want to have Google in charge), not technical(Google's service works great), therefore the solution out to be institutional too. Or something that acts like an institution, something like bl_______n.