Hacker News Reader

Comment by caconym_ | Hacker News Reader

I'm asking because I hate Matrix and actually want you to convince me: why should I accept the risk of migrating my friend group from Discord to Zulip, which has already "broken the seal" of restricting features behind a monthly fee even for self-hosted users, when I could migrate us to Matrix instead? Matrix seems like the much less risky option.

I see that you have a "community" tier that's free and doesn't restrict notifications, but it's not clear to me exactly what's involved in proving that we should qualify.

View on HN

Mobile push notifications are a special case because it's literally not technically possible to self-host them. Or rather, it's possible if you build the iOS and Android apps from source and distribute them through TestFlight or an analogous Android channel, but it's not possible for the developer of an App Store or Play Store app to allow its users to point it at a different push-notification server, because the public key has to be hardcoded in the app binary. So if you want your self-hosted Zulip server to work with the Zulip client apps in the App Store and Play Store, you have to use Zulip's push server, and there's nothing Zulip can do to fix that.

Matrix works analogously; if you use the Element app from the App Store or Play Store, then you're using Element's push notification server, even if your Matrix homeserver is self-hosted. It's possible that Element allows their server to be used gratis in situations where Zulip charges a fee, I don't know their policies or anything, but in principle Matrix still leaves you exactly as dependent on a third party's goodwill unless you make your friends install a privately distributed mobile app.

Zulip IIUC does not restrict self-hosting of any feature that's technically possible to self-host.

> because the public key has to be hardcoded in the app binary

Nope. On iOS the flow is:

1. Generate a "push token" on the device (with the user's approval).

2. Send this token to your server.

3. Now you can send notifications to the device via this token. Your server needs to authenticate itself with Apple, and this requires an Apple account. But it's not linked to an individual app.

The situation is different on Android. Google went out of their way to make it impossible to customize `google-services.json` at runtime. So the built-in "easy" flow won't work. But notifications ultimately work using veeeeery obfuscated remote procedure calls to Google Play Services and you can run them manually. I need to do a write-up about this....

i would read that write-up!

> Your server needs to authenticate itself with Apple, and this requires an Apple account

How does Firebase Cloud Messaging work with Apple without an Apple account, or is that implied in the client generated push token residing in Firebase?

I'm not sure, because I'm using Apple's notification servers directly, not through Firebase.

I believe it works just as a proxy, and you need to specify your own Apple credentials when you set up the Firebase project?

You're implying some difference here that I don't see.

Both platforms need some way for the client to register to their respective push services, Apple needs an Apple account, Android needs google-services.json.

Both platforms require your app to generate a token which the platform's respective push service holds, and send it to your server which you then use to identify the client you're pushing to.

Apple also requires the Auth p8, Bundle ID, Team ID and Key ID, which are roughly equivalent to the contents of the google-services.json.

On iOS you don't need a separate binary, the server-side infrastructure is not scoped to individual apps.

I don't think that it's possible on Android.

The solution for this is to install the self-hosted Zulip as a PWA, but unfortunately they don't support web push.

Yeah. This is exactly my worry: as soon as solutions to technical problems like this start going in the direction of "we'll offer a monolithic solution and charge users for access to it" instead of "we'll make it as generic as possible even if the alternatives for now are flawed", it makes me wonder about the long term trajectory of the project.

I don't mean to cast aspersions on the developers—I respect everybody's right to try to get paid for good work, and this looks like good work. I am just not convinced it's the right option for my specific needs.

What would "as generic as possible" look like in practice?

https://news.ycombinator.com/item?id=46954896

I think it's right to worry about this. However, the Zulip project has a long (10 year?) history and track record with a stable team.

Can someone explain to me why we should do engineering work to build features where the stated objective is to help corporations use our product without paying for it?

Remember, self-hosted mobile push notifications already have a free community plan!

I can't say whether Zulip might be a fit for your needs. But you can always email our lovely sales/support team. https://zulip.com/help/self-hosted-billing#paid-plan-discoun... should give you a sense of our policies.

> Can someone explain to me why we should do engineering work to build features where the stated objective is to help corporations use our product without paying for it?

I'm not sure what you mean by this. I am looking for a replacement for Discord for my small community of friends, and before today I had never heard of Zulip and knew nothing about its pricing or policies or history.

> our product

It's either your product first or an open source project first. If you care more about having a marketable product then it's fair for people who care about open source to go elsewhere.

It's not a technical problem, it's a policy problem from the mobile vendors. You'd basically be paying them to deal with Apple's and Google's messaging infrastructure through their (zulip's) infrastructure.

I understand that (IIUC in Matrix the client decides what push gateway to use, and the Element client just hardcodes matrix.org and lets anyone use it for free), but it doesn't really do much for my practical concerns. I'm looking for something my users can tolerate (which means no monthly fee) and that I can be reasonably confident won't rugpull us or vanish in the next ~10 years.

I guess my question is, what makes you confident that Element won't change the terms under which people can use their push server?

Nothing, but there already exist many other Matrix clients (shitty as they may be at present), as well as (IIUC) an Element PWA that uses web push (which is IIUC supported by Synapse) for notifications. Synapse also (IIUC) can be configured to use an arbitrary push gateway.

This is what I mean by "generic" in the other comment you replied to. I appreciate the value of tightly integrated server and client applications, and fully believe that Zulip's implementation of notifications may be both a) better for usability and b) a lower maintenance burden for the development team than supporting web push in a PWA, but---again---I am looking at this from a certain perspective where the way Matrix is architected and the breadth of the ecosystem imply less long term risk for my use case.

So it's not actually an architectural issue, it's just that in practice there are multiple competing Matrix apps whereas nobody has yet bothered to do this for Zulip?

> So it's not actually an architectural issue

I mean, maybe? It's not obvious to me that Zulip is just as easy to develop a third party client for as Matrix.

> in practice

What's true in practice is all that matters to me, because I have a real life use case. The only might-bes I care about are the risks.

But how ntfy does it then? It is one app that allows you to subscribe to multiple different notification endpoints. I have uptime notifications set up this way.

Wouldn't it be possible for Zulip to go this route as well?

Because ntfy doesn’t, at least not in a way that detaches you from a central authority.

On its own notification to your device will happen eventually when the ntfy app on your phone wakes up and polls. Pull, not push.

My ntfy server has a config line for an upstream, which is a service that then uses push. Basically it’s self hosted and handing off push.

The difference between ntfy and another type of push is that you don't need a server owned by the group that makes the app forwarding messages through apple or Google. You can have your chat server send messages to your ntfy server, which then arrive on your phone.

The default behaviour for self-hosted on Android is to have a foreground service which holds a websocket open, so it does get pushed from the server and doesn't rely on your phone being awake.

https://docs.ntfy.sh/subscribe/phone/#instant-delivery

The upstream approach you describe is only necessary for iOS devices that don't permit apps to do that.

https://docs.ntfy.sh/config/#ios-instant-notifications

On Android the OS implementation of "push" notifications is pull/poll based as well. At some interval, the OS polls Google's servers to see if there are any messages available. Firebase essential acts as a message broker, so that it only has to poll a single server, instead of a separate server for every service that wants to send notifications, and there is only a single service polling.

But I really wish Android supported specifying additional servers to poll (and/or replace the default server), so you could use a self-hosted service in addition to or instead of Google's service.

The same way that Element does - they host a service for you that relays push notifications their Firebase Cloud Messaging endpoint for Android or iOS Instant Notifications for Apple. I believe ntfy's hosted option is the way they offset the costs of hosting this, even if self-hosted options can take advantage of those servers free of charge.

I think it's reasonable for Zulip to ask for compensation for access to these gateways, since Apple and Google do not make them available to end users free of charge, and the burden of responsibility to ensure that these systems aren't abused is on them. Also, the fact that they offer mobile push notifications for any self hosted server of up to 10 users is pretty generous, and there seems to be a Community plan option for larger servers that includes "groups of friends" as a qualifier. It really seems they're offering quite a bit.

This isn't true, self-hosted Android push notifications in ntfy are provided using a "foreground service" by default (i.e: the app keeps a websocket open and listens), unless you set up firebase for yourself and build a custom version of the app with the cert baked in.

https://docs.ntfy.sh/subscribe/phone/#instant-delivery

So it either drains battery and gets significant delays (as per ntfy's own docs), or it still uses Google's solution. There is no free lunch.

I think you misread, the delays are if you don't use instant delivery. I use it and it's extremely consistently delivered instantly, which makes sense, it's a websocket.

As to battery drain, I'm sure it technically does consume more, but according to my phone it's an insignificant amount: <1% of usage which is the lowest stat it gives you. Their docs suggest the same thing:

> the app has to maintain a constant connection to the server, which consumes about 0-1% of battery in 17h of use (on my phone). There has been a ton of testing and improvement around this. I think it's pretty decent now.

https://docs.ntfy.sh/faq/#how-much-battery-does-the-android-...

Honestly it's a good solution that works well with few downsides, the only real one is that iOS doesn't support doing it, but personally I don't have any apple phones so I do get an essentially free lunch.

Listening on a socket doesn't drain any battery when no data arrives unless the app does other things that actually use CPU. That's just what Google/Apple want you to believe so you depend on their proprietary lock in services.

Also like, how else would the Google / Apple services do it? Probably via sockets right? I guess you could do it in a pull-based approach on a timer, but that doesn't seem more efficient to me.

But waiting on multiple sockets for each app is surely more expensive than a single one, isn't it?

A single process waiting on multiple sockets is basically no more expensive than a single socket, but if each app has its own background process then that is more expensive. So for best performance you really want to delegate all the push-notification-listening for all the apps on a device to a single background process owned by the OS, but it'd be fine for each app to use its own push server (though of course most apps do not actually want to self-host this).

Google doesn't have any magic way to do instant notification that nobody else has access to. The only thing they have access to in this regard is disabling any battery optimisations without triggering warnings.

Notification and battery performance is on par with google's solution except when an android build does dumb things to prevent the background activity, in which case notification performance gets worse and battery draw gets worse (not sure why exactly, it's just a common issue in these regards).

Well, there is an advantage, if everything is using the one service then you only need to have one thing alive to check it, so each new app is "free" if you already have push enabled (assuming that push notifications are rare enough the activity isn't the cost), as where each app doing it themselves is going to cause more battery use, so it isn't directly equivalent.

However, it also isn't a big deal, at least in my experience, at least for ntfy.sh.

It does.

Ntfy pays Apple/Google for the ability to deliver notifications to you. They use the free plan as a "gateway drug." It's just a cost of business to them, a marketing tactic to acquire paid users, no different in principle than plastering ads on billboards.

You can't set up your own Ntfy server (at least not without also having a private copy of the Ntfy app).

(Things may be different on the fDroid side, but many custom notification servers are a batterly life and privacy concern nevertheless).

You can self-host their notification server, and in their app you provide this server's url.

> but it's not possible for the developer of an App Store or Play Store app to allow its users to point it at a different push-notification server, because the public key has to be hardcoded in the app binary

Setting up stoat.chat right now, I'll let you know if I have any notification issues with it ...

There is a lot of truth in what you write, so I am just going to point out the UnifiedPush project[0].

Of course with the, rather large, caveat of that not working outside of

> TestFlight or an analogous Android channel

This implements a push service (caveat: Android only) that is less restrictive than what google provides, and allows the reuse of an existing notification server (ntfy, prosody, etc) by other installed apps.

Since f-droid exists, this allows for a halfway decently user-friendly-ish way to completely self host outside of relying on googles server's and zulip, for example, could offer the ability to receive notifications through it if there's an a unified push distributor available on the phone. It seems that there is at least awareness for this in the project [1].

But with google tightening the noose around alternative ways to install apps, who knows how long this will be even possible.

[0] https://unifiedpush.org/ [1] https://github.com/zulip/zulip-flutter/issues/1198

It would be nice if you could run separate instances of an app that were considered separate, and forking around a push notification key would make a lot of sense. Another reason I would like to do this is to be able to have different Discord accounts coexisting on the same device. (But, the idea of having some set of different Zulip instances is maybe the exact same use case, with better server-side support.)

Doesn't Element support unified push?

So you can use a custom push server

I don't think we've ever charged a friend group or other non-incorporated group of people a dime for self-hosted notifications.

For the community tier, you don't have to do anything up to 10 users.

If your server has more than 10 users, you fill out a brief form (https://github.com/zulip/zulip/blob/main/templates/corporate...). We work hard to consistently process these requests within a couple business days, and the vast majority of communities are approved for full sponsorship without further interaction.

(Large communities managed by a business are quoted nonzero but extremely discounted pricing for self-hosted notifications).

Regarding risk: I certainly won't blame you for feeling risk-averse given the history of the tech industry. I can tell you about some unusual choices we've intentionally made to minimize risk for our users:

- We eschewed VC funding. A big part of my motivation was that I felt that VC funding usually requires eventual enshittification. https://zulip.com/values/ talks more about this.

- Zulip has been 100% FOSS software for more than a decade.

- At the very beginning, we built a complete data import/export system that allows migrating between our Cloud hosting and self-hosting; we put a lot of care into maintaining it well.

I can't promise that we'll never have something to sell for self-hosting communities. For example, I could imagine offering a paid add-on for encrypted backups.

That said, I'd like to push back on the idea that charging businesses for a tool that's an important part of their daily work "breaks the seal". Organizations with a software budget should be happier to pay a fair price for ethical, user-first software from a friendly vendor than for a closed-source product from a megacorp. And Zulip's full-time development team should be able to make a living building ethical FOSS software.

Thanks for the response. I'll discuss it w/ my users.

> That said, I'd like to push back on the idea that charging businesses for a tool that's an important part of their daily work "breaks the seal". Organizations with a software budget should be happier to pay a fair price for ethical, user-first software from a friendly vendor than for a closed-source product from a megacorp. And Zulip's full-time development team should be able to make a living building ethical FOSS software.

I think you touched on the sort of thing I'm concerned about with your mention of enshittification, though I think you're probably right that VC funding is involved in most cases. It is good to know that you've been at it for a decade and have (apparently) built a sustainable business selling a product people like.

My concerns (which I hope are understandable) aside, I certainly support your right to charge money for what you've made, as I said here (https://news.ycombinator.com/item?id=46953048).

>Organizations with a software budget should be happier to pay a fair price for ethical, user-first software from a friendly vendor than for a closed-source product from a megacorp.

Yet we don’t pay for Linux, grep, vim, etc, etc. Why is your open source project the only one worthy of requiring payment?

IMO you should drop the doublespeak of claiming these are open source values while simultaneously charging money. It’s offensive to people who contribute to actual open source projects like matrix, clang, Linux, kubernetes, and on and on.

Huh? Where did he argue that you shouldn't pay for those things?

Grep and vim are a much smaller magnitude than Linux, so don't mix the two. And you do pay for Linux indirectly, it ain't written by some developer in their basements out of their good heart for a long time. It's written by Intel, Nvidia, cloud vendors' etc - full salaried employees. You just pay for it via hardware or cloud fees.

But to be honest your stance is extremely detached from reality. It's a huge privilege to be able to work on a hobby project, people tend to need food and a place to live, you know?

Grep and vim are obviously stand ins for a myriad of tools that together are much larger than Linux. And even Linux still has unpaid volunteers and even the majority corporate contributors are not that relevant to the discussion because none of them have control over the project to the degree that they could enshittify it.

> people tend to need food and a place to live, you know

That has never been enough reason to require that others support your business model. I for one don't need or want any more "products" in my life, especially ones that are or depend on services I can only get from a single vendor.

Etymologically, a product is a thing which is produced. But it's unpleasant to think about how the sausage gets made, so nobody wants to consider the goods in their lives as products. They want their Nikes to simply pop into existence before them.

Tools like grep and vim dangle by a thread based on volunteer work, and open-source maintainers are famously prone to burnout. Some tools survive by being very small—nobody's out there updating `ls` every month—but the only sustainable way to maintain a large piece of software is with a salaried workforce.

You may not want to interact with the systems that produce Zulip for you, but you should be suspicious of goods that hide their status as products. There's no such thing as a free lunch.

I worked on Linux, I know how it’s funded. And it’s not by charging people

How is it double-speak? The "free" part doesn't mean "free beer"...

Do you think clang, linux, and kubernetes could ever exist and survive in their current forms without the work of salaried developers? Volunteer maintenance is not sustainable; the long hours of unpaid labour are famously prone to causing burnout.

Free software is free as in speech, not free as in beer. If you want to save your cash, go use discord. If you're not paying, you're the product.

I use Zulip every day for the julia programming language (https://julialang.zulipchat.com).

I really like Zulip, and I'd like to migrate my friend-group onto it, but it probably won't happen. I think Zulip is just a bit too heavy-duty for a friend group chatting, and also lacks the visual polish that a lot of people want.

For now, my friends and I mostly just use Signal for group chats, which leaves a lot to be desired, but IMO is still just a better experience for our purposes than Zulip or Matrix.

That said, if you have friends who are keen to try things out, I would definitely recommend at least trying Zulip and see what you like and what you don't. It has a lot of really nice features and things to love.

Having interacted a fair amount with the Zulip devs over the years, and being an open-source product, I believe that they have no plans or intention of trying to fleece or milk self-hosted users or small communities.

Thanks for the input. I'm working on setting up an instance, so we'll see how it goes!

The federation of Matrix seems risky to me to the person self-hosting. I don’t want to host random people’s content. I’ve read some interesting articles about the design flaws of Matrix that led me to believe that it’s not a good option.

What is confusing to you about the community tier? It is basically describing any type of community of people who are not a for-profit business. Groups of friends, non-profits, volunteer groups, etc.

Zulip isn’t charging you anything unless you’re a business with more than 10 users and need push notifications, and that is still only $3.50/month/user if you don’t need more enterprisey things like SSO and compliance stuff.

> The federation of Matrix seems risky to me to the person self-hosting. I don’t want to host random people’s content. I’ve read some interesting articles about the design flaws of Matrix that led me to believe that it’s not a good option.

You can just not federate.

My experience with Matrix as a "discord replacement" at the scale of a few tens of people is that it works and is stable but is also jank and has a lot of features I don't really care about, federation being one of them. Hence my enthusiasm for a possible alternative.

> What is confusing to you about the community tier? It is basically describing any type of community of people who are not a for-profit business. Groups of friends, non-profits, volunteer groups, etc.

The part that isn't clear to me is how you prove to the Zulip people that you are worthy of being included in that tier. I'm certainly willing to write a few sentences explaining my use case, but the help page is light on specifics.

I'm pretty sure it's just an honor system similar to a lot of similar software.

Corporations tend to self-comply because they don't want to risk failed audits and lawsuits.

If you don't want to just random people's content, dont let random people use your server

Reply on news.ycombinator.com