Hacker News Reader

Comment by kondro | Hacker News Reader

I don’t mean to discount the work done here, but why would a dev team build this themselves these days? There are many practically infinitely scalable services that do just this with various levels of sophistication.

From the simple AWS SNS ($0.50/million messages) or Google Firebase (free) solutions that do multi-device push messaging whilst managing keys, redelivery and delivery responses to the more managed services like Urban Airship and OneSignal.

The actual Android delivery component has to end up going through Firebase anyway.

The only part of this solution SNS and Firebase don’t provide are the user fan-out functionality.

Surely there are other more important features delivery business value to be done rather than re-inventing (and then scaling and supporting) the delivery of mobile notifications.

View on HN

I cofounded OneSignal for this reason. We were previously a game studio that built an in-house solution. It was a huge maintenance headache and a distraction. Eventually, we realized that so many companies were one-off push implementations and decided to focus our efforts on building a really great third-party solution.

Firebase and SNS have since entered this market as well. I'm obviously biased, but I honestly can't recommend them. They work for some of the basics, but there's been very little innovation in either product and occasional service issues that go unsolved for a long time. They also don't make any money from them, which should be a concern for anyone that plans to use these services for such a core part of their business.

Pretty soon your app is just a bunch of proprietary cloud provider products glued together and now there’s vendor lock-in and much less flexibility.

You might want to throttle messages based on user settings or say ML models to avoid too many annoying messages. For critical stuff you might want to switch over to email or SMS.

These are startups that have the funding to do this kind of thing. This is a core part of the stack for any such company.

If that's your concern, lean on Firebase.

You have to use Firebase to deliver messages to Android anyway, having it deliver to APNS as well you get for free with that implementation.

Maybe you do want to do those things, but this implementation doesn't, nor do they mention the intention to do that. But if you did, SNS also provides implementations for delivering SMS and Email (preferably through SES).

Startups shouldn't waste their money on this type of thing, especially if you consider most startups will fail. Vendor lock-in is a pretty stupid thing to worry about for an early-stage start-up (and I argue it's stupid to worry about long-term too if that vendor is AWS, Google or Azure who are all very competitive with each other).

Having a middleware to abstract firebases API from the services makes sense if just to minimize the disruption the next time Google changes it. I've always left tokens up to the callers to sort out before calling the notification service, but I see the appeal of centralizing it if you have a lot of small products. I agree there is no longer any good reason to directly target APNS.

Nothing in the article really said how they got to millions an hour. In my experience the notifications are cheap enough to send that that figure is easy to obtain without any special effort on the push side. It's also embarrassingly parallel/easy to scale. How everything else scales around that (recipient filtering, reporting) is the interesting part.

I’m not disagreeing just want to point out that this company has raised 3 billion dollars in total. So imo it’s worth it for them they have the money. I wouldn’t do all of this starting out.

Gojek is way past the “most startups will fail” stage though.

Efficiently handling million of events per second (not per hour) using whatever cloud PaaS is a no brainer these days. The lock-in is worth it when you can reliably move that fast.

If you're really worried about lock-in, most event based PaaS support running containers. Moving between clouds would then just be the glue-work around your containers.

Honestly, how many companies DON'T rely on some proprietary cloud provider at some point?

* SMS - You can't just interface directly with some cell company (which is proprietary anyways). * EMAIL - Have you tried to send your own emails in the past decade? Just use a provider. * Push Notifications - Have to use Firebase for Android anyways.

----

Your core stuff should limit proprietary stuff. Edges are going to change over time no matter what, might as well use stuff that makes you faster now.

> * SMS - You can't just interface directly with some cell company (which is proprietary anyways).

Wut? People don't even know anymore that you can use cellphones to send SMS?! (And no, that's not the only option, SS7 works just fine as well.)

> * EMAIL - Have you tried to send your own emails in the past decade? Just use a provider.

Have you? Talking nonsense does not make expertise ...

Email in particular. Sending your own isn't the problem. Managing your reputation, deliverability, etc just isn't fun to deal with.

I don't think vendor lock-in is what small startup should worry about.

They are not small though. Gojek is big in Indonesia specifically

Gojek is a super app in Indonesia serving a population of 264 million. They can afford this level of engineering and are required to do it, to stay competitive. They have millions of motorcycle driver (called gojeks) for whom they are trying to find jobs while idling waiting for passengers to transport.

If uber built these no one will question them why don't they use some off the shelf solution. Gojek is a google cloud customer and under their startup program - surge. So I am sure they are aware of the solution proposed.

They don't call themselves technology company but company which helps gojek drivers to find jobs and earn extra. In the process they became a food delivery, transportation, payments, courier company. In spite of this they still focus on their primary objective to find jobs for gojek and let them earn extra for better living.

Before Gojek app most of the drivers were just idling waiting for passengers, now with a $25 Kai OS feature phone [1], they can just receive a message and earn instead of just idling, they don't have to use iPhone or Android or some heavy OS. Indeed based on social survey some of the drivers could increase there earning 3-4 times based on how willing are they to take up additional jobs.

I believe the beauty of Gojek lies in its model to focus on finding jobs and developing solution to help those motorcycle drivers using technology. Although they extended it to cars. I am intrigued by their idea since its not a copy of Uber, but a solution to a problem local to Indonesia, which can extend to any country in the world finding jobs to do while idling and earn more.

[1] https://en.wikipedia.org/wiki/KaiOS

I don't think the other answers here question the impact Gojek has on millions of people.

> If uber built these no one will question them why don't they use some off the shelf solution.

If it's over-engineered, they probably would.

Disclaimer: I work at Gojek.

> They can afford this level of engineering and are required to do it, to stay competitive

> If uber built these no one will question them why don't they use some off the shelf solution

Uber and Gojek don't have the same talent pool so we can't compare them as Apple-to-Apple.

It is a risky move for Gojek and not for Uber because of the talent pool in their respective area.

Why would you think Gojek will not have a talent and cannot manage engineering?

If you have used their app and done a review of their platform comparing it with Uber, can you please share.

Indeed Uber did try to enter Indonesia they failed, they were out of most of South East and East Asia, because they don't have enough engineering talent to build a system for those specific countries, local companies like Grab in Singapore, Gojek in Indonesia, Didi in China beat them. So why would you think those companies do not have a talent to build systems better suited to their own environment than Uber.

You guys missing the bigger picture. Uber didn't fail in Southeast Asia. The competition was so steep that Uber and Grab were losing a lot of money through aggressive marketing. I know how aggressive they were as I'm from Malaysia. So, Uber and Grab came to an agreement, where Uber sold off their business to Grab for 27.5% stake and a seat on the board. It have nothing to do with talent. It's pure business. Instead of lose-lose situation, they created a win-win situation for both.

> Indeed Uber did try to enter Indonesia they failed, they were out of most of South East and East Asia, because they don't have enough engineering talent to build a system

Because they don't understand the local market, period. Nothing more than that. It has zero relationship with Uber not able to hire engineering talent. They simply don't know the market.

If you put the smartest engineers in the room but they don't know the business domain, they will fail from Day-1. I think people put Engineers over everything else where in majority of the situations, business-domain-knowledge trumps engineering excellence.

That is one reason. The other reason is that Local will choose Local (China with tons of Chinese only service). Indonesian President, Mr. Joko Widodo, for sure will promote Gojek more than Uber.

> So why would you think those companies do not have a talent to build systems better suited to their own environment

You have to understand the context here.

Why do you think Gojek opened a huge "lab" in India?

Why do you think Grab opened a "lab" in Seattle and hired Steve Yegge?

You may not realized this but they are not the only South East Asia "Unicorn"/"Decacorn" who opened R&D Labs outside their "homebase".

I can't share much of the details why these companies (_and_ a few others) decided to do so but feel free to guess a bit here and there.

> Because they don't understand the local market, period.

Uber couldn't iterate the platform and technology according to local market requirements and conditions, engineering is one of the reason for failure. They then tried with money and didn't work either. It's a combination of business and technology platform due to which Uber failed.

If you prefer to live in a bubble its fine, there is no better engineering talent anywhere in this world except Uber. But ground truths are already there in the markets from which they retreated.

Gojek opened centers in India because it's cheaper than Singapore.

Grab costs in Singapore and Malaysia is higher too compared to India.

So like any other USA firm which has opened centre in India to save costs, they did too.

Eh... Grab opened an R&D lab in Seattle and hire Google folks. That's breaking the wallet big time in terms of employee compensation.

Grab isn't the only one from SE Asia that opened the lab in North America. There are at least 2 more from Indonesia that opted North America instead of Singapore/Malaysia/India.

Can't share details other than "talent pools just not there".

Feel free to disagree with the execs.

> Grab opened an R&D lab in Seattle and hire Google folks. That's breaking the wallet big time in terms of employee compensation.

Sure they will open in Seattle as it's 30% or more cheaper than Singapore. Grab's largest R&D team is in Singapore. They did bring in talent from around the world here. But as the cost in Singapore is very high, to get an expat is very expensive. [1]

So it does make sense to open in cheaper location an R&D, it also shows they have engineering talent from around the globe.

[1] https://www.businessinsider.sg/most-expensive-cities-in-the-...

Cheaper in Seattle? I don't have the exact number how expensive it is to hire folks in Singapore but I can tell you this: those google folks cost at least 150k USD base without bonus and RSU. Someone like Steve Yegge will have total compensation north of $500k USD at Google so Grab should at least pay him 700k-1M USD.

Google Fresh Grad total compensation should be between 150k - 300k usd in Mountain View. Fresh grad. Seattle usually cost 10-15% less total compensation compare to Mountain View.

Heck even recent Microsoft offer for fresh grad base:110k, bonus 10-20%, and rsu. That's MSFT, not FAANG. FAANG will give you signon bonus on top of that ranging from 50k-100k.

Keep in mind Grab RSU is paper money at the moment.

I highly doubt Singapore engineers, given the same level, make that much. Probably only the top 0.5%.

I don't see a good discussion as you still think that these folks don't come to North America and open R&D lab because it's cheaper than Singapore since that logic does not make any sense out of the gate: India is way cheaper, why not grow there and don't even bother opening a lab in Seattle. You do know that it is damn hard to grow an R&D Lab in the USA due to regulations and immigration policies right.

In order to avoid repeating the facts on the field, I would suggest you to reach out to the executives in those companies and ask why they do so as I have done before to avoid confusion, assumptions, and guessing games.

One big factor is being free from bundling proprietary, vendor-specific libraries that may also leak user's information or otherwise increase the security risk footprint of your app. Having control over the services that make up the core of your business, as opposed to outsourcing and relying on 3rd-parties, can be very freeing and also cost effective in the long run.

Once you're hosting on AWS you've pretty much already crossed that particular bridge.

Engineering Pride (and exercise for future employment).

While this might sound condescending but hear me out:

- if you keep doing the same thing like everybody else out there, gluing APIs from cloud-vendor, you won't stand out.

- if you're a company of _that_ size (Uber, Gojek), you gotta do something big because the name/brand is associated with "Engineering" excellence

- if you're a leader (or in management), would you tell your employee to "suck it up" and do the boring stuff like everybody else? or would you empower them to do more? hype them up? this is a strategy for keeping them happy too.

I don't begrudge that. That's how companies evolve: they start small and scrappy, they grow very fast, and they're at the level where "doing simple stuff" won't get you promotion or you won't have something cool to show during the interview for your next job.

It's work dynamic.

Did you read the post? They're using Firebase messaging, but that only covers a part of their workflow.

They’re only using Firebase for Android delivery (which is mandatory anyway). But it is just as happy to deliver to APNS at $0 as well with near infinite scalability.

The actual delivery component of their messaging system is unnecessary.

``` Surely there are other more important features delivery business value to be done rather than re-inventing (and then scaling and supporting) the delivery of mobile notifications. ``` the business value the service provided is that this allow the engineering organization to scale. Gojek has couple hundred engineers, split across multiple teams, where each teams own multiple services/products/applications, each with their own development cycle. Having a single abstraction provided as a service helps to mitigate the risks involved in changes to implementation details and external dependencies (e.g. GCM deprecation) since it abstracts it away completely.

It's not hard to do what they did, so I'm not surprised they did it themselves. It's just simple async queue processing, and doesn't really need a lot of sophistication, especially at just 1M notifications per hour. Now if they need to process a few orders of magnitude more, then AWS SNS would probably be a good next step.

Reply on news.ycombinator.com