Hacker News Reader

Hacker News

Comment by aldanor | Hacker News Reader

Full thread

aldanor

So, they've reduced 1.7M lines of code to 360K. Let's pause here for a moment:

- NumPy is 360K lines mostly C

- Postgres is around 2.1M of C

- Go 1.13 is around 1.5M of Go code

- Rust 1.37 is around 1.2M of Rust code iirc

- core llvm is around 3M of C++ iirc

...

A chat app is 1.7M. A chat app...

Makes me feel sorry for all the time spent/wasted by all those devs, many of whom are unquestionably brilliant and could've worked together on something truly awesome, big and useful. But we all know that doesn't pay the bills and so here we are.

View on HN

Honestly, a chat app that 1 billion+ use per month that includes payments, camera effects, extensive social integrations, stories, GIFs, reactions, games, polls, voice recording, calling, video chat, and works across all mobile platforms, with nine years of technical debt, that thousands of developers build simultaneously seems WAY more complicated to me than any of your examples.

Messenger may not have "HN clout" but the devs who work on it are having some of the greatest software impact in the world.

> that includes payments, camera effects, extensive social integrations, stories, GIFs, reactions, games, polls, voice recording, calling, video chat

To be fair, this is part of the problem ^

It's an inner-platform effect, certainly, but not one FB asked for. If they were allowed by the app-stores to factor their app into separate "plugins" that were auto-installed when a friend used them in a chat, the way iMessage does, they'd probably do that. But they're not allowed to have their own seamless plugin ecosystem, so into the monolith everything goes.

No, they specifically used Messenger as a way of pushing all of those different things on its massive install-base. Anything that distances them from the user in any way would compromise on that objective.

Facebook and all the rest benefit from developing closed ecosystems that others can't benefit from. It's not a coincidence that their systems are systems-within-systems.

Just because there are features that you don't use doesn't make it a problem.

In fact people who like stories and camera effects likely way outnumber the people that don't.

I interpreted “problem” as the bloat problem in a 1.7 mloc chat app.

Summarizing it as a chat app is probably the first miscommunication here.

I strongly doubt that. The number of people in my list of friends (400ish) that use stories is marginal.

The number of people that use camera effects as anything other than a gimmick is even smaller.

Maybe a different age group?

> my list of friends

I think that most of people of HN have friend groups which are wildly unrepresentative of world population at large. We're talking not even an average US or European citizen here - we're talking 1+ billion. People from developing countries who very often access internet only from a smartphone (they don't own a computer) and who almost don't visit plain http websites outside of the few social apps.

It's a completely different world out there.

If raw complexity was the fundamental issue, trimming it down to a quarter of the previous size wouldn't have been possible.

Some of the problems outlined, like having 40 different views of contacts, with unique code, seem like fairly clear organizational problems.

I'm on both sides of the fence on this one. On the one hand, it's people underestimating how much features are in the app that they've never seen, used or heard of. On the other, if that is the case then is their presence defensible?

I wonder if they could make apps more modular, and only load the binaries for e.g. payments on demand. I know Apple offers a system that does that for content chunks (e.g. levels and assets in games), and in a web-based application (e.g. react native) they could use bundling to remove it from the initial download.

But the chat app is only the client side, not the whole app entirely in itself. If we consider the lines of code in server side, there will be a lot more lines code.

it seem you don't have experience on large scale apps... and the people that make these type of comments almost never have had to work on large applications.....

Spotify had about 500k, lines of code in 2015, (not including external libraries).

We did make it in a feature based architecture, where there was a main container app (and a major core playback library) and for every major feature they had their own sub-project.... and there were 10+ teams contributing into it.

There were about 13 features at the beginning, (think, playlists, radio, shared ui libraries, etc...).

Each feature was about 10-20k lines of code in total. So, about 2-3 features equal a small standalone iOS app.

When I left, there were more than 80 features in the app. Some were essential, many were a/b tests, and some were probably just legacy/waiting to be removed.

80 features * 15k lines of code = 1.2mil loc, easily, and they are the equivalent of shipping 20 small apps standalone....

What are these features and why is just a stupid music app so large? : video, podcasts/shows, genius, etc.. etc.. etc.... To users just looks like another tab but almost every one of these major features could be its own standalone app for size and complexity.

I would love to read more about why large apps are so big.

I currently work on a project that is much larger than I would expect it to be, both in terms of codebase and people, and while I don't know who to jank, I also don't understand why it has to be so big.

Short answer: because it's way more feasible to remodel your kitchen than to tear down your entire house and rebuild it.

And then your bathroom. And then convert the garage. And then add an addition. And in 10 years you have this cobbled-together monstrosity

But your garage (the first part of what is now your house), was built with a weird kind of screw, that was only sold by one shop at the time, but was cheapest.

Now your entire house is built with those screws, but they’re not produced anymore so you’ve set up a small smelter next to your home where you’re making them yourself.

Unfortunately the quality is a good bit worse, so parts of your house start degrading faster and faster, but meanwhile you’re still building all new parts with your shitty screws.

Nobody wants to replace the screws because “It’s a proven method”, ignoring all signs to the contrary.

All new parts of the house are only as good as the worst part, the screws, and the moment anyone steps on the wrong plank the whole room comes crashing down. But nobody worries, next time they’ll just use two screws.

> I would love to read more about why large apps are so big.

Because nobody ever got promoted for not hiring people or for not writing code.

The problem is 10x as bad when you're an "IT company", i.e., writing code is your primary product.

Part of it also comes from the tendency of large orgs to ship their org structure.

It can oftentimes be worthwhile to have individual teams each fully "own" a piece of the product, and that also comes with the autonomy to choose to duplicate stuff that other teams might have already built rather than working with them and tweaking it to match their own specific new requirement.

This significantly reduces communication & coordination overhead for delivering new features, at the expense of consistency and code reuse, which can often be a worthwhile tradeoff at scale.

Interesting. I wonder if there's a corollary: "you will ship your org, so be careful how you structure your org."

https://en.wikipedia.org/wiki/Conway's_law

What language(s) does Spotify use for the backend? not the website or the mobile clients, but the backend itself.

Mostly Java.... some scala (some teams might use it), and potentially other languages. It built its own api lightweight java framework around 2014-2015 (called Apollo), which is actually pretty decent/nice to work with. There was a lot of protobuf usage early on, but most teams started transitioning to plain json.

So, each team runs its own servers, for the features it is developing/maintaining, and most of the api goes throw a common gateway, etc.. etc... don't want to give out more than this.

Some of the simple front end web pages (i.e. marketing websites, analytics) might even be on php/python/ruby. Some teams wanted to use javascript (Node), but those are not the critical main app, as java is pretty much required to be used for anything that affects the main app.

Also, if you are doing machine learning, then whatever it gets the job done (usually Python).

btw, I left few years ago, so things might have changed.... I know at least one team that wanted to use GO for some of their services....

Thanks for the insights.

I can't shake the feeling that the dominating factor in growth of LOC isn't the number of features per se, but the number of features worked on in parallel by separate teams.

Thank you for saying this. The overwhelming majority of comments on this thread are pedestrian - as if FBs engineers have been making decisions out of naivety, incompetence or self interest.

It might look that way in hindsight, but hindsight is easy. As is armchair theorizing about the competence of other developers.

Most people (including myself) will probably never see a >500K loc application or a project with >100 developers working on it. Biggest ones I've worked with had around 30-50 odd developers?

> as if FBs engineers have been making decisions out of naivety,

No.

> incompetence

Yes.

> or self interest

Yes.

Definitely the last one. They’re working for Facebook after all.

That said, I’m happy with React.

Turns out there's the same amount of code in the Spotify app as there is in the product I work on...

True though, if you want more features you need more code. Sure, loads of people on HN could build a 'chat app' in a lot less but if they kept it going for ten years (same age as our product) and kept on adding new features the lines would start to add up...

I thought this comment was making the opposite point, that it was impressive that Messenger was now only 360K LoC.

I honestly think it is. Compliers, DBMS, a library such as NumPy— they're all complex, but Messenger, even considering the features removed, is a very full-featured application. We're not talking IRC here. It has activity status, payments, stories, photo filters, audio messages, location sharing, video-calling, etc. It's actually pretty impressive that that fits in 360K LoC IMO.

I'm fairly certain that they also included third-party libraries in that 1.7M.

It seems you are correct:

> We also used quite a few of the OS libraries, including the JSON processing library, rather than building and storing our own libraries in the codebase.

Doesn't that mean they're not using third-party libraries?

The 1.7m was pre-rewrite and the quote about using os libraries was post-rewrite; it doesn't prove they counted the libraries in the 1.7m metric but it does imply they bundled more before.

> Makes me feel sorry for all the time spent/wasted by all those devs, many of whom are unquestionably brilliant and could've worked together on something truly awesome, big and useful

I'm really not a fan of this virtue shaming that people often tend to do as developers, where if you're not working to solve world peace then you're a waste of a developer.

Sometimes people just like getting paid? Sometimes people just don't really care, they just want a job or want to write code?

While I understand the sentiment, you are making an assumption here that all messenger is a chat app. It seems excessive ( and it is ), but you don't know what kind of heavy lifting is being done under the hood.

I am not really defending it, but number of code lines alone is not a good way to evaluate code.

All these programs have textual interfaces only (as far as I know).

Having a GUI doesn't justify 1 million+ LOCs, but just for a sanity check, pgAdmin 4 has ~260k LOCs, without counting its dependencies.

I have a set of code I use for my VR projects that I've been iterating on over the last five or so years. It includes a ton of not-VR stuff, too, just stuff that I tend to include in my projects. (aside: and I'm not trying to make it a "framework". It's just my own code and it's only publicly available in the sense that I don't charge for it, don't want to waste a private Github repo on it, and won't stop anyone from using it, but I'm also not going out of my way to teach them how to use it).

THAT code, my "all encompassing, personal repository for every VR app I'd ever make", is only 100K lines of code.

I once downloaded all of my dependencies and counted all of those lines and still only got to 200K. Well, not counting Unity. But at that point, you might as well start counting the LoC of the OS.

For what it's worth, GitHub private repos are now free so you don't have to sorry about wasting one.

Yes, I know, but I do also occasionally share code snippets out of it. If someone in a chatroom is asking how to do X, I'll just link to a source block in my repo. It's lazier than copy-pasting the code, but I also think it's good to let people see the broader context, maybe explore on their own a little.

I wouldn't mind if other people started using the code, but I've been burnt too many times during attempts to push my OSS projects over the years that I just don't want to put any actual effort into attracting people anymore.

It's a chat app that works very well even on the crappiest connections.

If it having 1.7M lines of code indicates anything it is that creating a "simple app that provides simple functionality over a network well" is hard.

Look at the garbage generated by Google - text messages and images on Google hangout are not guaranteed to arrive in order ( or even be sent in order ) unless you are on a great network.

Ordering messages and providing reliability does not require 1.7M lines of code.

Code wins arguments. FB messenger beats everything, Whatsapp, Wire, Signal, WeChat, Telegram included, in the suckiest network connections.

Repeating a point without adding proof doesn't make your point more valid. When people say "code wins arguments", they're referring to using source code to prove something, not "it wins because it has more source code" (???).

People say "Code wins arguments" because written code that works demonstrates the winning argument against the code that is either not written or does not work.

FB messenger exists and works on crappy connections. Google hangout exists and messes up on crappy connections. Telegram exists and messes up on crappy connections ( see the issues it has in Africa ). Signal exists and does not work well even on non-crappy connections. Whatsapp exists and sort of works on crappy connections. Matrix exists and sucks on crappy connections.

What we need is something that exists and works on crappy connections that has less LOC than FB messenger to demonstrate that it is possible to handle crappy connections and modern messaging in much fewer lines of code.

> written code that works demonstrates the winning argument against the code that is either not written or does not work.

That's why your comment is nonsensical, if not downright contradictory noise.

> Ordering messages and providing reliability does not require 1.7M lines of code.

This has been demonstrated ad nauseum a priori with loads of toy chat apps that reconnect and retain state. It's not really a challenge. EVERYTHING else crammed into it, should be the reasoning for 1.7M lines of code, but as they demonstrated TO THEMSELVES AND EVERYONE, it wasn't (360k now)

> This has been demonstrated ad nauseum a priori with loads of toy chat apps that reconnect and retain state. It's not really a challenge. EVERYTHING else crammed into it, should be the reasoning for 1.7M lines of code, but as they demonstrated TO THEMSELVES AND EVERYONE, it wasn't (360k now)

Yes and they figured out what was needed after first writing a 1.7M line of code thing that worked. That's the process of reiteration and a process of initial LOC bloat before LOC stripping.

Without that process by thinking it has all been demonstrated ad nauseum a priori with loads of toy chat apps that reconnect and retain state you end up with deciding that if Friday was Feb 28th then Monday is certainly March 3rd because date/time handling is simple and end up being Robinhood, down for a day.

If FB Messenger beat Whatsapp they wouldn’t have had to buy it.

Facebook bought Whatsapp for the social graph, not the messenger technology

Yes, exactly. The social graph they couldn’t acquire organically by having the better product.

> It's a chat app that works very well even on the crappiest connections

Does it? I routinely use Telegram, Line and Messenger. Messenger is without contest the one that gives the most trouble.

I've been using chat apps at least since ICQ (1996) and sometimes I feel the user experience has declined.

That being said, I'm sure that a lot of progress have been made and that there's a lot going on behind Messenger in terms of security and scalability. I'd be curious to know what's hidden between the millions LOC.

In my experience in places where connectivity sucks (barely able to get data handoffs between stations, sticky wifi, etc) messenger kicks ass while the other applications hick-up big time.

> I've been using chat apps at least since ICQ (1996) and sometimes I feel the user experience has declined.

You never used ICQ over a spotty 3G connection, though, did you?

I have, with an OpenVPN-over-UDP tunnel as a workaround. If they included Wireguard (or better yet, just used encrypted UDP) it would still be smaller (and probably more reliable).

Introduce random dropouts, crazy jitter and random IP switches and you get the emulation of what a crappy network looks like.

I don't know if this is what you were referring to, but for everyone else: this is actually super easy to add with Wireguard because Wireguard just uses Linux networking you can do things like:

  $ tc qdisc change dev eth0 root netem loss 0.3% 25%

(Lose 0.3% packets on average, but each packet is 25% correlated with the last packet loss, to produce bursts.)

  $ tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal

(Add mu=100ms, sigma=20ms normally distributed latency.)

Not a lot of 3G coverage back in 1996, honestly.

You don’t need 1.7m LOC to build a simple app that works well over different network conditions.

Speaking of garbage, looks like most of the 1.7m LOC on Messenger was actually garbage, now that they’ve literally thrown most of away.

> You don’t need 1.7m LOC to build a simple app that works well over different network conditions.

Also known as:

You can do dropbox with FTP.

If Friday is Feb 28th, then Monday is March 3rd. There's no need for those bloated date libraries.

> Speaking of garbage, looks like most of the 1.7m LOC on Messenger was actually garbage, now that they’ve literally thrown most of away.

Certainly. It worked before the change. So it is given that 1.7m of "garbage" provided a working product.

How many lines of code it took Google to make Hangout not get confused by image uploads on crappy connections? Unknown, because it is still confused.

How many lines of code it took Signal not to delay messages when a recipient is on a crappy connection? Unknown, because it still does that.

> It's a chat app that works very well even on the crappiest connections.

That's a joke, right? On a crappy internet connection (e.g. using O2 as carrier on the train cologne-hamburg you get ~12s ping, 30kbps, >90% packets dropped, via edge) most chat apps stop working, but Facebook Messenger is (except for Riot.im and Discord) probably the one that works worst.

Telegram works better, even IRC via Quassel/Quasseldroid works significantly better, and especially WhatsApp works so well you don't even notice the connection issues.

Facebook Messenger doesn't even properly start on such crappy connections.

Rumors are the FB iOS app have more lines of code than iOS itself

I would not be surprised if this is the case. I am fairly sure the corollary is true: Facebook has more people working on iOS than Apple does.

The only other application in your list is Postgres.

NumPy is a library. The other three are compilers. Why are you comparing these with a GUI-based chat application?

This would be a big part of this you know:

> [...] We accomplished this by using the native OS wherever possible [...]

I wonder how many LoC for Whatsapp, Which is comparable to Messenger. And what is it written in.

This "chat app" is used by more than a billion people in the world.

Number of users has little correlation with number of lines of code.

Sometimes making something simple is harder than making something complex

Reply on news.ycombinator.com