Superpack: Pushing the limits of compression in Facebook’s mobile apps | Hacker News Reader

Superpack: Pushing the limits of compression in Facebook’s mobile apps

engineering.fb.com

91 pointsdetaro21 comments

The fact that Facebook, something that really shouldn't even need an app, requires this kind of compression seems like more of an unintentional commentary on how bloated their terrible app is than anything else.

Yeah, and I think it really shows that WhatsApp, which wasn't developped by Facebook originally, doesn't benefits from the compression. The app seems very optimized originally, but maybe I'm wrong.

Also see the stark contrast with their Facebook Lite app. Makes you wonder what difference all those megabytes in the full app actually make...

Append-only development

I don't even have the app on my phone, just a shortcut to the mobile site, and it does everything I need.

How is Messenger? Last time I remember, they were rendering it useless on the mobile site in favour of downloading an app (more data).

I haven't used Facebook in a long time, though.

If you request the “desktop site” it looks great on a phone, not really tiny, with adaptive layout. And you get this older version of messenger that just shows the text without all the crazy features.

I have the android app called Lite, which is FB's Messenger app for low spec phones and doesn't have all the spyware aspects.

Yep, this 100%. I worked on Facebook iOS app perf/reliability/efficiency for a couple of years. The company is structurally and politically incapable of making a lean app (thousands of engineers have their performance reviews tied to launching new features, not deleting old ones). So they rely on a relatively small number of engineers making heroic technical fixes in order to make it possible for the app to even boot. Similar situation on Android.

The status quo is clearly dysfunctional but I would be surprised if the company ever figures out how to fix it.

A few years back the FB app reached Android’s limit for the number of classes defined in the codebase (2^16), so the FB engineers wrote a hack that ran before the app to somehow bypass this limit. Apparently they don’t delete anything and just keep adding stuff to the codebase.

What kind of app needs 65536 classes?

https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A...

Found the old explanation, it seems I had misremembered some details but it’s still super insane.

The method limit is the more constraining one.

Also, Google eventually lifted this limit on newer versions of Android because other apps were running into the same issue. It's not just FB :)

Bear in mind that number also includes classes declared in dependencies.

Drive-by associations of the morality of FB’s business (which I personally found objectionable enough to leave a few MM on the table over) and the caliber of the engineers who work there are suspect.

The folks are every bit as good as the hiring practices would indicate. These people meet a problem, and in the average case haven’t read the relevant textbook. So they read it, and solve the problem. Sometimes they invent something new. I personally had LSM in production before Dean et al.

Criticize the business all you want, I might even back you, but the typical HN smartass has no idea the world of hurt they would be in if they had to code against an FB new hire.

The comment you’re responding to didn’t mention Facebook’s hiring practices at all.

> The folks are every bit as good as the hiring practices would indicate. These people meet a problem, and in the average case haven’t read the relevant textbook. So they read it, and solve the problem. Sometimes they invent something new.

Give me a break. There are some really bright people, but a lot are mediocre, just like at any other company with tens of thousands of employees.

I’m not sure when you worked there. The only way I can make sense of your post is if it was a very long time ago, or you were in a very unusual team.

So basically the compressor is aware of opcode structure so uses multiple data streams and contextual encoding for operands. The article doesn't say much about the use of SMT solver, but my guess is that it transforms opcodes in a way that they run exactly identically as the original (verified by the solver) but compress better. Well-known stuffs in the data compression community, but also hard to execute efficiently.

Now if Google could do some optimisation of their apps on the Apple App Store, that'd be great. Currently, the latest Gmail update sits at ~328MB, the total of all nine Google apps on my phone sits at 2226MB (averaging ~247MB). It's ridiculous.

important: "We may someday consider open sourcing Superpack."

I hope they'll push out a paper as well

Didnt Facebook open-source their other stuff? I don't think they were very good with outside contributions, it was just "here's the code".

There’s a spectrum. Some stuff is explicitly intended for outside consumption and a lot of effort goes into documentation, marketing, polish, etc. (e.g. PyTorch). Some other stuff like you said is just thrown over the wall.

Reply on news.ycombinator.com