Introducing the LineageOS SDK | Hacker News Reader

Introducing the LineageOS SDK

lineageos.org

200 pointsigneo676105 comments

While we're on the topic, I was surprised to find just how poorly maintained TWRP—the standard custom recovery distribution for Android—is. Decryption of encrypted partitions is just broken many devices, for example. Issues pile up across the respective repositories as designated "maintainers" ignore all inquiries. Doing the work to fix something yourself requires piecing together the arcane knowledge of TWRP's build process from multiple tutorial posts on forums in various states of obsolescence. Then you need to understand from where whoever did the original work for your device pulled the library binaries that are inevitably sprinkled across its repository under vague commit messages and no other documentation. It seems no effort was put into securing the project's continuity in the long term.

> It seems no effort was put into securing the project's continuity in the long term.

But it's no small challenge, right? I'd suppose that regression testing these features against a bunch of devices requires lots of human interaction or complex test scenario descriptions.

I meant more along the lines of documenting your build process or noting where any of the binary blobs were sourced from. Simple things that make coming in as a new contributor viable. As it is now, basic, essential information vanishes along with each maintainer who loses interest in the project.

> As it is now, basic, essential information vanishes along with each maintainer who loses interest in the project.

Binary blobs come from the OEM's system image, I can't recall when that isn't the case.

The question of what version of the OEM system image to use isn't always super clear. I've definitely seen situations where rom devs will mix and match whatever it takes to get their desired results.

In which case the instructions start with, "Go to the following URL, download the file named foo, and use the following command to extract bar and baz." But write those instructions!

The entire android rom development ecosystem feels that way to me. Lineage is a bit of an exception with fairly solid and well documented build system, but for by and large, XDA feels a lot like the 90s warez/cracking scene. Lots of really smart people shipping hand crafted blobs on forums.

The documentation or lack of it thereof doesn't help with that. I can't think of a single google query that got an answer when I was building LOS for my own device. Another thing that doesn't help is that it takes enormous amount of times to make a clean build and quite often some final touches to the installation file have to be made to make them compatible with TWRP etc. My point is that no volunteer has time to document stuff and because of that no volunteer has time to document stuff repeat ad infinitum.

I was in a similar position a few months back. Was trying to simply build Lineage with MicroG and Xposed baked in for personal use and was essentially shouted out of the LineageOS-dev IRC channel and told to write a backup script and to flash the ZIP files instead of compiling it. They seemed outright hostile to fixing this situation, or maybe I was shouted out of the channel by some randos.

Yeah, LineageOS devs actively disapprove/hate/censor anything MicroG and Xposed related, don't EVER mention those things if you want any "official" help. I find it appalling, but too few care about it to backslash so that it had an effect.

Yes, my admittedly limited exposure to Android has been negatively tinted due to this. Rooting and installing stock Android on a first gen Kindle Fire and later on, a Google Pixel C tablet are both sitting high in my list of most frustrating technical endeavors despite being a software dev and enthusiastically embracing technology for the past two decades of my life. I was shocked to find that the process was only moderately more straightforward on first party android devices.

If I ever buy another android device I’m going to treat it like an iOS device and not even bother with the tinkering aspect. It’s too much of a mess to be enjoyable for me.

Seconded. I don't wish to denigrate the participants - I've benefited from their binaries, code and support many times - but common practices in the community feel _really_ dated.

I was involved in the original launch of TWRP and maintained a couple devices. Though I haven't been very involved in a few years, so I wouldn't say I speak for the team... but...

A big struggle is that things vary so much device to device.

General TWRP documentation can be provided by the core maintainers, but each device really needs a maintainer and there's no system of ensuring that each maintainer is documenting with the same rigor.

After some time a device will fall off and no longer be considered officially maintained (See TF101: https://twrp.me/asus/asustransformerTF101.html which I used to maintain). There's not much that the project can do about that.

If you've got suggestions or would like to help get the documentation into better shape, I'm sure folks in #twrp on freenode would be happy to chat.

At the very least, bold that "no longer supported" text and move it up the page.

Reminds me of tooling around OpenStreetMaps circa 2012. Not sure if things have gotten better, but back then if you wanted to run something like a tile server, you had to figure out how to build a behemoth C++ project that had no clear build process and absolutely no integration with any OS (as in no init files, etc.). The little documentation that did exist was in German. It was a rough time.

There's some better documentation now:

https://switch2osm.org/

I'd be very interested in seeing a Lineageos that does not include Google's proprietary and closed source Google Play Services. I understand you can download Lineageos without Google Play Services however a hundred different things you wouldn't expect the os depends on will stop working including GPS and even notifications.

I think it should be an ambition of the project, while remaining open source, to fill those holes. There are open source workarounds in the wild but they are not so easy to use. Google works against them with seemingly every update to android, and Lineageos helps Google by refusing to support signature spoofing. The mechanism by which use of alternatives is even possible.

Lineageos says that it supports an experience free from Google Play Services but in practice use of Lineageos without it requires a custom build of the operating system. Which is too bad, I think that would be a reasonably large portion of the potential user base.

I wonder if CopperheadOS would fit the bill?

[deleted]

> Copperhead is an information security firm located in Toronto, Canada, specializing in protecting mobile data and devices through security assessments and secure endpoint deployments.

Do you mean that CopperheadOS is derived from the Android Open Source Project (AOSP), which is developed by Google?

https://lineage.microg.org/

MicroG has been picking up steam. They're doing OTA updates of Lineage + MicroG now.

Up until now I wasn't aware microG was doing their own builds for Lineageos, when did that start? I'm quite happy to see that, it'll help a lot of people. I'd still rather see a world where custom builds aren't necessary.

Not that long ago. I use it on my Nexus 6 (shamu) but I also use yalp to get apps from the play store.

Here is a relevant discussion: https://news.ycombinator.com/item?id=15617615

I use this on my Nexus 6P and it's surprisingly great! The only app that I really need that I have problems with is Lyft, but other than that everything I need works smoothly. I get as many apps as I can from F-Droid and use Yalp Store to get the rest. Preferring F-Droid apps also means I have fewer apps that expect Google Play Services.

I so desperately want to use MicroG, but keep having to go back to the official framework. The handful of apps I use that rely on Play (Lyft, Transit App, One Bus Away) all insist the Play Framework needs to be updated.

The main problem was (IIRC) that other services explicitally refer to google play services, and to circumvent that they would need to disable a security check, which is against the project policy (there were discussion whether it was a political or technical choice).

Lineage builds usually come per default without Google services or apps. It's your decision whether or not you install them. Of course, lots of 3rd party apps do rely on the Google services and won't work without. There is no good workaround for that short of re-implementing the Google APIs yourself or trying to use microG.

A lot of first party apps and basic os functionality also relies on Google Play Services. As Google have been building more and more into the closed source portion of their operating system. They are attempting to create a foothold where the operating system doesn't work without it.

Hmm, do you have any examples of this? I'm curious as I've been running LineageOS without Play Services... seems to work quite well, but maybe I'm missing something?

Do you use whatsapp or any similar messenger app? If so, do you receive notifications when the app is not running (foreground or background)

Play services is responsible for connectivity with Google's servers which are used to distribute notifications to the device.

Play services is probably also responsible for backing up the App's data to Google servers (not sure about this one)

Those aren't first party apps.

The AOSP keyboard lacks functionality when compared to Google keyboard by default: there is no way to swipe on the keyboard to write words, and you cannot input emojis by writing their description.

I'm also wondering what basic OS functionality is missing?

I've been running LineageOS 14.1 on a Sony Xperia tablet for several months without Google Play Services or MicroG and am not noticing any missing basic functionality. I do tend towards open source apps and away from Google stuff but I'm by no means strict about it. I use Yalp and F-Droid to install apps.

Some of the apps I have installed (eg. SnapSeed) complain about missing Google Play Services and say they won't function but I ignore the warning and they seem to operate correctly... if there's some missing functionality I've yet to encounter it, I'm happily editing and exporting photos.

I'm certain that some apps would fail to function without Google Play Services, just that I don't happen to have fallen foul of any of them. My usage is probably not hugely standard but I use modern browsers (Chromium/FF), video/audio players (VLC/Black Player/Spotify), K9 for email, browse and edit RAW photos... and jump into Debian Stretch via Linux Deploy + Xserver XSDL where I often do web dev work with tmux/vim and proper desktop Chrome/FF including Developer Tools. I'm using a Bluetooth keyboard and speakers... can't really think what basic things I might be missing?

EDIT: (this is just the standard install of LineageOS btw - which doesn't come with Google Play Services, you have to install them separately - I just skipped that step)

> Xserver XSDL

How's that work for you? I've tried it but it seemed a little rough.

> It's your decision whether or not you install them.

FYI Even without GAPPs lineageOS contacts Google with IMHO way too much information, for example when you connect to a WiFi network a request to Google is made.

that's for internet connectivity checking, and it's an anonymous http request (no cookies). iOS/windows does it as well (not sure about linux), and I can't see how you can implement such a feature without phoning home to some server.

www.example.com

I don't think you can ship millions of devices that randomly hit on a URL that you don't control or own.

Example.com is for examples (so that if you copy & paste my code, you don't do weird things). I am not convinced that example.com would appreciate the worlds phones to use it at a whim..

(Plus, I'm not sure that's possible without controlling the domain. Not sure if the current ways to detect a capture portal are satisfied by a random 200 OK or actually _check the content_ of the reply. Which needs you to control said content)

GNU/Linux doesn't do anything like that by default unless you where to write a script to.

I honestly don't see a good reason to do it although one of my friends pointed out that using TOR is probably one of the better ways to check for this since it won't rely on a specific server.

Well, GNU/Linux itself is not an OS, but Ubuntu and Fedora do enable that check in NetworkManager by default, whereas Debian doesn't (for privacy reasons).

Linux doesn't include a desktop or browser by default.

Firefox will make these requests as does Google Chrome and Chromium. NetworkManager will do them too and various GUI tools for network configuration will try it.

Otherwise Captive Portals would simply break your internet without notice.

No cookies but an user agent and your IP, so basically your rough location in the world.

I am on a plain LineageOs, no play services, apps installed directly via their APKs.

GPS & maps: Google Maps and Open Street Maps both work normally.

IM: Whatsapp, Telegram, Skype work normally (but I suppose they make use of a custom notification system)

Browser: Firefox seems not to care about play services

Media: Vlc, Netflix, MX Player are all fine.

"co-founder"/one of many project leads here. AMA?

Non programmer here. What cool new stuff can the apps do on my phone with the new APIs?

Also, why do you think there is such dearth of custom phone OSs unlike the linux scene?

There's a list of API's here: https://lineageos.github.io/android_lineage-sdk/reference/li....

I believe this update was mostly around styles, "Change the system accent to some color based on some event" was listed in the blog post earlier, along with night mode.

FWIW I found the bug reporting schedule really off-putting.

What would it take to resolve the unlocked bootloader vector?

I've been interested in LineageOS, but not enough to carry around a phone with an unlocked bootloader.

Buy a Pixel which supports running custom images with a locked bootloader and verified boot.

This is actually a problem that bothers me. Most of the recoveries also ship with absolutely no security (see: twrp and the "I can just format your phone by booting into it"). I don't really have a good answer though - if you lock a bootloader, even with a secured recovery image, and that image breaks, you're stuck forever. Without a huge QA effort (that doesn't work for an open source project like this), I don't think there's much we could do sadly :( In actual OEM land they do a build and test it for weeks/months before releasing it, we just...build things and ship them.

Can you clarify for me? Why might the image break after I've locked it? Why can't I just unlock it again? What are the consequences of this? I'd like to run Lineage but so far I'm unclear of the unanticipated negatives.

By default, after an update, Android installs a new recovery image during the first boot of that update. We currently have this disabled. If you relock the only way to ever update your recovery image is to unlock (wiping the device & losing encryption keys), flash a new image, and relock. This would need to happen for any discovered kernel exploit (read: about once a month) to avoid having an insecure device.

If the update feature's enabled, there's a chance that after an upgrade, recovery gets updated to a version that no longer boots. We don't QA every build like you see from an actual OEM, they're provided as nightly builds (on a weekly basis). One bug and your expensive device is now a brick.

Thank you for LineageOS!

Are there plans to do a yearly or fixed release schedule?

Nope. Bunch of volunteers, things will be released when they're ready. Weekly builds have been working well so far though!

So this is like GApps, but FOSS (I assume?)? It's missing a license file I think

No, this is not at all like gapps, you're thinking of the 'microg' stuff?

Not at all. It's just a java library you can use in an android app to do additional things on devices running lineage.

Which devices do you recommend buying today for the best user experience installing and using LineageOS? And which manufacturers are being most helpful to the project?

stats.lineageos.org exists. Bit basic ('big data' problems, storing ~1.7 million anonymized checkins per day in mongo...isn't the best idea).

Personally I try to stick to something with an unlocked bootloader. That was the Nexus line, now Pixels (except a/b updates are still a bit rough under lineage)

Do phones still have bad camera performance when using Lineage? I'll need to move two Nexus 5x devices when Google drops support.

Not a question but a compliment for LineageOS: thank you for this project. It has brought to life and vastly improved over the stock ROM at least 4 devices that I use. Please keep up the great work.

Please write a proper "About" page for the site, because the current one is trying to be cute/minimal, but isn't helpful.

"Please see Wikipedia" under a dictionary excerpt looks like it's sending me to learn what the word means.

PR's welcome :P

Thanks for the AMA offer!

I'm currently working to bring my old Galaxy S2 back to life, but I have the Sprint-exclusive version of the phone: the D710, not the international i9100 that everyone else has.

Are there any guides on how to port LineageOS for the i9100 over to the D710? I imagine this is a common problem for anyone who has bought carrier-specific hardware, but I'm not familiar enough with LineageOS to intuitively know how to do it.

Thanks for your work on this project!

Samsung has this nasty habit of releasing hardware under the same name, with insanely different hardware. This ranges somewhere between "The screen resolution is different" and "This S8 has an exynos chip, and this one has a qualcomm chip". As such, a bit hard to say what's required...

I don't have a question, just wanted to thank you for what you are doing. LineageOS is the reason why I can still use my 2013 first gen Moto G as a daily driver today. Projects like this are a giant fuck you to the companies hell bent on closing off their devices and taking as much freedom as possible away from the user.

Mine was unbearably slow even with Cyanogen installed. How is it working out for you?

I am on 7.1.2 and it works surprisingly well. Though it depends on how you use it I guess. I use mine just for messaging, GPS, reddit and listening to music. Battery is about 20% left after a day of usage. Not having Gapps on my phone probably helps. I've been planning to replace the battery soon enough from eBay.

A little off topic, but there is a plan for an official recovery from lineageos?

TWRP is fine, but full device encryption is not supported on all device that are supported by lineageos, and also it dosen't seem to support OTA update from lineageos updater.

ps: Thanks for the AWESOME work!

In normal AOSP land, there is no encryption support in recovery. I know we had the same restriction in cm recovery (you couldn't mount/backup individual files from /data, for example). I sadly have no more information on a lineage recovery right now.

Thank you for LineageOS! I'm using lineageos.microg.org to avoid Google services but have the unifiednlp stuff. I understand this isn't your release, but I appreciate the work you've done behind it that I benefit from.

I have noticed a bunch of relatively minor bugs. I believe they affect LineageOS directly as well, as I've also seen some of them on an old device that was using LineageOS directly.

How should I go about reporting these bugs? Do you want the reports?

These are things like "Changing the default SMS SIM card to SIM card 2 does not survive a reboot", "Adding an all-day calendar entry for multiple days loses the last day upon save if the timezone is London (GMT) and not in daylight saving", "Messaging app deletes all messages when the phone boots believing it's 1970 because the clock isn't set yet" and "Setting the year to 2018 after the phone boots up believing its 1970 is really difficult because the year box doesn't scroll far enough".

Generally we don't accept bugs on devices that don't come off our build slaves. It's a (rather convenient) way of getting rid of all the downstream-produced bugs...

That being said I threw this in the project's slack.

Thanks. If I can reproduce on a pure LineageOS device, do you want the bug reports then? In other words, are these the types of things you would patch LineageOS for if you had the patches? I ask because I don't have much of an understanding of how this ecosystem works.

Yes please!

Take a look here for more info on how we do bug reports currently: https://wiki.lineageos.org/bugreport-howto.html

Any chance you could bring back the ability to have a different encryption and lock screen password? If not, why won't you? Currently it's impossible to have a usable device with just secure passwords. Google's silence on this issue for 6 years is really sketchy, and I was shocked when I realized LineageOS removed this ability from CM as well. https://issuetracker.google.com/issues/36945251

This is possible with SnooperStopper: https://f-droid.org/packages/cz.eutopia.snooperstopper/

ugh that issuetracker link requires you to sign in with a google account...

I was wondering about this as well. It looks like the feature was removed in this commit [1], which was discussed a bit in this Reddit thread [2].

[1] https://review.lineageos.org/#/c/167033/

[2] https://www.reddit.com/r/LineageOS/comments/630f9x/why_were_...

Is there anything you wish that Google/AOSP contributors did differently that would make the LOS development process easier?

When you look into rooting or bootloader unlocking, one or more steps involve running some tool that isn't open source, or at least is primarily distributed as a binary of unspecified providence. You don't really know that the tools won't inject some unwanted software onto your device.

What would be a good way to promote transparency and trustworthiness of the alternative rom tooling? Or is there an alternative set that just doesn't jump out when searching for the tools?

> or bootloader unlocking, one or more steps involve running some tool that isn't open source

fastboot is open source. That's literally all you need to unlock the bootloader on a great number of phones. The vast majority of users don't need to root their phones, and doing so is a pretty big security hole for the users that think they need to.

On the other hand, your alternative is to stick with OEM/carrier OSes, which are quite obviously compromised for benefiting the OEM/carrier and not the user.

Rooting is needed for backups and migrating your data to a new device or over a firmware wipe. And good ad blocking.

I agree the majority of users make do without this, and so have little choice but to send their phone data to Google servers for migrating / backup.

Many apps allow exporting their data without requiring root. There are adblockers that set themselves up as a 'VPN connection' on your device for filtering out ads, those do not require root.

That vpn ad blocking sounds promiding. Are there reputable FOSS apps that do this?

Edit: this looks promising: https://github.com/M66B/NetGuard/blob/master/README.md

DNS66 https://f-droid.org/en/packages/org.jak_linux.dns66/ https://github.com/julian-klode/dns66

Successfully being used on a 1st-gen Moto Z Play, no root, stock firmware. Works great.

Seconding DNS66. Another benefit over hosts file based ad-blocking is you can disable/enable it without having to restart, which is super handy for the occasional "is this site broken or is it my ad-blocker" sanity check.

I was under the impression that hosts based blocking just needed you to toggle airplane mode to reset it, not a full reboot.

That could very well be the case. This could have been just me being ignorant.

Upon reflection, I suspect those localhost-vpn ad blockers have to reconstruct your ip traffic back into outgoing socket api calls, which can't work really well for non-web traffic.

I .. only learned about these ad blockers from this thread, so take everything I write with a healthy amount of table salt.

That said, as far as I understand this DNS66 thing only intercepts DNS requests (which yes, it needs to break apart and reassemble) and nothing else? That is - if you don't talk to a server that is part of the application's custom DNS server list or a DNS server given to you by your network configuration, then I'd expect this packet to bypass the VPN entirely.

Maybe someone who understands the Android VPN service thing a bit more can chime in and confirm?

You can root Lineage OS with one of the su downloads from https://download.lineageos.org/extras

Afterwards, you can enable root in the Developer Options.

You don't need Chainfire's SuperSU, if that's what you're referring to.

Nah, I think OP is talking about bootloader unlocking and rooting without OEM support. Using tools like DC unlocker or kingroot.

Also, don't use Chainfire's SuperSU in 2018 regardless. It's owned by some Chinese company that's not very reputable. The current trend is towards Magisk, an open-source root (and systemless /system) modification framework.

I think the biggest problem is that most of those developers don't want to actually host things in a proper way, like a regular open source project. Everything is on XDA, there's no proper bug tracking, no hosting, etc.

What's even worse is that they don't work together. If all all ROMs are based on AOSP, why can't they work together to get device compatibility? For example, why can't Lineage use Omni's code to support 'potter', and Omni use Lineage to support 'gemini'?

And then you have all the single developers working on their own devices. Why can't Lineage pull their work, do basic code review, and publish them as "beta" (and you get free beta testers)?

I'm another "project lead" at LineageOS:

Despite being based on AOSP, omni/lineage/etc all have pretty different features, so usually it takes a nontrivial amount of effort to go between the two (although it's definitely less effort than starting from scratch)

Most devices within the project are maintained by single developers - the difference is that official devices have been submitted to us and have everything working.

We don't blindly support devices (even as "beta") because we have no way of verifying that everything actually works (e.g. blobs that depend on an old C library ABI can't really be detected before runtime, afaik). Shipping stuff that's broken is something we'd like to avoid. The idea is that when you download a LineageOS build from lineageos.org you know that it'll work flawlessly (that may not always be the case, but it's the goal :P)

I recognize the username :) .

>Despite being based on AOSP, omni/lineage/etc all have pretty different features, so usually it takes a nontrivial amount of effort to go between the two (although it's definitely less effort than starting from scratch)

I understand, the same way that Firefox and Chrome are very different, yet work on the same kernel, or are the differences between Lineage and others that deep?

So for example, Omni has athene up and running on Oreo, while despite officially being maintained (with three maintainers at that), according to Lineage OS' code review, it hasn't been worked on since last July.

So the way I see it (coming from a dev, though not a ROM dev): Android has multiple layers. It has hardware-dependent code (such as the kernel) and hardware-independent code (such as bionic or Java), sort of like in Desktop Linux you have the kernel and X, and there's bash, gnome and Firefox.

So why can't (I know how that sounds coming from the outside ... ) you take their hardware code (the kernel and whatever the Android equivalent of X is) and build on top of that? Is the whole system that integrated?

On the other hand, now that Treble officially separated the two, will it be possible to, for example, flash Omni and then flash a lineage image on top of that?

>We don't blindly support devices (even as "beta") because we have no way of verifying that everything actually works

But does every Debian maintainer (of a userspace program like, say, MySQL) have every combination of hardware? You just assume that the kernel and libc are a reliable abstraction and go from there. Is that impossible on Android?

>So the way I see it (coming from a dev, though not a ROM dev): Android has multiple layers. It has hardware-dependent code (such as the kernel) and hardware-independent code (such as bionic or Java), sort of like in Desktop Linux you have the kernel and X, and there's bash, gnome and Firefox.

Yeah, that's pretty much it.

>So why can't (I know how that sounds coming from the outside ... ) you take their hardware code (the kernel and whatever the Android equivalent of X is) and build on top of that? Is the whole system that integrated?

We could - and it would definitely be a good start - but there would be changes necessary because e.g. our extra bits are implemented differently, maybe our "device-independent" bits are missing some commits that Omni has that fix support for some OEM driver - before treble, Android's hardware-dependent code and hardware-independent code were often both modified by OEMs/SoC manufacturers in order to fix issues on their platforms/add extra features. So Omni might have some fixes that we don't have, and vice versa.

>But does every Debian maintainer (of a userspace program like, say, MySQL) have every combination of hardware? You just assume that the kernel and libc are a reliable abstraction and go from there. Is that impossible on Android?

It's much harder on Android, because there's so many blobs built against previous platform releases (often with extra ABI changes from the OEM added on top). That means we can't assume that the ABI is consistent: for instance, we've had nasty bugs in proprietary binaries caused by AOSP adding fields to structs. Whereas on Debian, everything's kept in-step, so they don't end up with ABI mismatches all over the place.

Hopefully with Treble, a lot of this will change - so the idea of a "universal Lineage image" might be possible.

Thanks a lot!!

So while I’m not against this per se, I’ll leave the obligatory “why this could be bad” comment.

This is a bit like Google Play Services, once you integrate it you’re no longer relying on base Android features. While you can engineer around it, it has the effect of closing off apps that rely heavily on it from the larger Android ecosystem.

LineageOS is on the fringe though, and this SDK will be open, so I’m not as worried as I am about GPS.

If Google doesn't accept patches upstream, then you run that same risk with or without this SDK.

Might be a good idea to put the trademark disclaimers in the bottom of the page ("Android is a trademark of Google...") just to preemptively avoid any trademark issues.

For a second I thought it was an SDK for temple os and I got excited.

Reply on news.ycombinator.com