64-bit ARM Kernel Development Demo on a Nexus 9 | Hacker News Reader

64-bit ARM Kernel Development Demo on a Nexus 9

osdevnotes.blogspot.com

89 pointsandreiw36 comments

A short demo showing that your Nexus 9 is actually the cheapest and most accessible 64-bit ARM development platform for systems programming.

I got pretty sick of "Linux on xyz Android device" meaning a chroot with varying levels of usefulness. This is the first post in a while that shows some actual low-level dev work being done.

How difficult would it be to get a (mainline?) kernel booting on this, maybe even into Debian?

Especially humorous, considering "Linux" is technically a kernel, so "a chroot on Android" actually has zero Linux going on, just the GNU userland.

Huh? Android uses the Linux kernel...

But people and apps often refer to "Linux on Android" while it's nothing but a chroot environment. Android is already running the Linux kernel. Calling it "running Linux on Android" is stupid.

Well, it's been forked for a while, and has grown apart in many ways... if we consider it to still be the Linux kernel though, then "Linux on xyz Android device" is meaningless from the start.

It's not meaningless, it's short for "regular Linux distro".

Nvidia hasn't submitted any platform patches, nor device tree contents, for mainline yet.

Nearly all the drivers carry over just fine between 32- and 64-bit, and they've been actively moving a bunch of the little glue code out of arch/arm lately, so there's not a _whole_ lot left to do.

I haven't seen anything posted yet so it's not looking all that promising that it'll make 3.19 either (but it's still possible).

ARM SoC support in Linux mainline is usually pretty good but support for individual mobile devices that normally run Android's version of Linux is almost non-existent. Basically we are never going to get the usual Linux distros running on Android devices without using Android's version of Linux. There isn't anyone with the available, time, skills and motivation to get new or old Android devices supported by Linux mainline. I expect the same goes for other vendor-controlled Linux based mobile distros (Tizen, Sailfish, FirefoxOS etc). Probably the closest to mainline is the very old Nokia N900 but that doesn't even support all of the hardware on the device:

http://elinux.org/N900

Does anyone know what the differences between the mainline kernel and the Android kernel are?

This is actually surprising to me - I had expected that the only differences were in username (unit system, ipc, etc). Is it a question of one time development to make driver format compatible in Linux?

(upstream arm-soc kernel maintainer here :)

It's not really about anything Android-specific that's in the "android kernel", it's about all the other code that's needed for a platform to run well.

The main problem is that besides the base kernel, most mobile platforms require a lot of code that hasn't been upstreamed. In some cases, "hasn't yet", in some cases "probably never will be" -- it depends on the vendor involved if they have any such ambitions.

Most of this is drivers of various kind. Some vendors are better than others at upstreaming them and the other pieces that are needed, but nearly none of them have upstreamed sufficient amount of power management to make a real device useful and have reasonable battery life. There's also usually drivers missing for modems, etc.

And, of course, graphics is a very sore topic in this area -- no vendor today ships a phone that uses an open graphics base. Ironically enough, Nvidia is the vendor that has done best here, with work happening in the open on their DRM drivers (but no products have shipped with those drivers at this time).

The vendors that have done best are normally those who have more embedded-type platforms and not primarily mobile phone chipsets. By the time upstreaming of a mobile platform is done, the next generation is already out and nobody will build new products with the old one. It _does_ get better over time as more and more share code goes in, but most vendors have enough of a backlog that they don't see those benefits yet and as a result don't prioritize it as high as I wish they would.

Then there is of course some vendors who don't participate at all, or does very very little. The Chinese manufacturers used to be notorious here, but even some of them have started doing better as of late (Rockchip in particular, but MediaTek has started posting some patches too).

The vendor that traditionally has done best is TI, but they've gotten out of the mobile business. ST-Ericsson was making a good attempt too, and they also got out of it. Nvidia has actually been really good at working upstream on their 32-bit Tegra support, it's unfortunate that it's taken them this long to get going on the 64-bit support upstream.

When you say that there are drivers missing, does this go as far as GPL violations? Or is it not quite as toxic as that?

Also, what is the situation like with regard to figuring out what has been changed in a vendor-supplied kernel vs. a generic kernel source tree? Is it reasonably straightforward to get a clean diff so you can see what was changed? Non-upstreamed hacks and bodges seem less terrible to me if it's possible to at least see how they work.

Not necessarily GPL violations (there are cases of those too, but the bulk of it is not). It's just a matter of nobody having cleaned up the drivers enough to get them submitted for upstream inclusion. It can be quite a bit of work to do, and the next generation of product is sitting there waiting on the same people to make that work instead.

You can usually download and diff the source tarballs. Some vendors keep git repos so you can see changelogs as well. There's usually a lot of noise in there though, lots of various imported vendor drivers that duplicate things, firmware files in hexdump format, etc. It's pretty common to see diffs of millions of lines.

Thanks. The noise issue sounds pretty frustrating.

could you elaborate on what you mean by "cleaning the drivers" ?

I'm really curious, because it seems that Android has built an architecture where vendor supplied drivers (binary blobs?) can be dropped in without too much changes. But upstream linux needs a considerable amount of rework to "absorb" a driver.

Is this deliberate (the same reason why GCC is obfuscated) or is there a technical reason behind it ?

Because Android Linux works on thousands of different platforms with a lot of stability - I'm beginning to think that the Android driver model is superior... and perhaps mainstream Linux would do well to move to that.

I work on an embedded Linux device. Sometimes, we do whatever nasty hack we need to make this kernel work on that device -- but those changes aren't portable, aren't maintainable, and don't meet the quality requirements to be in the upstream tree.

There can also be a perception it's "a waste of time" since "nobody will run another kernel on this anyways"

Another difference between Android Kernel and mainstream is the power management system. Android devices suspend aggressively. Userspace acquires 'wakelocks,' during which time the device is not allowed to suspend. As soon as no wakelocks are held, the device can (and does typically) suspend.

This is pretty different than the heuristics used in cpu idle style power management.

I have a N800 laying around, would it be possible to put a modern kernel on it?

I see there is still activity on linux-omap [1] but I haven't played around with it yet.

[1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/tml...

Best i recall, there will be some issues with the power management as those parts were largely Nokia proprietary.

It is fairly easy to get Debian on an Android device, the problem is you will end up drinking over 9000 litres of beer due to the mainline Linux kernel not supporting your device.

http://bonedaddy.net/pabs3/log/2012/12/03/debian-mobile/ https://wiki.debian.org/ChrootOnAndroid

Wouldn't the cheapest option actually be https://developer.nvidia.com/jetson-tk1?

The Jetson TK1 uses the 32-bit Tegra K1. As far as I know, the Nexus 9 is probably the cheapest ARMv8 device that you're allowed to run your own kernel-mode programs on.

Ah, right. Thanks for the clarification.

Gorgeous! I know I'm asking for much but does it have JTAG debug access to the CPU as well?

Even if the JTAG pins are brought out, the hardware is a proprietary NVIDIA architecture, you wouldn't have any development tools for it. I suppose they might use the DBT firmware to allow direct debugging of the ARM code, but I don't think it's very likely.

Jetson TK1, the Nvidia dev board with 32-bit Tegra K1 has JTAG. The 64-bit chip has a completely different instruction set of course, is that the problem?

The 64-bit Tegra K1 doesn't run ARMv8 code natively. It's a proprietary VLIW architecture with an undocumented (AFAIK) ISA. It executes ARMv8 using a firmware-level dynamic binary translator, similarly to how Transmeta processors were running x86 code.

Ewww, TIL...

http://en.wikipedia.org/wiki/Project_Denver

Not ewww, it's one of the more interesting developments (well, technically a revival) in computer architecture from the industry. I just wish they'd 1) document their architecture and 2) allow access third party access to the bare metal.

Well, I don't think it's valid to call that non-native, given that it has no other mode of operation. x86 hasn't been "native" for a long time either, since it's microcoded at the very least (although not to the point of doing BT perhaps).

There are lots of cheap boards out there, way cheaper than a Nexus, of course they require a bit of electronics knowledge, but that should be part of any proper CS degree (at least mine had it).

Cheap 32-bit ARM boards, sure. Cheap 64-bit ARM devboards? I'm not aware of any on the market just yet... In a year's time your statement will probably be correct but I don't think it is just at this moment.

> It's unclear if the "oem unlock" allows reflashing custom unsigned firmware.

I don't think it does. I think you usually need "root" to install another ROM/OS on the device as well.

The oem unlock allows you run a custom OS (e.g. a custom Android build) that is not signed. This is well known, and is what allows this demo to work. However, the comment I was making is whether the firmware (i.e. the bootloader, "BIOS" if you may - in particular the Secure OS) can be reflashed.

The reason the existing firmware could be desired to be replaced is that the Secure OS component specifically disallows an OS to run in hypervisor mode (i.e. no KVM or Xen), or because of terrible bugs in HBOOT that cause hangs if the boot.img exceeds a certain arbitrary size (a few tens of MB). All these limitations mean that even an OEM-unlocked device is not quite entirely available to the mercy of a developer.

Of course, the ability to reflash the low-level boot code is not very useful unless you have sources to build a better replacement.

Are there any devices that ship with Android that could be reflashed to support a Linux kernel with KVM or Xen bits? Either Intel Atom or ARM-based devices?

It's weird, because according to this page: https://en.wikipedia.org/wiki/Qualcomm_Snapdragon#Snapdragon... , there are already 25+ 64bits-capable Android phones out there.

Reply on news.ycombinator.com