Xen Project officially ports its hypervisor to Raspberry Pi 4 | Hacker News Reader

Xen Project officially ports its hypervisor to Raspberry Pi 4

theregister.com

147 pointssohkamyung36 comments

RPi4 is interesting because it's the first Raspberry Pi with a serious amount of RAM (8GB), moderate speed CPUs (4 x Cortex A72), that can boot and run from external USB 3 SSDs, with UAS and UEFI+ACPI. Essentially you can now use them for aarch64 server SBSA/SBBR development.

It's about a third of the speed of my Intel i7 laptop from 18 months ago, which considering the incredibly low price I consider to be fantastic.

I recently got Fedora Server installed on one using Robert Grimm's instructions (linked from my blog along with other useful links here: https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running...)

Are you running a 64 bit kernel actually? Which distribution?

I found I could either get a 64 bit kernel (lots of things) Or decent hardware support (raspbian) but not both.

There is now beta (?) builds of a 64bit raspberry pi os (formerly known as raspbian) available: https://downloads.raspberrypi.org/raspios_arm64/images/

Not sure how well it works, but it is a thing now

I'm not sure I understand what the purpose of the raspberry pi OS is anymore. In the past it was a pretty custom kernel, with a user environment stripped down to work with a minimal RAM configuration. Everyone put up with nonstandard SSH daemons/window managers/missing packages/etc because due to the constrained environment and the fact that arm wasn't a first class citizen with the likes of ubuntu/redhat/etc.

These days, every single major distro has rpi instructions, and 8G of ram is more than some laptops come with. So, its a disservice that they don't just tell people, here are a list of disto's which have been qualified to run on the hardware, pick one, and oh BTW, you can run those OS's on your laptop too.

Of course. The real problem is not the 64 bit kernel which you can easily get with Raspbian, but the 64 bit userspace for which you currently need a different distribution (but see sister comment). I'm using Fedora 33 Server.

  $ uname -a
  Linux rpi4.home.annexia.org 5.8.11-300.fc33.aarch64 #1 SMP Wed Sep 23 13:51:39 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

Everything I care about except Wifi is currently supported.

With the latest Raspbian, you can always turn on the 64 bit kernel by modifying config.txt inside boot image and add: arm_64bit=1

My homelab is Kubernetes running on Raspberry Pi 4 with aarm64 architecture. So far, it works great.

> My homelab is Kubernetes running on Raspberry Pi 4

Honest question- do you just run one RPi4 or do you have multiple? Either way- how much ram do you have and do you find you use it all/wish you had more?

I have a cluster of 3 RPi4 each with 4GB RAM because it is super difficult to get the 8GB spec.

The more the better because it is difficult to run multiple dbs or queue systems right now.

I run Arch Linux ARM with the 64 bit kernel and userspace on my Raspberry Pi 3 for tinkering.

GPIO, Wifi, Ethernet, framebuffer HDMI, USB all work fine.

Video acceleration was the last hardware not supported in 64 bit land, and recent kernels finally have some experimental support. I haven't tried it myself though.

Raspbian itself has a beta 64 bit for kernel for a few months now.

https://www.raspberrypi.org/blog/latest-raspberry-pi-os-upda...

I've found Ubuntu Mate even easier to use than raspbian, and now it finally supports the Pi4:

https://ubuntu-mate.org/download/arm64/

I recently saw an article on setting up jitsi on a pi, and I thought it required 64-bit.

https://gist.github.com/krithin/e50a6001c8435e46cb85f5c6c78e...

I guess there's a 64-bit ubuntu 20.04 server

Oh, that's excellent. The same group doing the RPi4 UEFI stuff also does a version for the Pi 3: https://github.com/pftf/RPi3

AIUI there are problems with that fact that the RPi3 doesn't have a "proper" GIC. The RPI4 has GICv2, but RPi3 has some sort of custom interrupt controller.

Actually I am more impressed with the ESP32, given how much powerful it is than what we had at the high school computer club.

ESP32 is fun, but its low-power states and transitions are said to be messed up, so not as useful as e.g. STM32 for battery-powered projects.

I have no first-hand knowledge, but would welcome reports from anyone who does.

If you're interested in something low-power and not running a general-purpose OS like Linux, watch for the launch of the Precursor [1]. The main processor (RISC-V on an FPGA) seems to be more powerful than an ESP32, and they rejected the ESP32 for the WiFi component specifically because of the low-power state transition issues. The intended application is a stand-alone secure enclave complete with LCD, keyboard, and audio, but it will be interesting to see what hacks are developed for this hardware.

[1]: https://www.crowdsupply.com/sutajio-kosagi/precursor

Before I read about Precursor, I did not realize that FPGAs could be at the center of a low-power design.

To be clear, it looks like they've had to compromise on power consumption versus performance in their FPGA-based prototype:

https://betrusted.io/betrusted-architecture/

They plan to move to a custom ASIC later:

https://betrusted.io/dev-plan/

Yes, the ESP32 uses a ton of power relative to most MCUs (external flash comes at a cost!) and has relatively poor peripherals, documentation and tooling. But it's cheap and not bad if you need WiFi or Bluetooth.

Most non-networked projects would probably be better served by one of the numerous Cortex-M microcontrollers out there.

https://lore.kernel.org/lkml/alpine.DEB.2.21.2007101521290.4...

Is this a parallel effort? Or somebody who did it first? Or a snapshot of the work reported?

And, if one of the former, how did they solve the problems reported in the article?

> Is this a parallel effort? Or somebody who did it first?

It's not a parallel effort; Stefano Stabellini is actually quoted in The Register article.

There's a tiny bit more technical detail in Stefano's own article here:

https://www.linux.com/featured/xen-on-raspberry-pi-4-adventu...

But for more details you'll have to either look through the git commits, or wait until the LWN article comes out.

Did ESXi on Raspi ever materialize? I remember seeing rumors about it being in the works last year.

There was another VMware demo recently. https://www.theregister.com/2020/09/24/vmworld_preview_esxi_...

Pay attention to VMworld this week :)

Thanks for the heads up, will certainly do!

Per watt, Raspberry 4 is probably peak CPU of our civilization. But 64-bit is only viable if you buy the twice as expensive 8GB model and have use for the extra bandwidth.

I'm building hybrid clusters of 4 (2GB) & 2 (the older version) on 32-bit so I can run my PaaS at record flops/W/$:

https://github.com/tinspin/rupy

> But 64-bit is only viable if you buy the twice as expensive 8GB model and have use for the extra bandwidth.

Benchmarks don't entirely agree: https://www.cnx-software.com/2016/03/01/64-bit-arm-aarch64-i...

If the difference between AArch64 and AArch32 were simply larger pointers, I would probably agree, but AArch64 includes a number of instructions that can do more work, so 64-bit programs can actually run faster than their 32-bit equivalents, even if you're not using much RAM.

AArch64 also has a larger number of registers, which can make a big difference for workloads.

Per watt, it's not even close. That's because RPis are a lot behind on manufacturing processes, with newer designs being more efficient too. (RPi4 is on 28nm)

Apple has had the performance per watt crown for quite some time. These chips are not very good for anything outside of prototyping, otherwise hyperscalers would run a sea of small nodes.

> if you buy the twice as expensive 8GB model and have use for the extra bandwidth.

Are you seeing increased memory bandwidth on the 8GB version vs. the 1GB/2GB/4GB ones?

So this is "Asymmetric Nested Virtualization", then?

One of the guys behind this hack here. The other is Stefano (and most credit goes to him actually). Would love for everyone to give it a try and help us improve it. I'm working on a few ready-made VMs that you can easily deploy on Raspberry Pi 4 with Project EVE (including Windows 10 VM and Plan9 VM). If you're interested -- drop me a note (I may even be able to do a few bounties/etc. for things we need to improve ASAP). Otherwise -- feel free to ask questions, etc. in this thread. It looks small -- but the whole thing took us a month or so (not full time of course) so there's quite a bit of subtlety there.

Btw, I just cut EVE release 5.12.3-rpi-xen-arm64 (there's also 5.12.3-rpi-kvm-arm64 of course -- but this is Xen thread ;-))) that fixes a few issues that were reported ever since we got slashdotted this morning. This is your best bet in trying Xen/EVE/RPi4 combo. E.g. docker run lfedge/eve:5.12.3-rpi-xen-arm64 live > live.raw is literally all you need to get going.

Reply on news.ycombinator.com