Pine64’s response to “Why I left Pine64”
pine64.org
pine64.org
I highly respect Martijn and the effort he has poured into pmOS and Pine64 and surrounding projects. Without wanting to make other important people feel left out, I consider him an absolute legend and the fact that he has felt compelled to write this blog post (and, unfortunately, to leave) is massively concerning.
I think it's very possible that there has been no malice here and it's all just an accident of one thing leading to another, but something has gone awry here and I think it would be nice to hear what Pine intend to do to put it right, or to know that there is such a plan at all.
> As for the reason why Pinebook Pro doesn’t ship with a bootloader on SPI – it isn’t caused by favoritism for a particular group of developers or disregard for good ideas, but rather something more trivial. Namerly, a functional Tow-Boot build for the Pinebook Pro wasn’t available at the time of manufacture and shipping.
One particular point I want to make: I'm willing to believe Pine about the above, but it's too late to hear about this after crucial people have been miffed. This seems to indicate that communication with the various projects has not been good. This seems like a crucial thing to improve on to prevent this recurring.
I don't have enough knowledge of the situation and am incapable of making any judgements as to what is really going on, but I sincerely hope that Pine will consider this situation carefully and whether something can be done to put things right again.
I really wish to see Pine and their hardware do well and flourish. That was never going to be easy, but it will be even less easy with the loss of people like Martijn.
This is a massive communication issue.
Pine64 was looking for somebody to commit to maintain PineBook Pro support in Tow-Boot (a relatively new project)
I don't know if the point-of-contact was made aware that PineBook Pro was not going to be shipped with Tow-Boot.
2022-06-01, the Pine64 dev IRC chat:
> [T] <tl_lim> I pause on move forward due to no one commit to maintain. If MayueIC interest to maintain, lets move forward and include Tow-Boot into the PBP factory build. @MayeulIC, you can send an email to info@pine64.org, we will ship a free Pinebook Pro to you so that you can maintain the Tow-Boot.
> [T] <tl_lim> On the integration, just follow the recent PinePhone Pro way that flash OS build to eMMC then flash Tow-Boot to SPI Flash.
> [T] <tl_lim> Frankly speaking, I am happy on developer commit maintaining Tow-Boot on Pinebook Pro. Thanks and a big shout out to @MayeulIC.
2022-06-02, in the Tow-Boot chat, which was discussing tl_lim's above comment:
> spikerguy: Tllim asked on. 25th April about pbp tow boot support i replied it just work fine as it should. But just because i am the one to work on factory images plus i am the manjaro arm pkg maintainer maybe he didn't wanted to hear from me instead the community that was assuring him about tow boot and none replied
> The current batch factory image is already submitted. So hopefully we can keep things ready for the next batch
I don't see all comms, but overall I see Tow-Boot on SPI at factory-time as nice-to-have but not essential (users can easily do this, distros don't need to require it, and the next batch can improve it if needded), Pine64 has showed willingness to use Tow-Boot. Tow-Boot has 1 developer, and I'm not sure if they were willing to commit to support this. I understand if they weren't.
This issue is blown out of proportion.
Maybe a good step would be for Pine64 to fund Tow-Boot's support of Pinebook Pro.
> [removing SPI] was based on the fact that for years SPI was largely unused on PINE64 devices.
People have been arguing that a u-boot firmware needs to be installed on that flash chip for years, too. Essentially everyone in the dev community has been pushing them to do this for essentially the entire lifetime of their product line.
Would you rather find, download, and flash a special-purpose, pinebook-specific image onto a microSD card, pop it in, and boot that up to install a distribution, or would you rather it supports a standard UEFI booting interface over USB, netboot, whatever else, like every other laptop can?
> We created a space for development talks to be held (as we always had)
My understanding from speaking to friends and colleagues who work with Pine64 is that these spaces are pretty toxic. Manjaro has the loudest voice and other projects don't get the respect they deserve. And I'm prepared to believe it if this flippant response quoted in Martijn's article is accurate: "people who want [an SPI chip] can just solder one on."
This is not the behavior that I expect from a project which claims to honor its community.
I'm very disappointed in this response and in PINE64 generally. I expected an apology and a promise to improve, and this isn't it. Pine, I believed in you from the start.[0] I thought you were better than this. The past few years of your behavior has disillusioned me to your project and I can no longer offer you my support.
[0]: https://drewdevault.com/2019/12/18/PinePhone-review.html
Although the firmware mine came with had an issue and I did have to update it, I now prefer it to my Hakko.
I will now consider buying second hand phone with postmarketOS support. It means I'll have to accept proprietary hardware but, as it will be second hand, it means I'll be more in line with my eco-environemental-whatever-half-baked principles.
Nobody cares, I just wanted to illustrate what are the reasons I would like to buy a Pine64 and how those reasons revolves around some basic principles and trade off's.
Regarding other postmarketOS supported devices, I quite like the Xiaomi Pocophone F1 and the OnePlus 6 - but be aware that bootloader unlock on the Xiaomi is a bit painful and charging on the OnePlus 6 is really slow (although that might have been fixed lately/be fixed soon).
It's possible to write Tow-Boot to the eMMC to do exactly this. SPI isn't needed, it's just more convenient. I happen to prefer SPI and Pine64 was willing to go with SPI too.
I think this issue is blown out of proportion.
Pine64 listened to feedback. The problem here was that timeline's didn't match, not bad intention.
That's how many of the SBCs I own (from differen vendor) are distributed. There's a space to solder the chip. (and you can pick the size you want)
You can also buy a Pine64 product and if you do not like the hardware you are free to modify it to your heart's content. However, they don't owe it to you to perform the modifications you want, certainly not if there is a shortage of the part you want them to install.
Listening and doing your own things anyways is the time-honored way of ignoring people...
Even as an Arch user (that hence "gets" Manjaro) that is sad :(
Much of the device still doesn't even work. And we're talking simple stuff like the cameras. Calling is also super sketchy. You can text but voice calls effectively is a fail.
The IMU does appear to work correctly, for things like rotation. It is pretty fast.
You can also fry your device (hardware damage!) if you get the pinephone keyboard and plug in USBC into the phone port instead of the keyboard port. You read that correctly. Hope you don't mess up if you're tired.
And if your battery on the PPP goes too low, the utter shit firmware goes into a boot loop and can't charge. You have to take off the case and hit the recessed button to put it in DFU mode. That bypasses the firmware causing bootloops. And it's slllloooowwwww charging.
I completely and utterly regret such a shit device. And I say that about the very hardware, AND the reticence in actually putting out base builds of working software.
They claim that it supports eMMC booting natively. I bought 4 of each. And I bought a eMMC USB dongle to pre-liad the images.
And .... No booting off of eMMC. And utterly NO help from anyone. No guides, no nothing.
And that's hardware stuff that Pine attested would work. And it completely doesn't. I still have to boot using mSD at a whole class 10.
Their whole "plan" and company is a shit show. It's all gongkai crap. Using or buying will give you pain and suffering if anything goes even slightly wrong.
And if they're choosing unsupported/unsupportable chips without doing the barest of legwork to get them functional (or hell, get docs public), then that's a huge problem.
Pine should be showing on their pages a hardware matrix showing what works and what doesn't. The Pinephone Pro doesn't qualify for the definition "phone".
If they were serious, they'd put this page up front, with a list of all the stuff that they imply works, but doesn't: https://wiki.pine64.org/wiki/PinePhone_Pro_Software_State
inb4 What works: You tell me!
Actually camera sensor was changed on request from Martijn. (or so I was told, when I was asked to add support for the cameras in the kernel)
"The PinePhone Pro Explorer Edition is aimed at Linux developers with an extensive knowledge of embedded systems and/or experience with mobile Linux"
To me thats pretty a pretty clear implication that many things may work poorly or not at all.
And, they call it a Pine *PHONE* Pro.
It can't even do the "Phone" thing.
And read up about the hardware critical failures that can destroy stuff when you plug in the wrong usb-c connection.
Yes, I expect a minimum functionality. You know, like NOT FRYING SHIT. They couldn't even manage to do that.
There's nothing wrong about Pinehnone Pro's Type-C port supporting circuitry. It's pretty much a reference design this time.
I'm actually on board, but do note that that's going to be a chart full of subnotes and caveats - things like "camera produces an slightly red-shifted but passable image on default software shipped from factory, red-shift is fixed with kernel 5.19-patch-foo but that breaks focus, Mobian ships that kernel with a userspace correction filter that fixes it in the latest nightlies[...]". You'll end up with a small table that's 90% little yellow "works with caveats" boxes and then several pages describing the caveats and various working and shades-of-working configurations. Oh, and it'll need to be timestamped and constantly updated (as in "major bugs fixed, and sometimes added, on a weekly basis")
In short, summarizing active development is never going to be simple.
One day I’ll probably end up with a Linux phone, but I mean, not if I’m worried I’ll get surprised when it arrives
My $200 pinephone can, on a good day, do 0.5hz. I dunno, but I expected more.
That's not a $200 proposition, though. :)
And I guess there's no huge motivation, given the expected quality of the resulting image. Maybe someone will write it for some other similar AW SoC that uses the same HW block.
Such is the power of FOSS.
I used to be all for early adoption, but too many people ran off with my money via Kickstarter and Steam Early Access.
edit- add phase x2 =p
It can make phone calls.
It took a separate $31,000 kickstarter and 4 years of a multi-vendor cooperation to create and upstream then non-existent accelerated stateless video decoding API for Linux. Pine64 would have never released anything if they waited on Linux API standardization process (have multiple users of the API in kernel, etc.). And even after 2 years of availability, the only major user of the API is gstreamer, and it's still bubbling through the stack so that you can just open Firefox and play an accelerated video.
Pushing through this process even nicher accelerated encoding feature, usable pretty much only for the cameras (which are quite crappy on the OG pinephone, so motivation is already low, there), is just not going to happen for giving someone a 7 year old outdated phone worth $150 only to some enthusiasts and $0 to pretty much everyone else, lol. That's just ridiculous.
I’d expect my PinePhone to come with software too. I mean, I guess it did, I think I just expected more from the open source community than from the nameless fly-by-night vendor.
Software takes time. But their plan is "cobble shit together in an untested and unmaintained state and YOLO".
* Didn't have cameras working
* Wifi didn't come back after suspend
* Had numerous modem reliability issues
* Bluetooth, sensors & audio were available but unused by software
* Had some hardware bugs
So this seems about on-par. Though on the userspace side things have of course progressed hugely for all devices.
I don't see where in Martijn's blog post that "coercion" was "supposed" at all. What he actually wrote:
> Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.
> After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.
He doesn't even claim that Manjaro objected at all to the idea of adding an SPI chip and preloading Tow-Boot on it; only that they have a possible vested interest in resisting it, and that PINE64 therefore also does for as long as it considers Manjaro the preferred OS for PinePhones.
I realize that’s healthy, but the way they’re going about it makes me feel like they’re trying to weasel out of it or something.
I’m left with a mild feeling of disappointment and I’m not sure why.
Maybe this, right here, is actually a major part of the bigger problem and not exonerating like it seems to be intended
> The reasoning behind not including SPI on the PCB wasn’t motivated by any single project or outside source – it was based on the fact that for years SPI was largely unused on PINE64 devices.
So if they only talked amongst themselves, how could they have any idea of if the SPI was already being used or not?
> And yet, despite this, we agreed to include it on the PinePhone Pro because developers from multiple projects – postmarketOS being one of them – were adamant that it was an absolute necessity.
So yeah, sounds like if they had engaged earlier, a lot of unnecessary frustration and grief could have been completely avoided.
Unfortunately this post does not even hint at the more significant point in Martijns post, the one about the relationships between PINE64 and the community distributions.
Seems like a bad idea to me.
Pine64 has listened to feedback, and made attempts to get Tow-Boot flashed on the PineBook Pro. It just didn't fit Pine64's manufacturing timeline.
Does any other manufacturer do better in this regard?
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is not that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem then so be it but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
If the device has the "ARM System Ready" certification it will run most linux distros just by using an arm build, similar to amd64.
Tianocore/EDK2 also supports many devices and adds a BIOS/UEFI that is similar to the BIOS you'd see on your typical amd64 mainboard.
I would guess all the user should need to do to use their own bootloader is to pacman -R manjaro-arm-uboot and pacman -S pmos-uboot to hand over control of the bootloader image if you buy a device flashed with Manjaro, or overwrite the eMMC bootloader with one that manually tries to boot from the SD card before resuming normal boot?
It seems that the frustrations about Manjaro go further than just the bootloader but I don't really understand why a SPI flash chip would even be a necessity in the first place.
The "bootloader" isn't just a bootloader, it's effectively a critical firmware component. If the "bootloader" is missing or garbled, the board is bricked and can only be recovered by removing the eMMC, which would be very difficult for most users. Such a thing should not be stored on a data partition and managed like any other distribution package. If they aren't going to ship a separate SPI chip, the eMMC should at least be repartitioned to include a separate "firmware" partition and the bootloader should be shipped there. Distributions should then boot via a chainload mechanism, just like on x86 hardware. (This mechanism can be UEFI based but need not be.)
No, it would require an SD-card to recover.
A lot hinges on a fact like that. It's the difference between not-convenient and not-possible.
Will the device actually know how to boot from sdcard if the part they are talking about is broken?
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
While they did in the end listen and add SPI _after_ manjaro finally agreed. that doesn't mean they listened to the whole community. Ofcourse PINE64 and Manjaro don't have to deal with how the other distributions boot.
Instead of delaying the PPP production by one day to get U-Boot sorted they decided to not ship mainline U-Boot at all. Which was only possible because Manjaro provided them with an image to ship with that.
In the end the only response there is is a blog post that never claims anything is wrong or should happen differently, after acknowledging all that I've done.
I only wrote why I left, that they themselves think they're manjaro focussed because the hardware that can't ship Linux doesn't ship manjaro doesn't change that this whole behind the scenes situation really burned me out, and I left.
Why does Pine64 favor Manjaro so much? What do they gain by doing so?
And for what it's worth, I know Android isn't without its issues - but why not build something on top of at least its Linux kernel and HAL and save so much effort in getting a system running? Why do people always have to reinvent the wheel despite it being almost impossible to compete with Android and iPhone anyway, leading to burn-out and failures?
That could have been a first step though, but at the risk of endangering the motivation to develop for mobile GNU/Linux and still be dependent of AOSP with nowhere to go when it really goes towards the wrong direction. More and more (essential) components are proprietary and shipped with the GApps. Even something as essential as the notification system relies on Google's servers and proprietary bits (this part has an open source re-implementation though, microG). At this point you need to recompile Android apps to avoid that… if you have access to the source code.
It's hard to fight against this direction, release after release, I think it is a lost battle.
I could live with AOSP today but I'd rather have a functional GNU/Linux environment with no bullshit in a few years.
Agreed, but a notification broker that's usable for more than just "on-device" generation of notifications or wants to avoid every app polling constantly will always need some sort of centralized server infrastructure that costs a lot of money to develop, keep running and secure (something like "user A gets notifications intended for user B" or "attacker manages to dump notifications" must never happen).
> I could live with AOSP today but I'd rather have a functional GNU/Linux environment with no bullshit in a few years.
Same, but as long as people continue bikeshedding and insisting on complete 100% purity every such effort is doomed to fail.
You don't need a notification broker unless you want to run non-free garbage. Well written apps will just service the socket they get data from just fine. This is already the way things work on Linux, the only thing missing for real time notifications is to have the network interfaces wake the device up instead of relying on eg rtc-wake.
>centralized server infrastructure that costs a lot of money to develop
Maybe but Mozilla seems to get away with doing it for free and they have way less money.
And the higher level comments are right. AOSP is absolute fucking garbage. Before Android 5 it wasn't clear if this was something intentional from Google but it's clear now. They hate people having control over their devices and are in many ways even worse than Apple here despite publishing some code.
>100% purity
Purity will be the only thing that results in success. Already the Pinephone works well enough for people who are used to desktop Linux (sans the hardware issues with the modem.)
> the only thing missing for real time notifications is to have the network interfaces wake the device up instead of relying on eg rtc-wake
Waking the device up for each incoming packet would probably drain the battery. But yeah, I had descent battery life on Android without GApps with mails and two chat apps (Signal and Element) so this can't be impossible to achieve. The modem has to stay up for incoming calls, listening a few additional TCP connections is probably fine. Maybe it's less power efficient than a centralized solution but still very usable. We could also imagine a solution where the used service is configurable.
> Purity will be the only thing that results in success
If you don't enforce 100% open source, you are less incentivized to fix broken stuff and develop open source alternatives.
The "open source but not quite" niche is already fulfilled and it's LineageOS and its friends. That's already done. So nobody loses here.
Face reality: most of what the utter majority of the people on this planet use is proprietary and ffs purity bikeshedding won't lead to any success.
> Well written apps will just service the socket they get data from just fine.
The utter majority of providers use CGNAT with extremely short keepalive times for cost saving, the worst I've seen was some crap MVNO dumping connections after ten seconds idling. That means your usual phone will have 20-50 apps that all have to send keepalive data all the time meaning the modem never goes asleep and drains your battery, vs one centralized notification broker that even shoddy MVNOs configure to be exempt from frequent connection drops.
> Already the Pinephone works well enough for people who are used to desktop Linux (sans the hardware issues with the modem.)
So... like what, 3% of users given Desktop Linux market share? LOL the only ones used to Linux on desktop are either forced to do so by their employer or are utter masochists willing to overlook catastrophic flaws for the sake of purity.
It's terrible in that it's huge piles of inefficient software that can barely run on hardware that's actually pretty powerful.
It's doomed in that as time passes, different FLOSS bits of it get "deprecated" and replaced with something proprietary. Or a new API appears, but Google phones actually ship a slightly better proprietary one.
There's a big conflict of interest when an ad company wanting to spy on people and lock them in is the steward of an open source OS.
EDIT: Also, don't forget that most devices out there use a custom kernel. There's no single "AOSP".
It may be less battery efficient than Google's builds but skipping all the spyware really saves a ton of battery.
I went from Maemo (Debian based) to AOSP Android when my n900 no longer worked. I've never experienced such a user-hostile platform as Android-- you even have to expend effort just to gain and retain root on a device you nominally own. And, I don't trust anything in the Android ecosystem to not be spying and tracking.
So I'd argue it makes more sense to first either get a decent alternative OS running on a proprietary piece of hardware (which is almost impossibly hard) or get an open piece of hardware running the only thing that comes close to an open-source OS first (so users will actually buy it because they can actually use it as their daily driver).
It doesn't make any sense for any project without billions of dollars backing it to attempt the 100% purist approach from the beginning. Even Mozilla wasn't able to pull it off, so how and why should anyone else succeed?
Mozilla did not try a purist approach. Their Boot2Gecko was built on AOSP downstream kernels and proprietary drivers. (Then again, there was no such thing as a truly 'purist' device back then, even in non-AOSP land. So they just did what could work at the time.) If you want to explore that kind of approach, there's things like UBPorts and Droidian. They might even make sense, if they end up being able to function as AOSP-compatible Generic System Images (GSI's) and run on mostly any Project Treble-capable device. Make no mistake though, that's a lot harder than doing everything properly in upstream and restricting one's attention to hardware that's compatible with that approach. If only because you're not forced to keep up with churn in the AOSP-native interfaces.
I'm glad you are a part of the community, championing for ease of installation across all distros.
Which other manufacturers provide any feedback channel, or engage with the community during manufacturing?
[1] At least as of a couple of months ago. I'm talking basic functionality being broken, like for example trying to add an icon to the home screen and the phone crashes. I encountered multiple bugs in just the first few minutes of usage.
The original "Why I left Pine64" article adds more info about what went on.