RaptorCS's redemption: the POWER9 machine works
drewdevault.com
drewdevault.com
You are a gem and I wish more people were like you.
(I think this applies both to hardware and software products.)
Of course, this is easier said than done, but ultimately this is the core function - are your customers happy or not?
As a CTO you need to prepare the company and your investors for this failure mode. You need to be able to explain to them that sometimes making sure that the product actually works and the existing customers are happy is more important than that sexy new feature of the month. It's not always a very easy balance.
In my experience, even semi-technical people with a more sales/marketing kind of focus in the company often get blindsided by this. It's on us to prep them for this.
It seems like this company learned it the hard way. The key thing is that they did learn.
This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again.
And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)
In fact I'm debugging a device in the wild behaving very strangely at this exact moment.
> the larger failure in my mind was the breakdown in the support process.
100% agreed. This is where your company failed. And now has redeemed itself.
I’m glad to hear that it was a one-off and that you have revised your support strategy, but... really, it made for a pretty frustrating read. It’s not chump change, and going out on that kind of a limb on... trust... which was clearly unwarranted at least at the time of that initial incident.
There’s nothing more frustrating than paying top dollar for something exotic and then being stonewalled by oblivious “tech support” when things don’t work as advertised, particularly when something is sufficiently novel or unusual that relatively few other users are out there to proffer help and the potential points of failure are plentiful and “unknown unknowns”.
As far as I’m concerned, that incident was extremely bad publicity. That board could’ve been shipped to me. I’d have been left to dangle from my solitary rope and squirm when management asked me what was going on.
That’s just not acceptable, sadly.
Being worried over what might have been spoilt milk if you had gotten a broken board in the past seems counterproductive.
I might still get a machine for my own amusement, but... I don’t know, reading of some poor guy left dangling for best part of a year just doesn’t build confidence and leaves me with a “there but for the Grace of God go I” feeling, you understand?
That being said, the assembled systems products are checked with a significant burn-in test etc. before they ship. We have never had an assembled system arrive defective anywhere, except if it was damaged in shipping (i.e. crushed box etc.).
No DRM, full-ownership computing is great, and the performance is literally astounding (particularly for our use-case), but at the time the decision was made, based on the evidence I had, the decision was clear. I do not regret deciding not to buy your product because based on the data available to me at the time as far as I knew I would’ve been alone and/or I would have spent a fortune sending faulty items back-and-forth through customs, paying import duty each time.
Whoever decided not to be proactive in this guy’s case procured your firm a sizeable amount of bad publicity.
Not to mention that the team I am part of would’ve probably, as a side-effect of the proprietary software we are developing, also contributed various patches and so forth to various parts of the underlying open-source stack.
Just in case you didn't see them (they're not exactly easy to find with Google etc. for some reason), there are a number of positive reviews or comments online from various people. I know one thing we haven't (yet) done is to start assembling all the success stories around our hardware; this is always challenging due to the number of people that use the hardware without posting online for privacy reasons etc.
Without further ado:
Slightly out of date (the software now available for POWER is significantly more robust than it was at the time -- we even have Fedora desktop ISOs showing up now for POWER):
http://tenfourfox.blogspot.com/2018/05/a-semi-review-of-rapt...
Various other (more recent) comments from our user base:
https://twitter.com/RobinBa40108750/status/11790782959813632...
https://twitter.com/tsundeoku/status/1171414555089494016
https://twitter.com/alnesbit/status/1170425263559254017
https://twitter.com/klauskiwi/status/1164273128588972032
https://twitter.com/notgrubles/status/1161438996586803200
https://mastodon.sdf.org/@jirka/102520125253219962
Just wanted to show a little bit of the other side of things, for context. :)
However, from a libre computing perspective, the WX7100 in my Talos II requires firmware which is non-open, even though it is included in Linux. I imagine that this market would include people interested in reducing the amount of firmware they don't control as much as possible, so for this niche, that is relevant.
We don't support this yet in wlroots, but it's definitely something we want to add, and not a limitation of the Wayland protocol at all.
Basically, it doesn't do anyone any good for us to lower cost by giving up the full owner control experience that is centric to our product lines. :)
Youmu Konpaku is best character. Just gotta love the sword-characters in a bullet-game. Although Flandre was an "extra" endgame boss character, never actually playable IIRC. Really good music for her stage though.
----------
Glad to hear everything is working out for your machine. One thing I'm curious about:
> Installation was a breeze, it compiles the kernel on 32 cores from spinning rust in 4m15s
Is that 8-core / 32-thread CPU? I don't think there's actually a 32-real core CPU available from them. If so, I think that's an impressive speed for 8-real cores.
I went through that website before posting. That's why I thought it was a Gitlab-like service. I like that it works without JavaScript, and it appears to have a wiki-like integration too.
It looks like a neat service.
All in all, I have been very happy.
For example instead of pull requests, you have mailing lists in which you send patches. You can see more in the website of Sourcehut: https://sourcehut.org
Does anyone know how this compares to a recent x86 machine? This seems quite fast for the entire kernel, but I have no idea what typical build times are like.
https://www.phoronix.com/scan.php?page=article&item=blackbir...
real 1m55.267s
Run from tmpfs, all protections from Spectre etc. on. Not too bad, really!
A fun benchmark would be a Power9 Vs a X86_64 cross compiling the same ARM64 kernel. I wouldn't bet heavily on the Power9...
Not that this matters to the end user, but it is still interesting IMO
Slightly OT, but if you limit the playing field to open ISA / fully owner controlled systems, POWER9 is so far out front in terms of performance that it looks like an outlier. ;)
However, yes, by far most of the remaining owner controlled systems are even smaller -- some of the smaller (older) ARM Chromebooks (sans GPU, of course), the BeagleBone black, those kinds of systems.
Does the x86 specification say "speculation will have no observable side effects on the memory subsytem"? I wasn't aware of that. In other words, Spectre/Meltdown are bad security flaws, but I don't think they're the result of cheating.
You can't design your software to protect against Meltdown, which makes the ISA guarantees useless. Whereas Spectre-type side channels can be mitigated through software, which is what cryptographic algorithm implementations have been doing for years.
There's far more room for debate regarding culpability for Spectre class attacks, though it's pretty clear that Intel deliberately pushed the envelope in ways that AMD and ARM weren't prepared to do.
Raptor does sell a 22-core SMT4 capable CPU, which, Yikes! that's a lot of computing power.
That Blackbird bundle seems like a beast. The Phoronix benchmarks show it outperformed by high end x86 chips though. I wonder if some of that comes down to gcc optimizations because x86 has been around longer.