RaptorCS Power9 Blackbird PC: An expensive mistake
drewdevault.com
drewdevault.com
Raptor is a small company and one guy is fronting most of the support load so keep trying, he's not out to screw any customers or provide an inferior product.
Can he actually manage returns? His suppliers should be eating the cost of faulty boards, and the article makes me think they aren't. Continued problems customer side should result in a replacement.
In the US you have full rights to a return to the person who took your money for 30-90 days (depends on state). Requests that you use the warranty or contact the manufacturer are unenforceable. Considering this, it is unwise to wait for longer than that period before requesting your money back. The federal statute of limitations for being sold a defective product is 4 years, but that is a lot harder to enforce.
Remember the AMD Ryzen returns? Why wait 2-3 months for a replacement? Demand your money back from Newegg and buy another, they have to process the refund. Also -- most FFLs either state gun returns are outright illegal (a lie) or only accepted for 3 days (against state law) with a steep restocking fee (also against state law). This is the worst case I know of.
But Radio Silence for a few months... surely that is a bit too much?
On the other hand I didn't know Raptor was this small.
However, I think people that do not see what we see in the support system are greatly underestimating the volume of Email an OEM/ODM receives on a daily basis, which then must be filtered and sorted through to determine the valid issues from the flood of spam and general inquiries. It's not something we have a solution for yet.
May I recommend a more customer-friendly way of doing this? Allow the customer to exchange their iffy hardware first, then do the investigations on your time, with the hardware in your labs.
A family friend will help me troubleshoot my electronics. So, I have seen the other side of this: I get something which is completely nonfunctional and send it back to the manufacturer who claims it is functional, and tries to bill me for the shipping both ways. That's why if I can not get a working replacement in 2wk or so I just send it back for refund.
For a real example: Are you using very well made power supplies? Maybe your board's power filtering could be better, and customers notice because they buy the cheapest supply that meets their power requirements.
In general we can diagnose and repair or replace the systems rapidly. This one case was an extreme outlier, and in the future we will just be authorizing RMA and replacement / repair.
The trick is to get your board manufacturer to eat the return cost. If you can buy testing, do so; usually it comes with some assurance that the boards work, and they either make new stock or remanufacture what you send back. If you can point to a comprehensive testing instructions that the customer would have performed it should not matter if you receive the board and it works for you, for some reason it did not work for the customer.
I'm curious as to why you think this "needs" to happen?
I have my own reasons, for example, openness of ISA and IP from up top at the browser all the way down to the firmware on the chip; however, I'm curious to hear yours.
- A monoculture of HW leaves us more susceptible to security issues. See Spectre and Meltdown. While not eradicated by other hardware designs, the problem is generally mitigates. See relative difference in what Spectre and Meltdown affected as evidence.
- A monoculture of HW may lead us to local maxima of capability. As we approach theoretical maximums of the hardware in specific materials areas, we are presented with more and more complex architectures to eek out smaller and smaller gains. This makes the pursuit of alternative architectures more and more costly to invest in from a business standpoint, knowing that it may be years or decades before it starts to compete favorably with existing products. The more concurrent alternate architectures that can be pursued at the same time will allow for a much shorter time to market for alternatives, should a specific niche fit their capabilities well. (e.g. it's taken a decade, but Apple is making ARM competitive to the point it may be a viable laptop processor competitor, possibly even the better option at some point).
- Multiple avenues of research often yield benefits of cross-pollination. CISC/RISC in the past ended up converging somewhat in the middle as they adopted the best features of each other. The most lines of research we have (and used in the market, so they get sufficient funding and attention) the more likely we are to see benefits for all related hardware.
I don't want to live in a world where everything new is an iThing that says I can't help you Dave. That could turn retro-computing from a hobby to an essential life skill to preserve freedom!
That sounds like the "war on general-purpose computing" that Cory Doctorow was talking about a few years back.
I think Intel has been too dominant for too long. They've gotten complacent and sloppy, as shown by microcode-level vulnerabilities like Spectre and Meltdown.
Also, I figure breaking Intel's dominance will further marginalize Microsoft. I still hold a grudge over how horrible an experience Windows 98 was.
Microsoft is not stupid, Microsoft has ported everything to ARMv8 already, both for the Snapdragon laptops and the servers (which they only use internally for now, but I hope Amazon's Graviton is pushing them in the ARM-public-cloud direction)
And by everything… I mean not just Windows, but even SQL Server now: https://twitter.com/ASGConf/status/1173163007733616641
Fwiw, the NT kernel was designed for portability from the ground up. During initial development they targeted x86 and the i680 and it has been ported to pretty much any CPU that had relevance over time. Itanium, Alpha, ARM and.... PowerPC. While that branch is probably not maintained currently it would be pretty easy for Microsoft to get that going again.
If you want to run Linux in the cloud, you've got a bunch of providers (Google, Amazon, Microsoft, etc)
Microsoft simply wants to be in that market, so they'll offer the same products their competition does. Anything they can do to make Linux better on Azure is considered a net gain for them
My former boss always said that all devs worldwide should get the fastest machine possible for development but should be forced to use their own software on a 386. :-)
I appreciate this thinking, and have always thought that web developers should be forced to test with lynx on 9600baud dialup. But I'm just a wee bit sadistic when it comes to web developers.
"No tears, please. It's a waste of good suffering."
It's an add-on. Click "Add to cart" and then you'll have the option to add it on the next page.
> Do you mind mentioning which RAM module you purchased? On the wiki it looks like some are officially supported and others have been added by users. Maybe if the issue is related to the RAM, others have experienced similar problems.
Not interested in troubleshooting in HN comments, but thank you. There are some leads in the IRC channel that I'll follow up on when I have the chance.
Source: I've been building systems, including POWER systems, for 30 years.
The raptor co staff was exceptionally helpful to me on IRC in fixing it.
I'm really glad I read this piece. The geek in me was playing with the idea of getting a Talos 2, but the pricing makes me really hesitate. Knowing that if I'd end up with a problematic unit their support will be absolutely worthless is important information.
With Blackbird there was more of a focus on trying to understand any new faults found in the field, given their tendency to disappear when the units were brought back for analysis. Over the months after Blackbird shipments started, we learned that the majority of those were related to improper customer component installation (CPU, RAM, PSU, chassis), and are in the process of expanding the existing RMA policy to cover the Blackbird line.
I got a reply relatively quickly, but this was before they started shipping the Blackbirds so it's possible they are just swamped with orders.
1.) This particular case is one that Raptor needed to consult with IBM engineering to determine the correct course of action for. There is a very rare CPU failure mode (we have only seen it happen twice) that manifests with similar symptoms and a ZCAL failure. The root cause here is an apparently defective CPU that slipped past IBM final test, and work is underway to make sure that these marginal, but defective, CPUs are caught instead of being shipped out.
2.) There is definitely more than one person in support, however Raptor has been overloaded with both spam and off topic inquiries. Raptor has attempted to manage this load without much success, despite deploying new features such as the troubleshooting guide at https://wiki.raptorcs.com/wiki/Troubleshooting/Support_Reque... we continue to receive support requests that are missing vital information, and with descriptions that are so generic that the problems could range from "PC not plugged in" to "used non-ECC unbuffered RAM" to "tried to install Windows" (yes, all of those have happened, repeatedly).
3.) We are currently gearing up for a major push on owner-controlled computing. Higher level technical resources are assigned to those projects temporarily, which means unusual issues like the ZCAL problem are being resolved at a slower rate than normal.
I'd ask that the general community please bear with Raptor as we work through these problems. What we're doing is hard, it's not as simple as shipping a RISC-V SBC in a known configuration, we have the entire complexity of the traditional PC ecosystem in play. AMD's release of the uber-fast (but ultra-locked-down and owner-disrespecting) Rome CPUs has not helped the owner controlled computing world, and I am still of the opinion that POWER is a better long-term investment than RISC-V in the desktop and server space. FWIW I use a Blackbird desktop as my daily driver, and have not had any complaints with it (in fact it's given me less problems over time than my previous daily driver -- a corebooted ASUS KCMA-D8).
Raptor has a Wiki with a ton of information available at wiki.raptorcs.com and I am definitely open to further suggestions on how to make the nascent POWER desktop community a bit more self sufficient than it is now. Remember that x86 vendors don't need to offer significant post-sale support, partly because the Linux community has already filled that role, and as a result they can continue to offer low prices, strengthening the lock-in effect. If owner-controlled computing is to succeed in the marketplace, we need to figure out the best way to ensure that vendors aren't bearing the brunt of those costs and being forced pass them on in higher product pricing.
I believe support has reached out to you now that we understand the CPU will need to be replaced. Can you confirm this?
Thanks!
>I believe support has reached out to you now that we understand the CPU will need to be replaced. Can you confirm this?
Yes, support reached out. I said as much in the email, but to address some of your comments publically:
(1) Totally understand this and I posess a tremendous deal of tolerance for weird problems with unusual hardware from small companies. But the support experience here was abysmal and no amount of weirdness in a problem excuses the lack of communication. I would be with you every step of the way if Raptor had communicated the progress - the first I heard that IBM was even being consulted about the issue was today - or even sent me communications at the times they promised to in earlier replies.
(3) I think you had really better fix (2) before you increase your volume. The reply I received from support today insinuates that my support experience is rare, but I can't throw a stone in the IRC channel without hitting 3-4 people who have had similarly bad support experiences. Whatever investment you're putting into your upcoming "major push", you should probably siphon off some of it for investing in better support.
It also bears mentioning that 50+ days of silence, not including an additional 30 days where communications were devoid of information outside of unfulfiled promises about future communications, followed by a detailed response on the day I publish a scathing review online, is worth note. I should hope that other customers who don't have a soapbox to complain on are also having their concerns addressed.
First, I'd like to point out that Raptor already identified one of the major problems here -- that front line support failed to keep you notified of back end progress in diagnosing the problem. Raptor Computing Systems is an OEM and ODM first and foremost, not a services and support firm -- their capacity to handle requests is limited by the amount their customers are willing to pay at time of product purchase, and internal management of RCS is trying to balance support costs with all other costs and overall product pricing. They may not always get it right, but they are trying nonetheless.
Yes, this was brought to my attention as a result of the review. That review has apparently caused sufficient damage that Raptor will now be reevaluating whether they can afford to service the individual market at this time, or whether they need to focus on business users only.
My personal take:
I am saddened to see x86 and locked computing profiting from this, but at the end of the day individuals have always been a difficult segment to target. They tend to want extremely low (below cost) pricing, high levels of service, and generally want to run various proprietary games on Windows while simultaneously settling for a false sense of security through open source firmware wrapping large hidden binary components. Out of sight, out of mind, purchase price above all other concerns seems to be the general takeaway for that market.
As a result of this mentality, individuals are not currently Raptor's core market, and at the end of the day the individual computing market will get what it is willing to pay for. If that's locked (but cheap) x86 computing, open source itself may be in serious trouble due to the impending final lockdown of those platforms. If it's instead cheap low power computing (RISC-V, low end ARM) there's an awful lot of fat that will need to be removed from near-essential projects like Qt, KDE, Gnome, Firefox, Chromium, GCC, Clang, etc. to better fit them on the lower end platforms. I have not seen any developers yet willing to put in the time and effort to re-architect those systems for low end computing resources; in many cases the fundamental design would need to be changed (i.e. remove interpreted scripts as much as possible, go back to C/C++ or a similar low level language, etc.). In the case of projects like Chromium, it may simply not happen as the intent of that particular project is partly to serve ads and DRM protected content through a portal system, which aligns nicely with the idea of locked end user computing on x86.
To anyone having support issues, please feel free to re-raise any issues that were not addressed to your satisfaction earlier this year (including ones that may have been auto-closed)! Management is watching closely at this time to observe the process and make sure things work well for Raptor's end users.
> Yes, this was brought to my attention as a result of the review. That review has apparently caused sufficient damage that Raptor will now be reevaluating whether we can afford to service the individual market at this time, or whether they need to focus on business users only.
See: the review caused the damage, not Raptor's failure to provide a good support experience. In any case, if you can't handle the support load, then you're probably right in reconsidering.
I am really rooting for Raptor here, I'm not your enemy. I agree that x86 monoculture is awful for the ecosystem and ARM hardly presents anything better. This is also why I got heavily involved in RISC-V, and I wanted to find something similar in POWER9. I haven't laid hands on a working Blackbird yet but if it is as good as I hope it is, then I think you have a good shot if you just focus on scaling and fixing your support problems. If you can solve the support and get the system working, I would be thrilled to amend my review and publish a new one with all of the things I like about my shiny new Blackbird system.
> I am saddened to see x86 and locked computing profiting from this, but at the end of the day individuals have always been a difficult segment to target. They tend to want extremely low (below cost) pricing, high levels of service, and generally want to run various proprietary games on Windows while simultaneously settling for a false sense of security through open source firmware wrapping large hidden binary components. Out of sight, out of mind, purchase price above all other concerns seems to be the general takeaway for that market.
I can't claim to know what the firehose of support tickets is like, but know that there are users who don't fit this strawman. I'm one of them. I'm not here to run proprietary games on Windows. My stated goal is to promote the development of portable software and the adoption of freer systems by using my board to introduce POWER9 support to builds.sr.ht, giving access to POWER9 cycles to tens of thousands of developers. It's causes like this which help address the problems you mention of fat software needing to be improved to make alternative architectures viable. No small part of why Chromium is slow on ppc and fast on x86 is because there's a JIT for x86 and not for ppc, not necessarily because of fundamental problems in Chromium's design.
But yeah, there are fundamental problems in our entire ecosystem with respect to disreagarding performance thanks to the luxuries afforded to us by fast x86 systems. Is the answer to this that we should be writing off alternative hardware models? No, the hardware should strive to be fast and the software should strive to meet it. Both parties have a responsibility to performance and we can't lay the blame at each other's feet and ignore our role in it. It's a slow process to change the industry, we must exercise patience.
EDIT: it's been pointed out to me that Chromium has a JIT on POWER9. This doesn't substantially change my response.
I don't think I was clear enough here, sorry. Yes, Raptor Computing Systems is accountable for its poor support experience. Their management is trying, unsuccessfully, to manage the support load. I am actually on the engineering and technology side moreso than the support and business side, and I have no direct ability to change how support is handled beyond making (strong) recommendations.
Fundamentally, however, the blame does lie primarily with consumers that are unwilling to pay for owner controlled computing. Raptor took a risk with the Blackbird, attempting to broaden the reach of owner controlled compute beyond business and high end workstation users, and fundamentally underestimated > 20x additional support load that new market would end up causing. Raptor does not have the resources to handle that kind of support load, and the increases in product pricing required to do so would put Blackbird back out of reach of everyone except business users.
Raptor was working on fixing these issues regardless, but AMD Rome has made that nearly impossible due to the (predictable, but unfortunate) customer reaction that I detailed earlier. We have had to pivot to a different mechanism to try to resolve the problems, and I am not allowed to say more publicly as to what that will be at this time, other than it is coming for October.
I am an enthusiastic customer, and in case you missed my comments to you elsewhere, if you do shift sales in another direction, please consider continuing to sell to the hardcore old customers willing to do their own work on this hardware whatever it takes. This project is too crucial to let randos with just a light PR attack choke off the supply of this kind of hardware to people to whom it is so utterly critical.
-mm
Not sure if you're following IRC, but just in case I'll mention it here as well: Raptor products are already outpricing most consumers. Even in casual conversations with hacker friends, most of them balk at the price. Point being: if you increased your prices most of your buyers wouldn't blink. At present your buyers are blinking at the poor support experience. I would have paid a 50% premium for a product with actual support.
A compromise would be to offer ala-carte support packages, perhaps billed yearly. I would also have paid for that.
I know these decisions aren't in your court, but I don't have anyone else to share my opinion with, so there you have it. I hope that I will publish a more positive review of Raptor with these problems addressed soon. Thanks for coming by the thread to share your thoughts.
(2) Having known the person (singular)* who said they'd be priced out at +50% from other interactions, I wouldn't take their comments at face value.
* Apparently it's two people.
And like I said, I would have personally paid a 50% increase, or for an ala-carte support package, but I doubt a 50% bump is actually necessary. One which doesn't push the board out of people's price ranges is probably feasible. Remember that we're talking about $2-3K, for the cheapest option with the simplest loadout. Customers already have disposable income. These are not priced like consumer hardware.
Good to know on the paid support side. This is something we've been considering for a while, and is probably coming as part of the mitigation strategy next month.
I'm glad that things are moving foward in a productive way. I hope to be an early customer of paid support.
I had assumed this would be a story about Power9-based machines...