How a few screws cost $2000 and a 240GB multinodes cluster cost $50
quantisan.com
quantisan.com
Renting is always cheaper than owning. That's a pretty universal rule. Complain about the red tape all you want, I have also felt the burn of the "approved supplier" list. That doesn't excuse an extremely faulty comparison, however. It just clouds the issue.
I understand coming up with a better comparison would be a little more work than the immediate but completely faulty on-hand cost of the server instance they spun up, but I know the cost of the computer I own and the cost of a server I rent and I wouldn't use them to make a comparison just because I'm too bothered to come up with a real comparison.
There is also the issue of the fact that the $2000 number was pulled out of seemingly thin air, the possibility that they use approved suppliers to avoid the issue of counterfeit or sub-standard parts, which have killed people when installed into regular things like bleachers that stay on the Earth. It's the freaking Canadarm, god forbid we splurge now so we don't waste millions later fixing it or kill someone. When the end result of the materialized idea is it's going to be launched into space, the costs probably aren't so black and white.
Interesting story, but with so many variables unaccounted for I have a hard time gleaning any sort of lesson from it. Other than the fact that building a piece of equipment that will be launched into space involves a fair bit of red tape, which I was already aware of and am honestly kind of thankful for.
Yes, but OP clearly says the boxes were never to be used in production, only testing.
Reductio ad absurdum says the table it sits on is "part of the test", and the power supply it runs off is "part of the test", and the engineers car that he uses to get to work to run the test is "part of the test"...
I'd _strongly_ advise that a screw that hold together the enclosure of a load-simulating-test-jig is _not_ "part of the test" in the way that requires ISO9000 compliance (or space-rating).
Instead he contrasted it to what you can do today as a startup.
What would be the point of testing it then? If the test isn't replicating the production environment before production, what is the point? At that point you might as well just hot-glue the safety cover on. What's the difference?
Me, I'd _happily_ hot glue the cover on if it were more convenient to me at the time. I see this as mostly a workplace safety issue - the screw holding the cover on is _not_ "part of the test", it's mostly to keep accidental fingerpoking out.
But it is an interesting connection to make, and there are interesting ideas that the comparison generates, so let's treat it like a parable rather than a critical business analysis.
If anything, one of the most striking comparisons is how work was done in a government-funded research lab versus how it is done in a startup. Own vs rent, meticulous simulation vs rapidly-iterated prototyping, waterfall vs agile, regulation vs free for all.
The superficial details of the story aren't actually that important, but there's a lot of meat to dig into.
All of this "agile," "free for all," "start up" cuteness is okay when you're dealing with software, which generally only ever results in frustration or at worst lost profits. The standards and practices in industries where things can go wrong in epic, expensive, and dangerous ways is obviously quite different.
Two people playing with that cluster for a couple of hours cost more than $45.
Still, the author is right that when building physical products sometimes things are a lot more expensive than you think they're going to be because you're following some strict protocol.
The main point is that in a huge corporation with rules for purchasing and choosing suppliers, there's an insane amount of labor spent in administration even for purchasing even the most mundane things, as soon as it's something deviating from "the norm".
I've worked in a highly regulated industry, in a pretty big company, and my experience was the following: If you always order €1000 devices from a company, this company is registered in the internal databases, and had an audit from your quality assurance company. All traffic-lights will be green. You order the device, you, your boss, the financial responsible or whatever signs the document and they will bill one hour of work for that single-hand-move operation, which will be in the order of €100/$100. Total cost for your department of project: €1100 for tie €1000 device.
Now if you are in the lab, and think it's cool to use that Arduino shield for $10 on seedstudio for your lab-setup (which is probably a completely reasonable approach, as you only want to turn on/off that light/motor/... you normally would have to do manually for your experiment/measurement/test) the pain begins:
Seedstudio is not on the list of approved vendors, the QA team has to be persuaded that this is not a item that will ever reach a customers installation so they don't travel there to do an audit of the ISO9000 compliance (or whatever special regulation your partícular industry demands). Purchasing will complain that seedstudio has never signed your "terms of order" (and they will just throw away the 10-page-legalese anyway for $10...). And controlling will complain endlessly about the "proper inventory code" for a "seedstudio stepper motor shield". They'll also give you a sticker and make it a "IT inventory item" because it can be connected to a PC. In the end you'll have 5 people bill each 2 hours on that issue, and your project will be billed 52$100 + $10 = 1010$ of the Arduino shield. And you'll be known as "that hacker guy" that "constantly orders strange things from chinese companies" through the whole company.
Everyone new to that kind of big corporation tries this once, or twice, and then decides that it's just not worth it. In practice, you'll buy it on your private credit card, and hand it in as "travel expenses" on your next business trip. Or you'll have subcontractor already working with you that will order it for you, forward the device, and just put "Misc. manufacturing material. $50" on the next monthly $20'000 bill that on one will ever check.
Each of those arcane rules was probably put in place because someone a long time ago did something they weren't supposed to do. "Working the system" by intentionally ignoring the rules begs two questions: who needs these rules, and how long until you get caught and more rules are piled on for the next guy to get jaded about?
At some point it becomes madness. Sometimes that's when your company gets its lunch eaten.
If you're just able to hand in an invoice to accounts payable then it's very, very easy to come up with bogus invoices. In fact this used to be a quite common scam.
Thus those rigid controls on all things procurement.
I worked for a variety of global companies and government entities. Setting up the whole invoicing process is generally a pain and usually you'll have to wait for a couple of month until the process works flawless.
Once it works it usually does work flawless.
I'm not arguing that this is great. But I can understand where a company is coming from, since the whole field of procurement is just a huge invite for fraud and abuse, if it's not rigidly controlled, which, I'm afraid, also involves an approved supplier list.
[EDIT : Spello corrected]
Once he sees a rather ridiculous cost overhead right there on paper and ideally even verfiable with your orders through other systems, interesting things happen. More often than not, management didn't know the excess of the situation.
In fact, in most large electronics companies the real money is made and lost in the supply chain so they have incredible power, annoying as it sometimes is.
The cost and value of using the bought products (screws/servers) is not included in either example.
Yeah, sure in this case it was just some screws on a plastic housing and there was a 99.999% chance that everything would've been fine. But you don't want to make exceptions, because if you play fast and loose here and there, eventually all the prudent practices go out the window and you're no better than the software guys.
This is a huge point that many HNers probably don't get(but they think that they get). Releasing patches/bugfixes doesn't work so well on physical stuff. If Amazon AWS goes down, a group of people can figure out what went wrong, fix the issue and "reboot" the system and all that is lost is a bit of revenue and a portion of their reputation. When pieces break on heavy iron you don't have the time to figure out what went wrong and then fix it because someone is about to crash. Hence the difference between the traditional engineers and the software folks is the traditional engineers have to fix stuff before it breaks, software engineers can afford to fix stuff after it breaks.
Absolutely. The mindset is just very different. I remember once running some software in debugger during a major tech demo, and fixing a display bug I saw was going to become a problem in the next few minutes using Edit & Continue in Visual Studio. That's the "software guy," "start up," way of doing it. That shit would never fly with "hardware guys."
One of my duties, along with being a radio technician, is ordering replacement parts for the systems that we work on. Some of the stuff is expensive and reasonably so - for example, a circuit board with $200 bucks in components is $20,000 because it has to be professionally made and is basically a one-of-a-kind part. The supplier has to charge that just to recoup the costs of retooling. We accept this because we need very reliable parts on radar systems and are willing to pay for the assurance that these parts will be good.
On the other hand, I just ordered an audio cable through the FAA. Simple two-wire audio cable... $333.45. I could've bought the exact same thing from Radioshack for five bucks. Or Monoprice for one dollar.
At one point, Raytheon told us that we were not allowed to do any intermediate-level maintenance (anything involving fixing circuit cards) on their stuff. We were expected to send the bad cards to them, and they would give us replacements and bill us for $10,000 each. The chief warrant officer laughed, said "Fuuuck no," and told us that if we could easily fix it, do so.
Raytheon realized we weren't sending them any circuit cards and called the commanding general to get him relieved. He didn't get fired, but we started sending the cards to them for broken 2-cent resistors. For ten grand each. Same thing with $30,000 power supplies, etc.
Right now, the government has declared that getting office supplies from Staples is horrible. We're supposed to get them from Servmart, which sells them for five times the price... or more. Cheap-shit ballpoint pens for a dollar each when I can get Bics for $4.60 for 72 of them.
Your tax dollars at work, gents.
Someone explain why a frying pan and a Raspberry Pi cost the same.
The following is speculation.
On AliBarBar, you can find wholesale pans at source in china for $1 each. The price of pi like devices is about $30 dollars.
So, why do pans get so much more expensive as they cross the ocean?
1. Cost of postage
2. Protectionist import taxes. Here is Europeans list of things they are thinking of levvying punative taxes on. It's long. http://trade.ec.europa.eu/tdi/notices.cfm
3. Regulatory approval. The burden of bringing a new electronic device to market is pretty high. You need specialist facilities (or to pay a lot of money to a test lab) to prove electrical emissions are fine. This is why so many Chinese designs never make it to the UK. Alibaba is full of lovely android powered arm net-tops, but you note the ones you see in electronic stores are from the same limited range, even the cheap ones. This is why.
I presume the same is true for pans. So although designing a pan and bringing it to manufacture is simpler than manufacturing a new ASIC or even embedded computer, only a very few pan designs can be sold in the UK. It just isn't worth bringing another one through approval (unless it's much cheaper). And the approved designs can sell for much more.
One hears about western companies keeping as much of the 'value add' as possible out of China and India, possibly because it's so expensive to retrieve money once it's in there. I wonder if this also is part of it.
My guess? Because pans get sold by experienced companies who recognize that the price of an item is decided by its utility and the amount people are prepared to pay for that utility, not anything to do with the cost of manufacture. The gap between the manufacturing cost and the utility value of an item only lets you choose whether or not to be in the business of selling that item.
Transport costs per unit are probably also higher for pans (cause they're bigger and heavier than arduino).
The price of an item is decided by supply and demand. The only way you get a retail price which is significantly higher than the manufacturing cost is a lack of competition.
For items like pans this is probably more at the retail level than the manufacturers. I kind of doubt Walmart is buying $1 pans for $45 and selling them for $50.
There's something going on in the "pan selling industry" which results in very little "race for the bottom price" type competition - there are clearly many individual companies importing pans from China, but the "well known retail price" of those pans is ~$30 no matter which retailer you buy it from (within reason - at least within enough reason that all the $30 price-point retailers are still in business).
Tech toys, on the other hand, quite often end up in the "race to the bottom" price category - with many innovative and desirable items ending up available from sites like Alibaba or DealExtreme at ludicrously low prices.
I think part of it is the audience - the "pan buying demographic" is probably a lot earlier in it's uptake of globalised web based purchasing than the demographic who's buying Arduinos and RasberryPis. Perhaps one day your average mid-western housewife will say "I need a new saucepan, I'll just check Alibaba before I drive to the mall and pick one off the shelf at Walmart - if I can save $20 and only need to wait a week I'll just get it delivered."
But like I said - I suspect it's "experienced companies" who are not exactly colluding or price-fixing, but who all know that it's better for _all_ of them to keep pans priced at ~$30 and have everybody make margins of almost $29 per sale, rather than end up trying to compete on price at $5/pan knowing that the total market for pans won't increase just 'cause the prices drop 85%.
(And sometimes, you get the evil-Walmart-effect, where someone big enough to absorb the temporary losses chooses to drop the consumer-expected pricing down to $2/pan, waits out until the competition goes broke and vanishes, then puts the prices back up near $30/pan while owning all or most of the market.)
That's what I'm saying -- they don't compete with each other. There isn't enough competition that anyone decides to try to improve their market share by lowering prices, because it's just Bed, Bath & Beyond and Sears and such like "competing" with each other in any given local area, and not enough people buy housewares over the internet to move the needle against those guys.
>I think part of it is the audience - the "pan buying demographic" is probably a lot earlier in it's uptake of globalised web based purchasing than the demographic who's buying Arduinos and RasberryPis.
More than that, electronics are purchased in bulk by corporate purchasing departments. If you're buying 25 new devices every two years you may not be such a big fish that you qualify for a volume discount, but you're certainly spending enough money to justify a significant amount of comparison shopping, which means a high price elasticity of demand and an immediate market share advantage for retailers who engage in price competition.
>But like I said - I suspect it's "experienced companies" who are not exactly colluding or price-fixing, but who all know that it's better for _all_ of them to keep pans priced at ~$30 and have everybody make margins of almost $29 per sale, rather than end up trying to compete on price at $5/pan knowing that the total market for pans won't increase just 'cause the prices drop 85%.
The phrase you're looking for is "conscious parallelism." You get de facto collusion without communication by means of everyone adopting the strategy that they not be the first to engage in price competition. It works only so long as you have sufficiently few competitors that all of them religiously follow the strategy -- because having a third of the market with 5X the margins is more profitable than having the whole market at 1/5th the margins. The problem comes when you have more competitors, in which case someone decides they would rather have half of the market at 1/5th the margins than 1/15th of the market at 5X the margins, and believes (perhaps correctly) that they will be the one to survive a price war. This has basically been Walmart's business model for probably the majority of the products they sell, and Amazon et al are now doing the same thing even more aggressively on the internet.
Edit: Here's an example, but at your local store in a sketchy area they'll be even cheaper: http://www.instawares.com/winco-gladiator-fry-pan.wdi-afp7xc...
I used to work in a lab with a researcher who designed complex mechanical systems with lots of rotating parts made from rare and expensive materials. Everytime he needed even the slightest modification he had to wait for at least three months to have a new part machined.
I don't generally insist that everything posted be transcendent, but I fail to see even a fair point being made in the post. If the point is "buying things for an aerospace corporation is more expensive than renting an hour on EC2", okay, but we sort of knew that already.
Designing and building hardware is expensive, really expensive. If you don't take the time to follow proper procedures you are not only wasting your time, but often hundreds to thousands of dollars of parts you are never going to use. Going ahead without thinking can easily put you 2 or more weeks behind just because that is how long it takes your supplier to make what you ordered. You don't just quickly spin up an instance of a test, you have to buy or rent equipment that is often in the thousands to hundreds of thousands of dollars range.
I can see exactly why you wouldn't want to just go to Home Depot when working on equipment like what MDA makes. Say you go to Home depot and buy your fasteners, now the company has to pay for your time and reimburse the cost, which is almost certainly going to be as much or more to process than the courier charge ($30-$40) to ship the parts from the company you are making monthly payments to regardless. Then there is the additional risk of bringing in parts that could get mixed up with real production parts. There is a huge risk if you somehow get that Home Depot fastener mixed up with the special aerospace fastener you are using, the extra cost is worth it as insurance that "bad" fasteners aren't getting into the product. I can say from experience, it is ridiculously easy to get parts mixed up even when you are careful.