Hacker News Reader

Comment by zelon88 | Hacker News Reader

Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?

No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."

I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.

View on HN

There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company.

The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).

If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.

>There were other considerations like the fact we were actually buying it from large reputable company and what happened was that some employees were doing it with no involvement of the company.

It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way.

Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?

"There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company."

I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.

> some employees were doing it with no involvement of the company

Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.

Also it comes across a little naive. Oh they told us it was just an employee and would never happen again, OK!

This still sounds nuts to me.

Two thoughts:

> they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them.

I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs.

And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify.

You could also be mandating how many boards are allowed or that the SNs go on early in the build process.

Altering components to something cheaper is common practice for every company that is willing to exploit the fuck out of their customers. Look at the food industry. In EU there is control, in US thare are sanctions. But it is up to the governments because companies are to opportunistic to change. As long you can (re)sell the junk you get, you're fine, if you can't you upgrade "QA".

Both the EU and US have internal controls on food quality and sanctions.

Contrary to the impression your post gives, the EU has much more "sanctions" against food (esp. imports) than the US does, and is generally taxing much heavier and restricting many more things. Importing essentially any milk product into the EU is only possible on an exception basis. Trump has a LOT more work before extra sanctions in the US will become comparable to the EU ones.

What is often not said is that the EU not only has sanctions on its borders against food products, it actually has internal sanctions and regulations that are purely political in nature.

For example "champagne" in the EU can only be made within a particular region in France. Which leads to funny, in a sad way, consequences:

https://www.theguardian.com/business/2017/feb/15/eu-fears-in...

This applies to many products. Wines, cheeses, cakes, fruit, meat, potatoes, candy ... and is generally absolutely ridiculous:

https://www.foodmanufacture.co.uk/Article/2014/11/13/Interac...

For your other part of your comment, I am unwilling to give a real answer. Obviously the US checks it's food quality. It's made different choices than the EU, famously the US allows hormone "enhanced" meat, but it does check food quality and safety. There are heavy penalties for breaking them.

and no kinder surprise in the us.

Champagne is literally the name of the area where the product comes from. It makes perfect sense that anything that doesn't come from the region Champagne, isn't champagne... Call it sparkling wine, it's fine, but not champagne. And it applies to most local products. Camembert? Comes from a specific town, which is called Camembert. Beaujolais? The name of the province. And the list goes on. It doesn't sound "absolutely ridiculous" to me.

"generally absolutely ridiculous"

Did you look at the map? It's nearly all things with a place name in the product name. For example, "Welsh Beef". Why is it reasonable for a consumer to buy Welsh Beef and for it to possibly have come from somewhere other than Wales?

It seems to me, more like generalising the concept of trademarks to place names rather than just company names and brand name.

This is just how low trust societies function for hundreds if not thousands of years. This is also why families tend to be much stronger in these places. The American idea of a strong family is different from what I'm describing, since it just centers on immediate family. My definition of strong family bonds, implies both closeness and dependency not just on your immediate family, but even on your distant cousins on both sides. It's close if not the same to how Mediterranean and Hispanic cultures view families. I guess you can call it a clan mentality. I'm now wary of countries where families are really strong ie clans. It's not always the case, but it tends to mean the rule of law is weak and corruption is crazy.

When you can't even trust that food is real, and when you have to bribe even low level provincial government employees (can you imagine needing to bribe DMW workers or even police?) - what else is an average person to do except to treat it as the norm in order to survive? Unless you're an elite, the only other option is to leave, and not everyone has that choice. Very little is considered wrong over there, aside from criticizing the powers that be; as long as it helps your clan. It doesn't help when Western companies and governments look the other way, as we see in past news stories and even the comments here.

I'm not condoning the behavior; just explaining.

Not much to add, but want to note that you're spot on. I've encountered this SO many times over the years and this is always the explanation I'm given. At first it just made me indignant, too, and to some degree I guess it still does, but now I just try to let it go.

It's not rewarding bad behavior when you take it into account in the deal. Moralization is useless when you do business with different cultures. When you notice something what would be out of ordinary in your culture, you estimate the cost and find out how locals solve it. Different cultures have different kinks and people living there know to adjust to them. It can be fun to figure out how other cultures function.

From a naive Finnish perspective you could see American business culture as dishonest. People lie all the time and promise things that are false and you need massive amount of legalese or they fuck you over. When they say something, you can't trust it as much as you can trust a Finn (or Swede). American way is to lie to your face in pre-negotiation but then be truthful to the legally binding agreement. Germans or Belgians are more truthful beforehand and they stick to deals better without legal enforcement looming over them, but require more detailed written deals than Finns. Who is right? Should we punish Americans for their behavior or adopt their culture?

Of course not. You must understand that that the level of acceptable exaggeration, overpromise and bullshit and trickery is strictly culture dependent and relative. Someone is dishonest if they "lie this amount above average" in their cultural context. After you make an adjustment, you know how to get the truth.

Chinese manufacturers have the capability to stick to standards, take responsibility and deliver high quality stuff. It's just more dependent on personal relationship. Just placing an order has less trust as default. Either you develop "quanxi" with your suppliers or work trough some third party that understands both worlds. Third party understands what you need and knows how to get it from the Chinese suppliers because they have a relationship.

Chinese pay the cost that comes with their business culture. They can't network as fast as you can in open western cultures with more default trust between strangers.

It is called business. The company's ONLY duty is to bring profit to its investors and management is legally bound to maximise it. Now, the definition of profit may differ as well as what is profitable and what is not, but the company is legally obligated to work to bring profit and if you throw a good deal because you don't like it you better explain it to your investors.

Whether you like what your supplier does or does not is of no concern. If couple of units can be detected and written off it is treated as cost and you move on to decide if it is still profitable and whether you can get better deal somewhere else.

You let you emotions rule and it just means you are not fit for the job.

All employee actions are company actions. You partnered with a company that can’t control what it’s employees do? No internal audits to make sure their reputation wasn’t being tarnished by a few employees?! Your loss.

I understand the indignation etc etc. And the suggestion to not use these kind of companies anymore.

And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it.

One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets.

The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.)

If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.

How much do you think you would have to pay them to make it worth their while to not use cheap components (so as not to risk losing you as a customer)? It’s amazing how people never consider using economics.

Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?

As somebody else noted: it doesn't have to be the company that does it, but a lower level employee who cuts a deal. So paying the company more wouldn't solve the issue completely.

All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.

If the solution to the problem involves line checks, independent audits, personnel checks, increased pay, etc., thenyes, throwing money at the problem is a large part of the solution.

And again, you should make it the company's problem if they can't control their own employees. They can learn to put some controls in place once it starts hurting their bottom line.

> All of these things are extremely cost sensitive.

Say their profit margin is 1%. If you offer to pay them a 1% higher price, you're offering to nearly double their profits on whatever they sell to you.

Admittedly, that only works if you are a sufficiently large portion of their business. If you're a tiny percentage, then it may take a lot more to motivate them if their other customers aren't willing to pay more.

You have to pay them as much as it would cost to manufacture in the Western country doing the out-sourcing...so unless the cost of these extra quality checks outweighs the cost of on-shoring you might as well measure angular momentum.

Madness if you ask me but there we are.

> I understand the indignation

IMO this is not indignation, it's supplier sourcing 101.

> If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.

I will definitely agree that holding suppliers to standards regarding consistent output, unadulterated products, and conducting audits all make production much more expensive.

But if you're a device manufacturer, these sourcing controls are key to shipping quality products.

lol. you haven't worked with a vendor in China before, have you?

But make no mistake, this yet another globalized race to the bottom, but one that hits capital (or the end consumer), rather than labor or the environment.

the indignation was not that. you are still missing the point.

after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.

Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.

> Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.

I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened.

However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.

You all recognize the irony, right?

PCI DSS allows for "Mitigating Controls" if you need to deviate from specified requirements, provided it is well documented and is equal to or greater in security. Doing teardowns to review circumspect hardware, and applying one's own tamper protection deal (and with accompanying documentation and tracking/logged information) would very likely be sufficient to maintain complaince.

>The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty

Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).

OMG, that is so so so so brilliant! I'm going to go tell my boss right now that the suppliers better suite up, otherwise we are going right to their more expensive, less experienced vendor and will delay our product launch for a year....and likely still suffer the same problem. Those Chinese vendors better shape up or we're going to really teach them a lesson by driving ourself out of business right quick!

Just imagine for a second if someone was advocating on HN for the use of a web framework that has knowingly allowed itself to be compromised to steal passwords because it was cheaper to use.

How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.

So no more using Intel?

You mean like couple of well known platforms that let you reach their users and in the meantime steal, mine and monetize user information?

You can be as sarcastic as you want, but these stories are absolutely indicative of a much larger problem. It might be beyond the capacity of one company to fix, but in the aggregate they represent a serious political and economic threat and need to be dealt with one way or another. Preferably by literally anyone other than Donald Trump.

Speaking as someone who is _not_ a citizen of the USA and is well aware of his issues, he seems to be first US President who actually appears to have the guts to do something about it. I doubt an establishment politician would ever have imposed tariffs on 200 billion dollars of China imports.

The tariffs aren’t the problem, it’s the stupidity of going it alone and alienating allies that could have turned the thumbscrews with you on China to get them to really capitulate.

If he stuck to just China I would agree. But by trying to change up numerous different agreements, he lost the goodwill needed to have a collective crackdown on China. Even if he is the first to be willing to pick these sort of battles in a long while, he has a new issue of picking too many battles. Seems like a Goldilocks problem.

Leaving an abusive relationship tends to be scary, especially if the victim is convinced that the abuser is the least bad. Doesn't mean the thought of ending that relationship should be dismissed out of hand.

I used to work on the server-side stuff for telecom devices. We designed hardware that went into customer homes, but only downloaded the (encrypted) firmware upon home activation as otherwise the Chinese manufacturers would have ripped us off and sold them to telecom companies without our cut.

So, realising they could copy our hardware, but didn't have our software, they responded by trying to hack my servers, multiple times, from the same IP they sent manufacturing data from. A quiet word with their management would stop it, and it'd start again a couple of days later.

These people have no shame, and if we are going to go for lowest cost at all times it is what manufacturers should expect to happen.

I am reading it as "we don't any any option outside China and we have to live with this". I guess even after adding cost of doing additional of products you buy from China its less than what you would pay to buy it from other place.

Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.

Sounds like a libertarian oasis.

>If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.

he's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive.

I can't believe the parent comment is so brazen. It makes me physically sick.

I guess it's news to you that the financial sector and especially banks in fact do exactly that:

https://www.creditcards.com/credit-card-news/credit-card-sec...

But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.

knowing about it and willfully facilitating it are two different things

like shoplifting. Most large chains know exactly how much "loss" they experience, and they just raise their prices to cover it. Theft and fraud is a part of life unfortunately...

This problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.

Honestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one?

Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.

Maybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)

Profit. Changing suppliers is going to cost money.

Or: you can expect this kind of shit from pretty much any supplier. Even if you go to a western company, they all source their components from China anyway.

A good read is 'Poorly Made in China'.

The product is different, but the problems are the same.

* They drop changes and problems at the last minute, so you're over a barrel with your customers.

* Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise.

* Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far.

* The new supplier will 100% play the same games.

* Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way.

* Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business.

* Are you going to admit to your customers and bosses that your products were faulty and you knew?

It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.

> * Are you going to admit to your customers and bosses that your products were faulty and you knew?

If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical.

When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice.

Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure.

All this tells me is that Made in the USA is more valuable than I once believed.

Not having the backbone to reveal it when you discover a fault is not what that comment was getting at

I think you’re right about the value of Made in America and societies where honesty (even about problems or risks) is valued.

The book I referenced included a number of instances where factories actively undermined individuals so that they had very little choice than to either go along with the situation, or press the ‘nuclear’ button and scupper the entire arrangement, including their own livelihood and career. That’s a tough choice for people to make.

Probably the cost per unit is too attractive

Also it forces their customer's customer to essentially subsidize production cost.

Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage.

As a result this immoral company externalizes aspects of their cost structure to greater society.

This is also such a good anonymous story that I'd give decent odds it's made up.

He's not anonymous, though.

I've heard very similar stories from people in the card industry before. I don't think it's made up, unfortunately. Card readers from China are systematically untrustworthy.

Using the technically incorrect term "angular momentum" rather than the more correct term "moment of inertia" made me think the same thing. I would think a person tasked with building such a device would know their physics well enough to use the right term, but I may be wrong.

Maybe he is afraid of giving too much info away. By mentioning it, now the opposing side will be measuring angular momentum.

So now he needs to make a new test. Ow that that is out of the bag

The fact that they didn't just run the units through an xray machine would seem to support your theory.

Reply on news.ycombinator.com