Hacker News Reader

Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.

It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible.

I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed.

I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.

> Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed.

No. Their job is to use the resources they’ve been allocated to manage risk within the organization to the risk level senior management has agreed to accept.

Maybe that shouldn’t be their job, but it is. There isn’t a security team on the planet that gets to dictate security to the rest of the organization or unilaterally make decisions that influence the running of the business to achieve the level of risk mitigation that they themselves would prefer to attain.

That is putting aside the fact that defending against a determined nation-state adversary is a nigh-on impossible task that would require countermeasures like ‘don’t connect to the Internet at all’, ‘don’t hire anyone you haven’t personally known since childhood’, ‘hand-deliver your product to your customers’, and other equally impractical mitigations.

Nobody outside of Solarwinds knows if their security team succeeded or failed in the mission they were given.

I hear what you're saying and I appreciate that perspective. I agree, and there's also something slippery in that argument taken to its extreme that we need to be careful of.

Some people here on HN made the same argument when Equifax leaked personal information about millions of americans. They said it was ultimately management's fault and not the engineers' fault for not allocating enough resources to security. And the same argument was used by the engineers who made the Therac-25 radiology machine. In that case, software bugs resulted in a handful of deaths due to lethal radiation.

Upper management can't be responsible for everything that happens in their business. Engineering isn't their job or their expertise. Thats why they hire software engineers and security engineers - to be the local experts. We need to bear responsibility for the decisions we make in our field. And engineers have a duty not just to the companies we work for, but also to society at large. If we leave our personal judgement at the door in the morning, we fail in our duty to society.

To go back to the bridge metaphor, if a bridge falls down, its not good enough for the civil engineers involved to blame management for not giving them enough time / budget / whatever. They also bear some responsibility for the disaster. This has been enshrined in case law too, at the Nuremberg trials. "I was just doing my job" wasn't considered a good enough excuse for the guards in WW2 concentration camps. These are big examples, but I think the principle is fractally true.

And the inverse also holds. Praise and blame go together. The biomedical engineers in the labs also deserve praise for the covid19 vaccines they've invented, even if upper management told them to do it. We aren't management's slaves.

The difference, when it comes to civil engineers and building bridges, is we as a society have recognized their expertise and made it illegal to build a bridge except as designed by a civil engineer.

A more apt analogy, in my opinion, to the day to day realities of managing production applications and infrastructure is the regulation surrounding the maintenance of certified aircraft. There are minimum competency standards that are enforced by law, it is unlawful in almost all circumstances for a non-certified person to perform any maintenance or repair on a certified aircraft, and, crucially, an aircraft cannot return to service unless a certified mechanic signs off on the repair. Not the CEO of the company that owns the airplane, not some middle manager, only the expert (mechanic and, sometimes, inspector) can sign off on returning the plane to service.

Without that kind of legal cover, management can and will steamroll over anybody who is impeding their initiative of the day.

Sure; but politicians don’t know anything about technology. They usually don’t even decide what’s right and wrong. They take what culture has decided is right and wrong and codify it in law. The law is a trailing, not a leading indicator of ethical practice.

Do you think planes were falling out of the sky left and right before those air safety laws came into effect? No. The engineers at some companies pushed for sane, safe practices first. Later they were adopted by the industry and later still they were enshrined in law. Before those laws were passed, airlines still had a duty of care to their passengers, ethically and (I think) legally.

Likewise it’s up to us to decide what sane, secure software engineering looks like. Not politicians. Not management. It has to be us. Nobody else is qualified to make those choices. At some point those ideas might be codified in law; but we need to figure out what that looks like first. (And to be clear what you’re arguing for - imagine the reverse. Imagine if inventing security best practices was outsourced to politicians!)

The idea that management should feel free to steamroll over their own employees’ judgement for the sake of the initiative of the day is toxic. And that’s exactly the sort of work culture which creates global security issues like this one. Of course a balance has to be reached, but you don’t do anyone any favours by being management (and the law’s) highly paid keyboard.

> Do you think planes were falling out of the sky left and right before those air safety laws came into effect? No.

That is exactly what was happening. In 1924, prior to the introduction of the first federal aircraft safety regulations in 1926, there was 1 fatality per 13,500 miles for commercial flights. Between 2000 and 2010, the average was 0.2 fatalities per 10 billion passenger miles.

http://www.parabolicarc.com/2016/03/03/early-aviation-safety...

Thanks for those numbers!

Imagine yourself as an aeronautical engineer around that time. You have a sense of what good safety practices could look like - you’ve been to conferences and talked to your colleagues, and you have some thoughts yourself. But management at your airline doesn’t want to spend the money.

Would you argue for meekly going along with management’s choices, knowing those choices will kill people? I would say, if you did, you would have blood on your hands. We’re people first and employees second.

The stakes are lower and there’s a middle ground here. But you have a voice, and usually more power than you think. The siren song of dumping all responsibility for your actions onto upper management makes you into a victim and a child. It’s bad for society, usually bad for your company in the long term and bad for your psychological health and development. And a disaster for your professional development.

I don’t know if that lands with you, but it’s certainly a lesson I wish I could give to myself over a decade ago.

Keep in mind that although that regulatory certification approach does indeed increase security, it can also greatly constrain innovation. It's most appropriate once systems have matured a bit.

FWIW, there was an average of one steam boiler explosion EVERY WEEK in America (frequently with loss of life) when the ASME was founded to set standards for safe design and certification. So it can take considerable pain before efforts like take off. The FAA had the advantage of already having that kind of certification as an already established model, plus airlines were eager to have a stamp of safety approval.

It's hard to see how a "security certification" standard could really provide much assurance in today's world - witness the inadequacy of FIPS, SOC, the outright laughable HIPAA, etc. PCI is one of the only certs that really provides any kind of assurance, but it's driven by the banks that insist on it being there to protect themselves. And recent events have shown that we have way too much centralized control of electronic payments processing already...

Those regulations wouldn’t be needed if our industry could govern itself and act in healthy, responsible ways.

Unethical data collection leads to regulation, which leads to less innovation in the long term. Fight for ethical behaviour in your company and team and we can, en masse, delay the need for that.

And as for regulation, if it were up to me I’d make EULAs mostly unenforceable. Which would give leave for the people and companies affected by security breaches like this to sue anyone and everyone responsible. Which, by the way, is how the law is designed and how it works in every other facet of life. Sell a faulty ladder that kills someone? Get sued for negligence.

But senior management is responsible for establishing the resource budget and dictating the requirements. Given infinite resources, conceivably an engineering team could make an infinitely secure product. But nobody has that (except maybe the military). Like all other computer problems, this comes down to the constraints and requirements.

> "We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material""

You might be interested to read about the Morandi Bridge collapse in Italy: https://www.engineering.com/story/italys-morandi-bridge-coll...

"When the Morandi Bridge was built, encasing a cable in concrete was innovative.", "the decision to pre-stress it was debatable", the concrete meant they couldn't check for cable rusting underneath, the concrete could have been supplied by the Mafia and under specification, "“We have used materials that are destined to deteriorate quickly, like those of the bridge in Genoa,” Bercich said in a post-collapse analysis".

"Settimo Martinello, director of bridge inspection company 4EMME Service, told CNN that there are “about 15 to 20 bridges collapsing every year” in Italy."

From another source: https://www.theguardian.com/cities/2019/feb/26/what-caused-t...

"In the 1960s little was known about the interaction of materials, or the effects of pollution and climate on corrosion.", "Morandi himself was surprised to see the structure age faster than he had anticipated. In 1979 he issued a report detailing a number of interventions to protect the structure against pollution from nearby factories"

It's not a collapse on its first day, but it undermines your suggestion that bridges are a solved problem, designed once with traditional methods, are well understood inside and out, and therefore never collapse.

Or if that's too old, how about the celebratory Millennium Bridge in London, a footbridge opened in June 2000 and closed two days later because it couldn't withstand the footsteps? https://en.wikipedia.org/wiki/Millennium_Bridge,_London#Open...

The Millennium Bridge was altered because the swaying was "alarming" but there's no particular reason so far as I know to think it would have caused structural failure, it's just not useful to have a pedestrian bridge that some fraction of pedestrians won't cross because they're scared, it isn't a theme park attraction.

I think a better analogy is the human analog of what this is - spying.

Not noticing a spy in your company, university, army headquarters for 15 months is hardly an indictment of one's counter-espionage - especially if as it seems they did very very little till Feb, and did it very carefully.

seems is working hard here.

Apart from what the sibling comment mentioned, it's not a great analogy because we have a very limited things that can go wrong with a bridge. This is knowledge shared between all engineers - you analyse the design for multiple known forces and you're done.

Compare this to the CI systems designed to take unknown code and run it in a safeish way. In a bridge analogy it would be something like "one of the screws turned out to be a remote controlled drilling device which hollowed out parts of steel without visible changes" - of course nobody would notice that for some time.

...it's not a great analogy because we have a very limited things that can go wrong with a bridge.

Security engineering is still evolving.

But it seems reasonable to say that if you engineer a thing that may be subjected to unlimited, unknown stress, you engineered it wrong.

These many large companies giving themselves a single point of failure by installing SolarWinds binary, not knowing the contents of home-phoning and so-forth is a terrible security engineering solution. Sure, maybe they "couldn't have done better under the circumstance" but someone allowed the circumstances to happen too.

> But it seems reasonable to say that if you engineer a thing that may be subjected to unlimited, unknown stress, you engineered it wrong.

A computer is that by design. Anyone using computers is using a system subjected to unknown and unlimited inputs, especially when connected to the internet. (But it can be exposed via employees if it isn't)

Any controls we have at the moment are "this will not happen (under the assumptions we're currently aware of)".

If those companies didn't run this monitoring solution, they would use a different one. Unless the system is perfectly covered with access policies (none are), the monitoring solution is a global access to enterprises. You can mitigate the impact, shard things, etc. but there are still many cross-cutting concerns which will (suddenly?) turn out to be a single point of failure.

The problem is that Security Engineering is still treated as if it's a separate thing that is evolving.

It's systems engineering, where security is a necessary feature that's being grafted-on after-the-fact.