Tech audit of Colonial Pipeline found ‘glaring’ problems | Hacker News Reader

Tech audit of Colonial Pipeline found ‘glaring’ problems

apnews.com

65 pointsjtdev28 comments

I would call Colonial Pipeline a _negligent victim_ like most other companies that found themselves in the same situation of being hacked and held for ransom. Negligence stems from the fact that they consider Cyber Security as part of IT sunk costs that they do everything to minimize. Until there are real consequences from personal career costs to CEOs all way to jail time nothing will change. All these security consultants are part of Cyber Security Theater to shift blame away from company leadership and limit the media blast radius.

Of course, but let's not pretend otherwise about other companies that have not yet found themselves in the same situation: qualified negligence has been the name of the game in both IT management and software development since at least the '90s.

> Colonial says it has strengthened data-loss-prevention defenses with three different software tools that provide alerts when data leaves the network.

So they hired a consultant to produce a report. Then they purchased enterprise software. In my experience, this will not lead to a good outcome.

Ideally, it shouldn't be the case that Colonial needs to have information security expertise on staff, just like ideally, they shouldn't need to have physical security expertise on staff.

But in the real world, that obviously isn't the case. In 2021, every company is a tech company, and information security isn't something you can outsource. I wish that weren't the case and that we did a better job building software and designing systems to be secure by default. But we're nowhere close to that world.

Here is the two step solution.

1/ Digital security insurance

2/ Digital Security Jump Starting… like when your battery is dead you connect it to a functioning battery with jumper cables. There needs to be a service industry for bringing in fresh IT security systems and (re)establishing baseline IT security that meets the standard of the digital insurance industry.

Looks like you just created a great job, startup, opportunity. You should run with it! Great idea.

Step 1. Fire the current CEO

Step 2. Tell the new CEO candidates why the last guy got fired.

That’s all the board has to do. You can be damn sure the next CIO and CSO will be empowered to build and run a security-first organization.

On the other hand, if the current CEO is let off the hook, nothing material will change as hacking will be treated as a PR problem to be solved again in the future by PR people.

Step 1. Fire the current CEO

Yeah, in an ideal world.

No, in the real world he's doing exactly what the owners want. They are typical of the private equity crowd.

This pipeline was built 60 years ago. It's been paid for many times over. Now it's being managed for maximum cash extraction.

The Colonial board is motivated only by money. My alternative solution would be:

Step 1: Fine the pipeline company $1,000,000,000 because they're deliberately being a bunch of incompetent asswipes.

That's the way you get the attention of those people. That's the only way.

Sadly, the US Supreme Court has taken a rather dim view of "excessive fines". Still, I'm sure that smarter people than I can think of some way to hit the company's board of directors with a financial clue-by-four.

I hear what you’re saying but I think it could also be done with the carrot instead of the stick. I believe Colonial’s largest shareholders are Koch, KKR, and Shell. If Secretary Granholm privately tells Messrs. Koch, Roberts, and Mackenzie that the US government would look favorably on a CEO replacement, they’ll each ask for something proportional their firms want that Granholm can help with, and Blount will be out the door tomorrow with his golden parachute.

This is the same pipeline that leaked over a million gallons of gasoline without Colonial even noticing. Two teenagers reported it after smelling it while out riding ATVs.

https://www.wcnc.com/article/news/local/ncdeq-colonial-pipel...

If anyone is in the Atlanta, Georgia, area, Colonial Pipeline has (had?) a posting for a Cyber Security Manager:

* https://www.daybook.com/jobs/jDuPoWB4gbFMpS8x5

* https://www.indeed.com/jobs?q=Scada%20Cyber%20Security%20Man...

* Via: https://old.reddit.com/r/sysadmin/comments/naq415/

Can't imagine why anyone would want to take that job.

These types of attacks don't just "happen", they're caused by chronically underfunding/ignoring security staff. Generally that means being security staff at one of these companies is probably going to be a crap job.

Afaik they stopped the pipeline itself because they could not figure out who to bill.

When you are transporting cargo worth $X, and your fee for doing so is $X/100, it really matters if a small percentage of clients take more of the goods than they should...

In many cases, the client might not even have their own records of volumes - they just run the pumps till the tanks are full, and then wait for the bill from Colonial.

If Colonial can't produce said bill, they have made a massive loss.

That's outstanding. They avoided the loss! If there was ever a better example of cutting the nose in order to spite the face, I never saw it.

If we've automated ourselves out of the capability to operate for a few days with paper and pen, when the alternative is a nationwide crisis where untold million humans were impacted by resource shortfalls...

How many pennies did we save by shutting down the pipeline, and did we successfully externalize the costs of the worst of the effects of the hack... if so then job well done! LOL

Never change, Colonial Pipeline! Never change a thing.

This was my original theory.

I was looking at Google satellite photos of their Roanoke, VA site, and you can see there is a regular gas station nearby, but is connected and has a giant loop for tanker trucks to fuel up.

In past jobs where I had driven a company vehicle, there was always a special gas card with a PIN, and when you would swipe at the pump, it would ask you to enter the current mileage of the vehicle.

If you cannot verify or authorize transactions, there is no point in giving away free product.

The tech audit was 3-4 years ago and colonial may or may not have heeded all the advice from the audit. They are claiming that they have followed at least some of the recommendations and that the pipeline control system was air gapped and not affected by the ransomware.

>Robert F. Smallwood, whose consulting firm delivered an 89-page report in January 2018 after a six-month audit. “I mean an eighth-grader could have hacked into that system.”

Can't say I'm surprised. This is what 0 regulation and free market capitalism looks like. In theory they would secure their systems because a hack would be bad for business, but in reality with bribes, lobbying, and networking, once you have a contract you pretty much have it for good so there's no incentive to spend extra money on things like security, environmental safety, etc unless it's required.

> This is what 0 regulation and free market capitalism looks like.

Nope, that’s nothing like what this is. Oil pipelines are heavily regulated, with ZERO competition. As government protected monopolies, they’re the exact opposite of a free market.

I was exaggerating. Sure they're regulated, but "heavily" is also an overstatement compared to how much they should be regulated. This is no different than how when talking about installing a new pipeline they say there's all these fail-safes, protections, monitoring, inspections, etc, but then oil leaks happen and it's found that the pipe had been deteriorating for months if not years which shows they actually aren't under the scrutiny that's claimed and there aren't protections in place. Or they lobby to have their pipeline self-inspected similar to what Tyson was able to negotiate with their meat plants. That's just one example of many. Sure in theory there's all this regulation, but somehow we keep having disaster after disaster that would have been avoided if all these regulations were enforced.

> Sure in theory there's all this regulation, but somehow we keep having disaster after disaster that would have been avoided if all these regulations were enforced.

Making new regulations is not helpful when the problem is the existing regulations are not enforced.

But making new regulations that also won't be enforced is the easiest way to show that Congress and regulators are doing something. Following fiery hearings of course.

"We've got to protect our phoney baloney jobs!"

outsider here, but I suspect the time and effort is put into controlling your debt and in-fighting among senior people; anything in operations is "blue collar" plus many leaders are older alcoholics who do not learn well by that stage of their career. They are not dumb, but something outside of their experience may never be questioned, or easily glossed over. They have people coming at them who want to sell legal or business services that will say anything the decision maker wants to hear, including actually wrong details, to close a contract and get paid.

I will add to this comment that I am very, very concerned that incidents like this lead to changing the rules on communications for hundreds of millions of people. Why do others pay for your mistakes in guarding your wealth for yourself?

I agree, most of senior management at places like this are ageing country club members who know how to maintain their monopoly by wining and dining regulators, but are incompetent at running a business.

Because government agencies (ie: literally not free market capitalism) never get hit with ransomware...

https://www.bbc.com/news/world-europe-57111615

https://www.sungardas.com/en-us/blog/ransomware-attacks-on-u...

So yeah blame capitalism and not inherent human nature that applies to every large scale system built by humans.

> said he prepared a 24-month, $1.3 million plan for Colonial.

Sounds a bit cheap to fix such a thing.

I'm sure it would cost at least $5 million more now ;)

Aging private infrastructure built by tycoons.

Once the sizable venture capital is paid back the original shareholders rake in the bucks and money is no object when it comes to protecting their cash cow.

But future generations of shareholders pay a premium market price based on the cash from the cow at the time, which can be quite a favorable investment, but nothing like the VC bonanza.

Public, or private shareholders as in this case.

The maintenance, modernization and dedication to integrity that the original shareholders could easily afford might still end up out-of-reach to future generations.

But the business started out so good and "nothing changed" so it can be ignored.

Where grandpa took great pride in spending millions per year maintaining the assets which is so much less than they were spending building the company, Thurston Howell III just has financial people looking at his numbers and couldn't build anything his own self anyway. Plus if it all does go "down the tubes" he'll still be fine regardless.

Wear & tear plus obsolescence creeps in undetected until it rears its ugly head and they say "who knew?"

It would be really nice if they let "Robert F. Smallwood, whose consulting firm delivered an 89-page report in January 2018 after a six-month audit" do the audit again.

It doesn't really matter how much money the company throws at a problem if they don't fix it.

Reply on news.ycombinator.com