Hacker News Reader

Comment by onetimeuse92304 | Hacker News Reader

Here some more information about what really went down, technically, behind the scenes: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...

"One member of the development team, David McDonnell, who had worked on the Epos system side of the project, told the inquiry that “of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”."

(Just in case somebody says I am putting blame on developers) Obviously, the responsibility is firmly on management. People making code bugs should not be held responsible for other people going to prison for it.

View on HN

> Obviously, the responsibility is firmly on management. People making code bugs should not be held responsible for other people going to prison for it.

This is a controversial opinion but I disagree, at least to a point. Managers don’t really know what we do. The only people who really understand the engineering trade offs involved are engineers. When lives are on the line as a result of our work, we shouldn’t be insulated from the consequences of our choices. That’s not good for society and ultimately not good for us. We change the world with our work. It’s healthy to understand and own the consequences of that.

The law agrees in parts. The principle of tort law is that everyone is responsible for foreseeable harm caused to your “neighbours”. Your degree of responsibility - and in turn liability - scales with how much expertise you have in the domain. An expert should have been able to foresee the harm more than a novice. The senior engineers on the team should have done better. I believe they are at fault.

(IANAL, this is not legal advice, yadda yadda)

> Managers don’t really know what we do

That looks a lot like incompetent management.

Management aren’t trained engineers. They shouldn’t have to be. We’re the experts in the room. That’s literally what we’re hired for. We should act like it and stop trying to pass blame to other people.

Whilst I agree in some respects, the biggest gulf to me is between companies like Stripe who successfully manage a large chunk of the words commerce (led by a brilliant engineer-CEO) and the 'IT Projects' that seem to plague the public sector here in the UK.

My point is that particularly in the UK we have this culture that the Geeks should just do their job and let us Business Types take care of the rest. Countries like Germany have a much higher respect of technical people and qualifications e.g. it's very common for CEO's to have PhDs

It’s not about having a PhD. If I pay to have a house built, the point of paying money to a construction company is that I don’t know how to build a house. I’m hiring them because they’re experts and I’m not. If the house falls down and kills someone, the construction company is responsible. I don’t know how to tell if a building is safe because I’m not a working engineer. (And even if I was, it’s still the construction company’s job to build my house properly. That’s what I’m paying them for.)

In software, it’s the same. Your employer hires you because they need an expert. It’s your job to take responsibility for the software you write. Even if the CEO has a PhD, it’s not the job of senior management to review your code. And - trust me - they don’t want to. Instead take responsibility for your own work. You are worth your paycheck because you know how to build software well. Stop trying to shirk your job.

I think your analogy is a bit off, construction projects are primarily in their nature one-offs. If I was a restaurant owner and needed a new restaurant built then sure I'd expect someone to take all the responsibility. If I wanted a new chef to design a menu, keep standards up and also roll out new dishes then the restaurant owner should know about food, it's their "bread and butter"

Most companies nowadays are more like restaurant owners i.e. computers are a core part of their business. They can't and shouldn't rely on engineers to get things right all the time because humans are flawed and therefore their code will be. The CEO has the responsibility to put people and systems in place to account for this.

I hear what you're saying; but I think its hopelessly naive and unrealistic to expect the CEOs of large companies (like banks, telcos, insurance companies, etc) to learn enough Python that they can make competent technical decisions.

Of course, if the CEO defunds their infosec teams, they bear the blame when they get data breaches. But I also don't think the engineers on the ground get to avoid blame when their crappy software leaks customer data. "I thought about making it secure but we had a deadline so I didn't" is a lazy excuse. Blaming management for everything is a lazy excuse. We're engineers. Not typists.

Of course, we shouldn't expect the CEO of the PO to dig into code, and we shouldn't excuse the standards of engineers.

To continue the restaurant analogy, if customers are consistently getting food poisoning due to the sloppy hygiene of the kitchens then the buck stops with the restaurant owner. They'd need to diagnose the issue, put in place better training and supervision of the cooks and have systems to regularly check that standards are being met

Right. But also, if a surgeon doesn't wash their hands and a patient dies as a result, the surgeon is at fault. Not the hospital's management team. Or, not just the hospital's management team.

Yes but this was a case of a very filthy/corrupt hospital

The title of a singular "glitch" is very misleading in this case, as this was a cacophony of cock-ups and cover-ups for 20+ years

Upper management doesn't have to be trained engineer. However immediate direct management of development team definitely have to be trained engineers to be competent.

Also, a good manager should detect if a team contains a few "dead weights". It eventually always lead to frustration from more competent team members and this is something that can be detected and addressed if you talk to your team members.

The only thing that may be difficult for managers in the public sector is firing people unless they are contractors. I don't know how it works in the UK but I have worked in gov agencies in another european country and firing someone who was just working badly (to the point of wasting time and energy of others) was almost impossible. And sometimes upper management would throw unfit people in your team just because they needed to put them somewhere. I had a few time wasters in some of my team that had been put in our team by the unemployment agency. Basically they had spent money on them taking "classes" (which were just Microsoft certification classes) and threw them at us while most of them weren't even interested to begin with. You really had to do direct fault to be fired and it would take months or even years. The only one I have seen being fired was a project manager who booked fake meetings to go play golf during office hours.

A good jockey isn’t a guy who can run like a horse. A good manager doesn’t even need to know how to program, let alone be an expert debugger who can spot errors made by their team.

A jockey needs to know what a horse is and how a horse lives, trains, runs, etc.

A manager that has no technical knowledge is useless, like a jockey who doesn't see a difference between a poney and a horse or distinguish a healthy horse from an injured one.

> A manager that has no technical knowledge is useless

Thats taking it way too far. Sure; its useful for managers to understand programming concepts (deployment, testing, etc). But the job of a good manager isn't managing code. Its managing people. Making sure Sally is happy in her new role. Setting up a meeting between Jake and the sales team so they can get to the bottom of that important bug. Helping mediate that conflict between the software team and the design team over what features to prioritise.

A manager is hired to be an expert at humans. Not an expert at computers.

If managers are hired only because they're experts at managing humans, then they're inadequate for the role of managing a dev team. These guys are not just managing people, they're managing processes that they have to know something about to manage effectively.

You can't expect managers to have the same level of technical expertise as the people they are managing. In your view of the world, specialization doesn't exist.

I'm not saying i expect that at all. I'm saying they should have some knowledge about what they are managing than just being a people person. Wouldn't you hate your direct manager having no clue what a code review is? What scrum is? How features are estimated? How estimates can be widely inaccurate? What tech debt is and why it should not be swept under the rug?

>An expert should have been able to foresee the harm more than a novice. The senior engineers on the team should have done better. I believe they are at fault.

Well, the reality is usually at these large software development agencies that senior engineers are prevented from doing what they think is right.

For example, they might have been pressed to deliver new features in an extremely inefficient system. They might have been inundated by low quality code from less experienced devs. They might have been busy with communication with stakeholders and unable to do much about it.

So singling out these developers would be like singling out couple cops for being racist when the entire Police department is known for racism. You know, technically you are right but that still does not seem to be right thing to do in case of a systemic problem.

The question would be if these developers had any way of knowing the consequences of what they were doing.

Also, "good developers" is a relative term. In an organisation like that a "good developer" might simply describe a person that is at all capable of writing working code. It does not mean they were experienced or aware what is happening around them.

It is the responsibility of managers to recognise the issues with the environment their people are working in.

> Well, the reality is usually at these large software development agencies that senior engineers are prevented from doing what they think is right.

Can you imagine how that conversation would go if the engineers were personally liable? “Hah you want me to sign off on that? No - I don’t want to get sued when it inevitably goes wrong. I’m sorry boss but I won’t do it. And you won’t find another engineer in the building who will. Lives are on the line. We either do it properly or we leave it alone. I’m not sticking my neck out to make the business a quick buck.”

> Also, "good developers" is a relative term.

Legally, as I understand it the courts look at job titles, education and experience to make a judgement.

I agree, but only if we have the same standing and legal protection as professional engineers.

If I’m going to be legally responsible for software bugs I must have the legal right to tell management that their timelines are not possible, and that they can’t deploy software I won’t sign off on.

That and for outsourced software the executives become personally responsible.

I agree with every one of those points. There were two massive data breaches in Australia recently. I’m livid that nobody was held liable for damages over it.

So, you're telling me your manager doesn't care if the software is broken and has legal consequences for it's users?

That strategy only works if the software itself is inconsequential.

The only one who can have responsibility for anything is the one who has the authority. So long as developers are in a position of "just get it done or you're fired" as well as being outsourced to save costs, they have no authority in this and therefor zero responsibility. If management "doesn't know what we do" and doesn't want to have the responsibility then they have to give us the authority to say "no, this is not going to be done tomorrow and we're not cutting any corners".

> just get it done or you're fired

Who actually works in a job like that? Do you seriously think you’re a slave to your boss, with no personal agency? Do you think your employer wants you to be feckless? Do you think that’s good for your career?

Your capacity to take responsibility is the differentiating factor between junior and senior engineers. Learn to step up. If nothing else, your pay check in 10 years time will thank you.

Incentives do matters. It should be pretty clear to anyone on this forum. The parent is absolutely right. Responsibilities do matter. In this case, there was clearly a systemic problem that was repeateadly ignored by mulitple levels of hierarchy.

You position is a position of principle. When peopke lives are at risk (boeing, fukushima and yes even the post-office) pragmatism must prevail. Those in charge must pay the price, otherwise you incentivise financial results over everything else. People die? Oh that's because Greg in engineering is an idiot, burn him!

The problem is with insufficient QA and processes in place. If a coder delivers code that is insufficiently tested before deployment, the onus is on the product owner. It's the reason "owner" is synonymous with "person who is responsible."

I don't know where you live, but in my (extensive, but rather local) job experience so far, it's possible to raise concerns and propose solutions, etc., but once the IT director or VP of whatever says "we're not going to do that", it's over. You have to work on something else. Or find whistleblower protection, and good luck with that. Otherwise it'll be "Can you step into my office for a minute? Close the door, please."

> The only one who can have responsibility for anything is the one who has the authority

So if a cop gets an adresse wrong and is a bit too trigger happy and ends up killing innoscent people. Its their chief of police who should go to jail because they told them to go arrest a suspect? Unless we change the system to allow cops to just do whatever they want whenever with not leadership?

The idea that just because a programmer doesn’t have complete autonomy over their work that they suddenly become unaccountable for negligence and errors is ridicules.

I agree, except that it's important to remember that code by itself is as meaningless as programs with no side effects. A bug becomes a problem when it's a part of a running program.

> Managers don’t really know what we do.

Uh.....

> The only people who really understand the engineering trade offs involved are engineers.

Maybe you should have engineering managers who understand the subject?

From that article (and a few others)

> In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems, the inquiry heard.

This has always bothered me. Sure, it's possible to build APIs that audit access completely. But I can easily write code that circumvents those APIs. Code isn't like a building where the walls are impenetrable and the doors the only possible access points - we can redecorate without ever touching the door. Building in an unaudited backdoor for operators seems bad, but if you can edit the source code the backdoors are infinite.

There should be application level auditing and database level. The people with access to managing the database level auditing should be extremely limited.

Listen. We all know what should have been done.

They were not able to do the first thing about running a transaction (ensure that one side of the transaction isn't executed multiple times). What you are saying is an obvious thing and yet it probably is well beyond the maturity of the team that was working on it.

Interestingly, it seems they may have built their own master-master xml-based database. It's easy to guess that they didn't add an audit feature etc.

They were using Riposte from https://www.eschergroup.com/riposte-platform/ and Oracle.

They were using dial up ISDN lines to send the data back, but Riposte didn't support that, or scale to 20k terminals, so that was all new code

In general they had a distributed database that couldn't do ACID

https://www.postofficetrial.com/2019/12/fisking-horizon-tria... https://www.computerweekly.com/news/252496560/Fujitsu-bosses... https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...

Accounting 101 use journal entries to correct mistakes. Dont edit original records... Have a transaction log...

> there were probably three or four who just weren’t up to it and weren’t capable of producing professional code

See the other discussion on HN front page about coding tests in interviewing.

Horizon was a child of the 90s. The software industry has changed a lot since then. Back in those days only Microsoft was routinely requiring programmers to code during the interview, so hiring was nearly random. Software teams often looked like that, with a tiny number of people who could write working code covering for many more who just couldn't at all.

The Daily WTF dates from this time. It's full of stories like the Brillant Paula Bean:

https://thedailywtf.com/articles/The_Brillant_Paula_Bean

You don't hear stories like that much anymore. The industry settled on testing concrete skills before hiring, and that washed out a lot of the people who previously managed to get hired into projects despite not being able to code properly. Whether Fujitsu does it or not now, no clue. But that situation wasn't unusual back then.

I would say another child of the 90's is management / users being too trusting of technology which IMO is the real reason for this mess. These days, everyone knows that software can make mistakes and one shouldn't rely on a single system / data point when it comes to critical decisions (ie. sending someone to jail).

I don't think that was so much the case in the 90's, attitudes seemed to have been that these sorts of systems don't make mistakes, and therefore can always be trusted. I look at this as the 90's version of "The Titanic is an unsinkable ship"

EDIT: I would add that you can rely on systems / single data points for some decisions. How I see it is trust in a single point of data goes down as the criticality of the decision goes up. If someone sends you a message to meet them for lunch downstairs, that decision has low criticality and therefor you can rely on a single data point / application to make the decision. However for more critical decisions (ie. should we send this person to jail for stealing money), trust in any system should be low by default and a consensus from multiple data sources is required.

> that situation wasn't unusual

Um, I'm working as a professional computer programmer today, in 2024, and I can assure that programming continues to be replete with incompetent programmers who are not capable of producing professional code.

> People making code bugs should not be held responsible for other people going to prison for it.

Why not? If I'm the single developer and seller of this app, should I not be held responsible? What if there's also a QA person? Two of each?

Should the person selling or marketing the app be held responsible instead, even if they aren't technical? Why are the developers who didn't care enough to double-check their code free of responsibility?

> If I'm the single developer and seller of this app

In that case you are also the responsible manager or product owner.

> Should the person selling or marketing the app be held responsible instead, even if they aren't technical?

Of course. The person who takes the customers money is responsible for delivering the result and any warranty.

> Why are the developers who didn't care enough to double-check their code free of responsibility?

They are not the product owners. They don't decide what is the correct way the product works. Maybe they created the bugs because they implemented the specification exactly as written?

> They don't decide what is the correct way the product works.

Of course we are, we're the ones writing it.

> Maybe they created the bugs because they implemented the specification exactly as written?

If your argument is "maybe they were told to write the bug in", I don't know what to tell you. If I were told to write a life-destroying bug into the software I worked on, I'd quit, because I don't want that on my conscience.

In your hypothetical situation, I would blame the justice and banking system. It should not be so vulnerable or eager to believe an app made by one person, a self made "expert" on something, new theory etc.

Like, you as a single seller are also responsible for making false claims. But, the justice itself should be more robust then that.

Reply on news.ycombinator.com