Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B | Hacker News Reader

Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

itprotoday.com

417 pointsRmDen319 comments

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128 million [2].

[0] https://www.scribd.com/document/100544020/Indiana-IBM-Decisi...

[1] http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf

[2] https://www.indystar.com/story/news/2017/08/07/court-ibm-owe...

Sometimes, you just gotta love the judges that handle these cases. The first 7 words you quoted plus the ensuing judgement are written very well.

Or Oregon v Oracle over their state healthcare exchange.

http://www.oregonlive.com/politics/index.ssf/2016/09/post_18...

Or in Queensland with the $1.5b health payroll failure.

https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-...

Ha, funny story about that.

My brother-in-law's uncle is a doctor (specialist) in Queensland. A couple of years ago he told us a story about this. Because of the massive screwups a bunch of private practitioners like himself weren't getting paid by the state government for work they did for the public health system.

It reached the point where a lot of doctors were simply not going to provide nonessential services if they continued to not get paid (this went on for a year or more). So some workers in the state health department intervened to help resolve the situation. I don't remember the specifics but they essentially approved an advancement that sort of looked like a loan as far as the government system was concerned.

So this guy did get paid. Almost a year later he started getting threatening letters from the government for defaulting on his loan and I think they were threatening court action.

The health department who organized this were sympathetic but said because of the system their hands were tied.

This actually reached the point of the government initiating an investigation and potentially having ramifications for his medical license with the health department seemingly powerless to stop these wheels that were put in motion and looked to be headed to court.

So this guy ended up sending a letter to the health minister threatening to not provide any medical services to the state government at all if this wasn't resolved. Now this would actually have left the state in a pickle as for some things he was the only provider in the state, basically.

Sure enough it was resolved within 2 weeks and the health minister apologized.

Anyway, from this one anecdote it certainly seemed like Queensland health was a disaster.

A similar thing happened in New Zealand when the Ministry of Education rolled out their new Novopay payroll software.

Teachers were getting overpaid, and then collection services would be sicked on them to recover the excess pay, often without contact first to simply ask for the money.

Meanwhile, there was no compensation for teachers who were underpaid.

I didn't know about that issue. Although your story also include:

> Since then, Queensland has initiated - and > comprehensively lost - attempts to recoup damages > from the failed project. A judge has ordered the > government to pay all IBM’s legal costs in the > failed suit.

Ouch. I know nothing about the rights and wrongs of that case, but many of these agencies wounds are self-inflicted.

I heard informally of one case where a federal department demanded a "big bang" change introducing multiple new systems all in one go, even though it was obvious (and IBM advised) that the thing be done incrementally. Of course the budget and timeline ended up blowing out.

I expect that sort of thing is quite common, and companies like IBM will be perfectly happy to take advantage of the blowouts.

Or the Australian Census

http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-i...

I have known two people who worked in the Queensland Health payroll roll-out. Both actually blamed the government for constantly changing specs, and politicians making claims in parliament about deliverables and deadlines that they then had to meet. They also said IBM (or whatever actual contractor they had on the ground) should have managed things better.

I now work with someone who was at Main Roads Queensland who claims the whole of the Queensland government has an implicit ban on IBM for any new work, and are trying to replace all current IBM products and services.

Edit: My previous job used DB2 on an IBM server. During end of financial year time we had to ask them to turn on more server processors on the server. Jeeesus they charge like a wounded bull!

I thought the QLD Gov IBM ban was explicit. It certainly was as of 2016: https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-... (this is the same article linked by GP).

I'll bet there's plenty of IBM salespeople who hate that one guy who made bank on the QHealth deal...

Also US Defence's DEAMS - Accenture and Oracle.

15 years, $2.2billion and not finished.

http://www.dtic.mil/docs/citations/AD1002341

Apropos HN, he then says,

  officials set out to fix Indiana's poorly-performing
  welfare system by inserting an untested, theoretical
  experiment, and substitute personal caseworkers with
  computers and phone calls ("remote eligibility"). This
  is now admitted to be an error, and there is nothing in
  this case, or the Court's power, that can be done to
  correct it, or remedy the lost taxpayer money or
  personal suffering of needy Hoosiers.

ML based welfare eligibility screening seems a few short steps from Kafka. I'm not sure if that's what they were doing, but it seems like something that somebody would think is a great idea.

The ONLY person who would think all that is a good idea are the engineers who were able to update their linkedin with "implemented ML based solution to replace human resources with computer resources to streamline the welfare system for the state of X"

I don't think any engineer would consider that a good idea. But if somebody tells you that's what you're working on for the next few years, make lemonade I guess.

Still sounds like fun to build.

LinkedIn is basically the epitome of HR pointlessness. so that would probably be a bullet point to not include

That's not at all true. The only engineers who would think all that is a good idea are those who could update their LinkedIn. The salespeople and non-technical leadership though? They're going to love the idea of ML (read: magic) based solutions to replace human resources with computer resources to streamline their welfare systems.

The quoted paragraph isn't ML, it's telecommunications.

In France a few years ago I registered for unemployment benefits (that was quite a quest), I received a letter from the electricity utility telling me that I qualified for a reduced price. Later when I found a job, they put back the normal price.

States were successful in moving Medicaid enrollment to outsourced providers. THis was probably seen as building on that theme.

I think IBM bought an Irish company that did similar work in the UK, so it probably looked achievable, and a potential Cash cow for other states. (Accenture was able to sell child welfare solutions at great profit in the 90s) Its a great business as it generates alotmof complimentary sales and services.

Accenture was able to sell child welfare solutions at great profit in the 90s

In the 90s that would have been Andersen Consulting rather than Accenture. Although having worked with them at the time and found them worse in every way than IBM is described in this thread, I wouldn't be surprised to learn they profited at the expense of poor children.

I've worked in the public sector and in support of a large complex social services system at a very large scale.

It's a frustrating place to be put in charge of for a political appointee charged with reform, usually without alot of knowledge, or with a idealogical bent fueled by conservative dogma in this case.

IGS is a shitshow, but outsourcing stuff like this is ridiculously dumb. IBM was in a position where failure was the only option, and anyone attempting to do this will fail. IBM underestimated the difficulty of the task, probably because the customer had zero clue.

Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.

But after selling the deal, their workers have to solve a difficult engineering and organizational management problem simultaneously (the project).

They use contractors (devs) which proves they are not an engineering but sales culture. So, you need to bridge the two worlds. That's the PMs. These are the most incompetent members of the team. The devs very often are very smart.

Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.

Reasons for failure:

1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.

> Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.

An EDS executive's alleged overpromising, in landing a big contract to build a CRM system for British Sky Broadcasting, ended up costing EDS big bucks: USD $460 million, or more than four times the value of the original contract [0].

[0] http://www.oncontracts.com/eds-british-sky-overpromising/ (self-cite)

"After the decision was handed down, Sky announced that it expected the damage award to be at least £200 million. Had it not been for the misrepresentation claims, the pure-contract damages presumably would have been capped at £30 million. The difference works out to about US$270 million."

Seems like a math error.

I agree with some of your points but I don't think that deconstructioning the pieces of the project is a panacea. What frequently occurs when that happens is that you get distributed work along with distributed responsibility. Then whenever anything goes wrong or needs to change, there is no one who can make it happen.

You get team A who needs team B to make a new api, but they won't do it till they get a new device because their kpis don't improve for any work they do for team A, so HR needs to be involved the do hiring but they need to talk to accounting to approve the budget and on and on and on.

If you've ever seen Rick and Morty, the episode where aliens have accidentally pulled another character in and the leader is trying to find out why, when every department blames another and he says "oh, so it's nobodies fault" is a perfect example of most large projects

Not a panacea, but if you imagine - every effort is begun with a built in base probability of failure (battle won or lost before it's fought kind of thing) - that a smaller effort reduces likelihood of failure cause you have less unknown variables.

Breaking things up, not necessarily with the teams, but with time. Milestones, separate the projects over time. Also, the shorter deadlines I've found help keep focused. Longer than a couple months - things can languish.

This cannot be stressed enough, IBM is basically the opening scene of the movie "boiler room"

Early in my career I had a brief stint as an engineer at a consulting company in the ERP industry. Our projects always had at least one functional consultant whose expertise was _using_ the ERP systems and understanding how they integrated with other systems and procedures. This was a different position than PM, who was often an employee of the client company who drew the short straw.

The functional consultants had varying degrees of technical experience (some were highly technical, including having CS degrees), but in general they were people who in a previous life became really good at managing and hacking their company's ERP system, became the goto person to deal with crap, and figured out they their domain knowledge was highly valuable.

The competence of our technical consultants was questionable. On my first project the senior tech consultant told me that I was the first person he worked with (himself included) who structured his code into modules. He was like, "it makes it so easy to use your stuff!" :) But the functional expertise was excellent and the reason the company had an excellent project success rate.

The company basically imploded after some senior technical engineers and managers bought into the Java and XML fad. Their plans failed horribly because the technology was too complex for the technical consultants (not to mention too immature), and it left little room for leveraging the expertise of the functional consultants as pivoting to Java and XML effectively required reinventing everything. Chaos ensued and, after being bought by a major ERP systems vendor (ironically for their strong project success rate), effectively disbanded.

Yeah, our projects were generally

A functional consultant on our team who is great. Connected to their functional consultant who was wildly differing in skill level, from expert to can barely use a computer.

Technical Consultants, which involved 1 Senior who knew everything about one segment of topics. Another Senior who knew everything about a different segment of topics. A junior to learn things. And the PM which was also the sales lead looking for more work.

1 of those Seniors needed to be able to have social skills and diplomacy, the other didn't. You could hide the 2nd through preplanning.

The junior just needed to know to keep his mouth shut.

Then you'd have a variety of other seniors who you could call in on a particular topic, but you'd try not to use, as they are on projects.

And in our case, we had a GM who was more functional than all of us, almost as technical as all of us, and the best in front of customers. He could be brought in to deal with any special scenarios, and to gut check our plans.

So we were really, ridiculously successful with that model.

Basically, really well paid, SMEs, no cruft except a younger guy to learn on the job. Such a good structure.

Amusing to me that people consider a ton of modules to be a sign of high code quality.

What was your role in the project? Did you swoop in and do a copy and paste refactor into a bunch of tiny files and declare your victory?

I’m going out on a limb here but I’m going to say you’re a little bitter at the value of the consultant vs. your own textbook, indignant idea of value.

Not ton. What was stated was the modules, at all.

You obviously never used Allaire ColdFusion, before it got fancy features like functions.

IIRC, a "tag" was the closest equivalent to a function, but it was basically a parameterized textual include. A "module" was a "tag" that didn't need to be installed into a special directory. During development the only easy way to not have the source code for your entire application in a single logical source file (split across unparameterized includes) was to use modules. Using tags was a PITA because of the need to install them in a special location, but for [reasons] people never bothered using modules at the time.

So to better understand the context, s/modules/functions/.

The fact that I uniquely used modules says less about my competency (or the competency of any particular consultant) and more about the general technical competency of the organization. As I said, some of the technical consultants had CS degrees so they fully understood the concept of functions, as well as more difficult concepts. The senior consultant I mentioned had a CS degree. I didn't mean to imply that I thought I was more competent than he was; quite the opposite. I remember what he said to me precisely because he did have a CS degree (which I lacked), was one of the most experienced consultants at the company who had worked with most every other technical consultant, and was someone I generally looked up to.

And you conveniently skipped my larger point which was that the organization was successful without strong technical competency. All things being equal you want strong technical competency, but especially in the world of ERP systems where everything is highly customized and a tremendous amount of code is ad hoc business logic, functional competency is incomparably more important for achieving project success.

> The sales guys are at the top. They are generally competent. Proof = they sold a massive deal.

I do understand what you meant here, and it's mostly right. But there are also obvious exceptions.

You can often make a sale by promising more than your competitors. If your competitors' bids are calibrated by what's actually possible, and yours isn't, then you'll win the bid... and then, years later, be unable to deliver what was specified, and have to renegotiate. But hey, you won the bid!

I guess that, if the market doesn't keep track of these renegotiations and failures-to-deliver, this is the optimum strategy over the short- and even medium-term. But it gets your company known to devs as a company that chews up and spits out talent. Devs that get stuck on projects trying to do the (literally) impossible, slogging forward each day with the knowledge that all of this is going to be ripped out when the deadlines pass and the renegotiation hits, don't tend to recommend to their friends the companies where they had to do that. So, over the long-term, this is a recipe for a talent shortage.

But, like you said, there's always contractors: fresh pools of devs who never signed up to work for something like IBM, headed by either unscrupulous or just plain naive management willing to take on such jobs with literally-impossible specs.

I disagree, IBM is a multifaceted company.

These kinds of projects fall under what used to be called IBM Global Services https://en.wikipedia.org/wiki/IBM_Global_Services. It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell) etc. I've never heard of over-delivery from any of these kinds of outsourcing arrangements, they always seem so obviously destined for boondoggle.

IBM proper, the one that makes mainframes and POWER and DB2 and a ton of operating systems and storage etc is very much a technology company. Some of their best products have the worst sales and marketing efforts. I'm working directly with the senior leadership of the POWER group right now and there are no salesmen in sight.. the technology will either sell itself or not. When we met in person the first time the GM told me "we can build any kind of computer you want" - meaning microarchitecture changes, SERDES configuration, new board layout, sheet metal, OS, application tweaks. Not a lot of companies can do that. There is hubris, less technology, and lack of technical value at FANG or most startups or whatever your benchmark is in comparison.

IBM Research is one of the only remaining great industrial research organizations https://en.wikipedia.org/wiki/IBM_Research. Their results speak for themselves.

I was looking for a proper answer to that parent comment. I could not have come with a better one.

I’m surprised the Stratix FPGA platform isn’t getting more marketing. Half a TiB/s memory bandwidth [0] (white paper) when GPU’s memory transfer is the bulk of the overhead could help Intel make up for lost gains in SIMD application marketshare.

[0]: https://www.altera.com/content/dam/altera-www/global/en_US/p...

Maybe misparented comment?

HP has GenZ, IBM is going to move from DDR or DDR buffers to CAPI attached RAM -- expect to see HBM2 attached to CPUs in 2019. I'd be happy to discuss that kind of thing in email.

Certainly misplaced! It was late, and somehow I scrambled IBM Research and Intel. I was thinking about how Intel's marketing is generally quite bad, whether it's unfair benchmarks or not knowing which products to actually push.

Please disregard.

Although this is misplaced, I'll reply anyway.

There isn't more buzz around the Stratix 10 MX because it's a phantom chip. Intel's December press release clearly states that the chips are available but you can not buy one of these chips today. A blogger did some research and came to the conclusion that the press release claims were simply not true:

https://semiaccurate.com/2017/12/19/intels-claims-fpgas-hbm-...

In contrast, Xilinx released a video of their competing chip working with all channels running at the full 460GB/s on day 1 of silicon bring up:

https://www.xilinx.com/video/fpga/first-virtex-ultrascale-pl...

Yeah I was referring IBM Global Services.

And my intention not to bash IBM or even IGS, but describe WHY some, not just IGS but many others - mega-projects blow up.

My fault for not making that clear. I’d edit the post but I can’t now. Their technology LOBs I’m sure are a different story.

If only people knew how often employees of companies like this scrambled to build something in a mad panic because some exec made a promise to a client about software that wasn’t even designed yet.

Companies as large as IBM don't have a single unified culture. The sales groups are sales oriented, and the engineering groups are engineering/ tech oriented. Most of the time the two aren't in the same building, and often not even the same city.

> Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.

> Reasons for failure:

> 1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.

It seems like you're stretching to blame PMs, and I don't think it's deserved.

The sales guy's job isn't just to sell the biggest deal - it's to sell the biggest deal that the company can actually execute and deliver.

And it's no secret that there at least as many incompetent devs as there are smart, amazing devs. There's a reason "fizzbuzz" is a weed out question.

Likewise, there are good and bad PMs.

Without all of the information it's impossible to place blame on a single group of employees.

Why are the sales guys necessarily competent in seeing that a project adds value? They only need to sell it.

I have been on the bad side of IBM Global Services both times I’ve encountered them and was warned off by someone with existing beef.

They will build the most complicated thing that could possibly solve the problem. And then you can’t get rid of them because holy shit nobody else wants to deal with their code.

If Rube Goldberg and H P Lovecraft had a child it would weep in despair knowing that in all its life it would never create something as sinister and complex as the stuff IGS makes before breakfast.

They were trying to charge $80k a year for a proxy server to handle XML RPC. For a system that was basically ftp with code signing. Our team took care of the code signing, soup to nuts. To this day I don’t know what they were doing with all that money for a tiny part of the system. Except try to take over. They didn’t expect the wall of competence they encountered.

That seems a little over-analyzed. Sales is seen as a profit center, engineering is seen as a cost center.

This is no different than a good salesman selling some shipping contracts and then the company failing to deliver because some penny pinching nit-wit "saved" some money by skipping oil changes and tire maintenance causing the trucks to break down before the job was done.

Its just on a bigger scale.

> The sales guys are at the top. They are generally competent. Proof = they sold a massive deal.

Seems very flawed logic. Anyone can make a sale if you promise everything, charge a low price (that can't sustain your organisation/solution), and have no responsibility for actually being able to deliver within the timeframe they promised.

I fail to see how that shows the sales person is competent.

Pretend you're a customer. Would you buy from someone who promises everything at a price that cannot sustain his organisation and accepts no responsibility?

Have you met your local politician who is trying to make decisions on things they don't understand?

I'm an expat. My politicians aren't local.

The two I've spoken to were also both knowledgeable and intelligent, so your rhetoric is glib but a little lacking in factual accuracy.

Pretend you are a CIO with an average job tenure of 18-24 months.

Do you give the CEO the right, expensive and difficult answer? Or assign accountability to a third party, bug out and leave the mess to the next guy?

In most companies with mediocre the latter is the answer.

Still, I can confirm from personal experience this is how the world works for many people. Been in multiple companies that went bankrupt because the sales people showed this behavior.

The project sizes are fine. The team sizes are too large.

There's too much empire building and career-minded politicking going on in companies of this size, which gets in the way of actually working on the product. Managers increase scope to increase their budget, then do busy make-work to justify the budget so they can get more next round. The engineers need to look busy even though things aren't defined, and optimize to internal-facing metrics as opposed to product-oriented productivity. Anybody who steps back and mentions that progress isn't being made towards actually getting the customer a working product is reprimanded as undermining the unquestionably accepted established processes (which usually aren't working anyway).

The actual deliverable gets lost in the shuffle, as the lumbering hulk of the company and career paths within it overshadow the customer.

This is the correct answer. I saw it first hand how PM and engineers don't care about the product, they just care about their careers and company internal metrics.

A friend of mine has had an illustrious career in I.T. with many successes in the last decade. Right before he struck out on his own to build his brand he was unhappy with his then job and decided to work with a recruiter to find a new role.

The recruiter got him a job with a large bank going through a transition from a major platform from a third party that had been long neglected. The platform was on what we will call version 5 of the software while everyone else was running what we will call version 9 of the software.

My friend was hired as a technical project manager. He worked there for less than a month before he struck out on his own and got to realize his full potential.

In that brief period of time he learned the following:

* The bank had not started to put together the requirements for the new software to be integrated. * The software vendor had no upgrade path from the very old version to the new version. * The bank needed the migration done in under ninety days or they would start getting fined millions of dollars by regulators. * The project had already committed to spending $3M / month on a five year commitment with a hosting partner but they didn't have any developers working with them yet.

What I took away is that when big companies do stupid things they are big stupid things. I'm not surprised that government being as big as it is would also do stupid things on a larger scale.

I wholly disagree with your criticism of the middle. The original article points to the cause of failure which was dismissal of SMEs prior to deployment. Almost all projects of this size are doomed to complexity overload but that is surmountable, but loss of Product Owners and SMEs is not.

It wasn't sales who decided not to have a SME on the project. They already got paid; why would they worry about personnel? That is a classic middle management decision.

And Pennsylvania suing IBM: http://fortune.com/2017/03/10/pennsylvania-ibm-lawsuit-tom-w...

I’ve had nothing but terrible experiences with IBM Global Services. With deals like this, the commission check is cut long before the problems arise.

With Oracle involved, you've also got the strong possibility that the salesperson who got the commission check on the deal is long gone by the time the product is delivered. Turnover is unreal on some of those teams. I have a friend who no longer works there that had 100% turnover on their team in six months.

IBM consulting and many state governments are a mess. It’s no surprise a large project requiring both to function properly failed.

I was the CTO for a large, Fortune 100 company.

When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had top priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the business, and wasn’t allowed to analyze if what was specified made sense, or would work. Luckily I had a CIO that was willing to back me up when I started rejecting these projects. In six months I killed 10 or so very large multi-year software projects, in each case the business was forced to use existing software and change their processes. We saved a ton of money and implemented the solutions in weeks not years. These would have projects that had upwards of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect” process.

I consulted to the DoD and this pattern was repeated, over and over again.

Most assuredly this is what happened with the Canadian Payroll project. Every person had their pet requirement(s) and they made sure they “got them in”, in the end the system that was specified couldn’t be built.

It’s like SAP, if you buy SAP take it out of the box and adapt your processes to SAP, if you do you’ll have great experience. If you try to customize SAP, it going to get very expensive, you’ll have a ton of problems, you won’t be able to take upgrades and hou’ll Need a ton of staff just go try to keep it running day in and day out.

Been there, and truly understand.

Governments everywhere would function a lot better if there were more people like you in them.

I'm probably preaching to the choir, but government managers are deathly afraid to stick their necks out and say "no" to the ridiculous spec-by-committee approach to projects.

It's a lot easier to shift the blame to contractors, particularly when most projects take so long that they're obsolete by the time they go live.

I have had a similar experience. People hate change, and will do their best to not change their processes, ever, unless someone makes them (which rarely happens). They'd rather spends hundreds of millions of dollars inventing some new piece of software just so that don't have to spend a minute pressing a button they don't want to press.

I am convinced 90% of corporate IT software would be irrelevant if the businesses were willing to change 5 or 10% of their process.

I've participated on several >$10M Canadian Gov't IT projects over the years and you nailed it.

Business groups will fight tooth-and-nail to keep their existing processes - partly to maintain status-quo, but also because the folks who represent the business know little about system's analysis/design, so they just simply aren't in a mindset to re-engineer.

Combine this with the vendor's (not so hidden) agenda of milking the gov't and it's a recipe for disaster... every damn time.

A factor here that might be overlooked (and I'm not making excuses for the bad process you're describing) is that changing business processes means re-training all your non-technical staff (and everything that comes with that - training materials, manuals, forms, public-facing processes, etc).

In a unionized environment like the Canadian federal government, that would likely incur not only a fairly high cost (but less than a failed technology project for sure), it's also a massive HR undertaking that business-side managers and leaders want to avoid at all costs.

So they'd rather push all that work onto the technology teams to ensure process remains the same, even if it's the wrong thing to do.

My personal anecdote on this: years ago in a past job, I led a team that worked on a project for Bell Canada to build a retention portal for their call centre agents, and the requirements were full of crazy and sub-optimal flows and features, and every single one we questioned was justified with "any other way would require too much re-training of unionized call centre staff".

In one case, for a call centre in New Brunswick, the union actually had a clause that major process re-training could only be done every 2 years, regardless of cost.

I've consulted to the Canadian Government on both a data science project and on cyber security as well. The technical people inside GC know what they need and it isn't IBM building custom software for something like payroll. They should use off the shelf stuff for the 95% of employees that have run of the mill circumstances and just use humans to handle the super long tail of bespoke needs that no other Canadian employer has to deal with.

The problem is that the political side of the government doesn't trust their own technical staff and they get convinced by these horrible consulting companies like IBM that the only way to write good software is to spend hundreds of millions on it but don't worry it will pay for itself with all the cost savings on staff.

They try to minimize risk through bureaucracy but they end up getting the opposite of what they want. Less useable, less secure software at 50x the price. What they would do if they understood software is communicate to the public about how good software projects need fast iteration and that mistakes might be made sometimes, but that the important thing is that they are easy and fast to fix and that there are risk mitigation strategies to stop mass leaks when penetrators get into networks.

Also, the CSE should just be empowered to take GC servers offline whenever they don't conform to a predefined list of security practices. It's 2018 the fact that all of GC isn't on HTTPS / HSTS is quite frustrating and it requires all departments to get on HSTS preload lists because of the stupid way gc.ca is a subdomain of .ca.

I've seen quite a lot of this too, though from other vantage points.

I don't disagree with anything you say, and would probably go down a similar route. These projects have enormous and unacknowledged risk. One of those risks (the most common pone) is that the project does actually arrive at completion, is good enough to go into use, but it's still pretty crap. Not better than what could have been achieved at a fraction of the cost. You should (as CEO, CIO) take this as a given, and act accordingly. IE, minimize these projects and find alternatives.

So, this isn't a criticism...

Here's the problem as I see it: Custom, "enterprise^" software is important, very. A lot of things could work a lot better if we had better enterprise software, much better. Especially for government, the ability to produce good software that works is tremendous. Transport systems could be better, welfare systems could be better, democratic systems could be much higher information, education.... Google maps really raised the minimum standard for public transport "timetables," especially buses. This changes public transport, enabling multi-stop routes (you would never find them otherwise), casual use, tourist use...

So... How do we do this better? Personally, I think the recurring failure details are probably a red herring. You imply problems like not having a designer, just a dump of requirements from anyone who has the juice to make them. Problems like cargo-cult design around existing processes, rather than adapting to new tools. IMO these are symptoms of bad meta-systems, the "economy" around making decisions, buying services, designing solutions, etc. Also, naturally, the incentives.

Will this be a problem forever?

^I'm at a loss for a good general term. Hopefully everyone understands what I mean.

In this specific case a simple Google search would have sufficed[0].

[0]: https://www.smh.com.au/technology/worst-failure-of-public-ad...

As a Canadian who has worked at IBM, I'm not surprised that this didn't find success. In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale.

With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful financial engineering and sales firm that sells things, then scrambles to hire/acquires to get things done to an ever-evolving, always justifiably rough spec. They quack and then start the hack-show.

What do you expect from the organization that invented "FUD" (https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt) as a way to emotionally manipulate perspective clients into making a purchasing decision?

I know folks who work at IBM as well - don't mean to put you on the spot - but IBM should be far better at leading and guiding customers to success.

Except at $250-350/hour, below average performance is more profitable than successful and a lot of good people suffer on both sides.

Didn't IBM effectively give them that guidance?

It seems kind of simple in a way, if IBM was hired to build X and failed to do so, then there will be a law suit or some form of arbitration and some money will be recovered or something. It's pretty standard breach of contract type stuff. You can audit everything too, maybe IBM was billing for stuff they shouldn't have; I kind of doubt it but maybe there was something really amiss.

On the other hand if IBM was hired to build X, then it changed in to X' and then X'' and 10,000 changes later it turns our, Canada really wanted Y from the start, isn't IBM guiding them along their process to figure that out? It's an insanely expensive way to build and buy stuff.

Is there supposed to be some sort of internal customer's advocate that looks at the project and says "hmm, this is a payroll system, it shouldn't cost a $billion typically, or this was a $100m contract that is already up to $300m, something must be wrong..." Don't get me wrong, in most sales driven organizations, they're high-fiving whoever inked that contract and trying to get invited to the parties he throws on his yacht. The catastrophic failure sucks for everybody, it's hard to see someone else finishing the job more inexpensively that getting IBM to do it though.

Interesting idea.

What if X wasn't defined to begin, besides a high level of saying "one way or another we'll get you a working result".

Any reasonable firm is hired to map out X to Y as best as possible, or at least be sure they did everything to uncover the possibilities, as much as they may, or may not occur. I have a hard time believing IBM has only experienced this once. I'm not saying IBM would do this, but maybe it happened that certain risks or unknowns didn't get the full attention or support from either side.

Contingencies in contracts are often for when things go to hell, taking up any action or arbitration is still not getting the client the result on top of taking resources away.

Saying we tried when we took a lot of your money isn't exactly a satisfying experience and use of taxpayer dollar. If this had been in the private world, would it have happened as easily?

Your point about competency in procuring digital solutions is a neat one - I don't know of many organizations, be it commercial, government, or academic that are good at this.

I guarantee that "X" wasn't defined when the contract was signed. Not defined as something that could be implemented. It was a problem solving mission.

I also want to clarify, I'm not saying IBM plotted X to Y in the most efficient way or that they did it when others could not in my last sentence. I was saying that now that Canada is a $billion deep. They have 3 choices: 1) kill it all and be $billion poorer and have no solution. 2) Somehow beat IBM in to a better deal and get a solution for like $1.2billion, I think news like this is part of the beating. or 3) Probably spend more than like $1.2b to get someone else to get it across the finish line or start over from scratch. Presumably they need something and at this point IBM is probably the closest to the finish line; change horses midstream and you usually get wet. Hey and let's throw out the other condition, IBM might own the product so switching vendors might be a defacto start from scratch.

> IBM isn't a successful technology company with a proven record building good products.

Whoa, what? Sure, some of their achievements are in their past, but that claim really overlooks a ton of history.

In this context, the only history that matters is that which speaks to its current ability to good work. I don't follow IBM closely, but the impression I get is that its current reputation is spotty.

Only huge failures make the news. CBC doesn’t run a feature on a successful software implementation.

You're right. That was too sweeping, in fairness. I've edited in 'good software products'; that record is categorically abysmal. IBM makes amazing mainframes and physical technical systems, and conducts some fascinating cutting-edge research, and even built really effective punch-card machines for use within Nazi concentration camps. (https://en.wikipedia.org/wiki/IBM_and_the_Holocaust)

If only they could run a payroll system as effectively as they can run a genocide.

You're mistakenly conflating IBM's consulting business with the rest of their current technology, both hardware and software.

No, IBM has conflated their consulting service with the rest of their technology by running under the same brand name.

If they want the benefits of sharing that brand between operations then they deserve the negatives that come along with it.

No, IBM has not done that. IBM Global Business Services and IBM Global Technical are their own business units.

Agreed, IBM has a long track record. Their recent history doesn't have a lot of W's though.

That's not true, IBM has way more Ws than Ls over it's history. Now the last few decades, that's a different story I think.

That's exactly what I said.

You must not read much technological news. From the top of my head, I can cite Quantum, Photonics, the Neurosynaptic chip, the 5nm chip fab process, POWER and Z, ...

Depends on the time-scale you're using...

It's only a slight exaggeration to say that they're living on a reputation they built doing everything except what they're trying to sell now.

When their other ships were going down, they shifted to a growing area of demand, not an area of their expertise. A brand shell game.

> In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale.

How in any way is this "In IBMs defense"? We should think it OK that they pursued and committed to a large government contract that in reality they were not qualified to complete? The sentence reads more as a condemnation.

It’s likely more complicated than this and there is probably tons of blame and failure to go around on all sides, particularly on the management and oversight side.

“Our” in this case clearly refers to Canadians, not IBM. Do you know what public sector means?

I read the comment you're replying to more as "IBM shouldn't commit to projects if the client isn't mature enough to make it work."

I can imagine that management meeting. "I don't think we should take the project because the client has more money than sense."

I've worked in consulting on a much smaller scale. Like $1-30MM kinda deals. It was priority 1 to make sure the client knew exactly what we needed from them to get a successful outcome.

Sure, me too. But this doesn't mean the management doesn't change, their requirements don't change, and idiots in consulting will bend over backwards to make them happy.

Maybe it's time to start firing people for choosing IBM.

As an American who has worked at IBM, I'm also not surprised that this didn't find success.

They messed up my own payroll, by sending too little, then too much, then we had a 6 month back and forth trying to resolve it.

> perspective clients

prospective

Not having ever worked on anything like this, the death knell for these projects must be the inability to ever reduce functionality into a reasonable core set. Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely?

You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

Also there's a good chance that the decision to go with IBM as a vendor, and the conduct thereafter, was corrupt from the start. That would go a long way toward explaining why so little due diligence was observed in assessing whether or not to continue the project with them at any of the stages it could've been.

Yes, people do make huge blunders in powering forward with overpowered, unnecessary solutions to problems they largely do not have; but blunders this big usually don't take this long to suss out.

> blunders this big usually don't take this long to suss out

This is actually pretty minor league compared to U.S. government failures.

e.g. https://www.wired.com/story/us-border-patrol-hasnt-validated...

Idunno, that doesn't really seem like such a big deal. The whole idea was rather pointless (in the face of all the bigger fish they have to fry) to begin with. With such headed, small minded planning, it may have been a blessing if they never bothered to implement it.

At the same time, "Nobody got fired for choosing IBM" is a statement for a reason. If you hire someone else, people will say, "Why didn't you hire the best?"

If you hire IBM, you're avoiding that reasoning, even if on any given project IBM or other large enterprise vendors may not be the best.

>You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

this is where crucial distinction and choice comes - enterprise software which supposedly have enough flexibility and configurability to cover all the cases or the software which allows and requires a bunch of custom code. The former is a monster and the latter requires good tech chops. And choosing the software with the right balance between what can be configured and what would have to be developed requires deep tech wisdom and experience on the part of the leadership which is usually a very tall order...

Having worked on similar things on a smaller scale but higher up the chain, I can tell you the list of possible issues is a mile long. Everything from lack of leadership and conflicting goals to ever-changing scopes and insufficient resources is a candidate for the big blame. A lot of people, organizations, and systems have to work well together to succeed.

As techies, we think the technical solution is the most superior solution but that is not always the case. Here is an example of something that just happened to me 2 hours ago today. User called and said osTicket system I manage treats '0' as blank. So when they fill out a form with question 'How many X happened today?' and they enter '0', the system just treats the field as unfilled and does not print '0'. That's a big problem for auditing because they absolutely need to be able to show that it was '0'. I started digging into the source code to see where '0' int was treated as a blank. osTicket being PHP where 0, '', null, and false can all be treated as FALSE, it could've been 8+ hours worth of work digging into the right piece of code to figure out the fix.

So I called the user up and we discussed the best way to handle this. In the end, I added a new question above this one: 'Did one or more X happen today? YES/NO'. If it's 'NO', it doesn't matter the 'How many X happened today?' with '0' doesn't get printed.

To make a major implementation work, this kind of discussion needs to happen 10000x a day among 1000+ people, many of whom may not have the authority to just decide things on the fly.

And, often times that discussion leads to a decision maker saying - that shouldn't be the case, it's your problem, not ours that you can't handle that.

So you end up assigning someone to 8+ hours of work to do it, you also have to end up having 2-4 meetings on the topic, and on and on.

8 hours of work to fix a small issue like this one is just bad design. In this case using php.

You're right, but at the same time, it's a totally unhelpful statement in a project.

Everything is bad design everywhere, at least when you're a consultant. You're generally not brought into projects that have good design, and everything's roses.

I've been on a few projects with amazing design, and it was a very different experience. I didn't even argue for more work for them, because they had it all under control.

They essentially, wanted to be confirmed how great they were. I don't know if the CIO was curious if it was true, or if they all just wanted pats on the back, but we taught them a few really detailed things they didn't know, helped on a few really tricky edge cases, and said you're golden, keep killing it.

I would have gone for the 8 hrs option. Polluting the ui with redundant information and flow on impact to reporting potentially. The impact of saving some hrs might end up amounting to even more. Also, testing and lifecycle of this approach could be worse. And a sign of how we sometimes get carried away by tge tech definition. Why call it 1 or more tickets rather than any tickets?

Well actually...

That's the point I was trying to make. This was something where we could've bikeshedded for weeks with numerous interdepartmental meetings and spent way more hours than necessary to try to come to the optimal solution. Instead since both the user and I had the power to come to the final decision together, we picked this option.

If it works, then we spent less time on the business problem than I've spent writing my two comments here. If it doesn't work, we go back and decide if 8 hours to fix osTicket is worth it or not. Or maybe we find an osTicket consultant to do it for us. Or we find a hosted, mobile-friendly alternative to replace osTicket. Or we integrate this feature into our existing ERP system.

There are a million ways to do anything. Nobody has the time to do any of them in the real-world when we all have hundreds of things to do on a daily basis. The right/optimal/correct solution isn't often the most appropriate solution and vice-versa. People willing to take risks and defend their stances on difficult decisions can get work done. Otherwise you get 'Nobody Ever Got Fired for Buying IBM'.

Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely?

Zeno's Featureset

That's the perfect way to put it! Definitely going to steal this for later use.

> You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

I read something in a trade journal probably probably about 15 years ago, that really stuck with me: When businesses buy big software solutions (I think the article was about ERP systems), they should prepare to adjust and change their processes (within reason, of course) and not just insist that the software be modified endlessly to suit every little ossified process and preference. Not just because it's very expensive, but because the resulting system will be brittle and incompatible and taking advantage of further developments.

Too many buyers of software would benefit immensely from taking a good, hard look at those remaining 20% and making some hard decisions about what is actually important. Sure, for a government payroll system, there is limited room for movement, but I'd suspect that the majority of the long tail are people on weird, custom contracts and the correct solution is actually to hire a room full of clerks to handle these in Excel - and/or applying political pressure on the hiring parties to get their contracts adjusted to fit standard payroll schemes.

> Not just because it's very expensive, but because the resulting system will be brittle and incompatible and taking advantage of further developments.

But that's where the lure of the IBMs of this world comes in: they claim no change will be required, it's all free. Of course, we know that eventually it'll be a big fuckup.

Having worked on ERP software for medium businesses, that sounds spot-on to me; in my experience the biggest challenge was not so much technical as getting clients to change their processes so they can be systematized in a sane way.

There is a lot of resistance from employees not willing to change the way they work. Even seemingly simple things like standardizing language / glossary between departments so we can have a single representation in the system can be an uphill battle.

Same for me, working on MRP-ish software for small businesses.

As an employee of organisations I can tell you one reason for resistance: this sort of systematization ends up being endless meaningless churn.

My favorite quote illustrating this is from former GE executive Craig Kipp, quoted in the book Softwar: An Intimate Portrait of Larry Ellison and Oracle[1]:

“So Oracle has this great product that will do eighty percent of what you want just by throwing the right switches in the software—no special code or anything—and it all falls down because someone asks that innocent little question: ‘What else would you like it to do?’” (p. 231).

https://www.amazon.com/dp/B00AK78QVI/ref=dp-kindle-redirect?...

Except that there are often legal reasons for things to be in a weird way. Or it would require a long negociation with trade unions and 3 strikes. I have some sympathy for the complexity large organisations (public or private) have to deal with (even if “keeping it simple” is still too underrated). I have way less sympathy for the ERP software than can only handle one use case and then it takes a bunch of non technical consultants months to hack a way around the ERP through configuration when really the problem is that the ERP has been designed in a too narrow minded way or that the consultants aren’t the right people for the job.

Incidentally, the error was here:

"The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs."

It's not software problems, or not primarily software problems anyway. They fired EVERYONE who knew anything about payroll in the entire Canadian government, all at once, and hired 500 brand new people in one call center to work with the brand new software as it rolled out. The new software doesn't include code for a lot of "special cases" which together represent a lot of cases. The new people have no idea how to solve most pay problems (people problems OR software problems). They get four days of training.

http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-cen...

"Phoenix was supposed to cut our work in half. It actually doubled our work. So, you can't get rid of 2,000 people ... and double the work and expect the work to get done," one said."

Overall, even if the software worked perfectly the execution would have been a disaster.

Was not surprised to see Oracle and PeopleSoft involved.

I hope they get the crap sued out of them. That stuff is toxic, and I pity any organization that Oracle got its claws into using that junk.

Unless you are one of the people who makes a medium six figure salary as a deck officer on Larry Ellison's yacht, Oracle is utterly toxic.

I doubt they are on medium - probably low. I once met the IT guy onboard his yacht. Told me a funny story about trying to get the superbowl for one of Larry's mates one year.

Captain position ignored for a moment, I have no idea what second or third mate on a giant Russian/saudi oligarch class of yacht might make in yearly gross, but if I were a billionaire I'd be looking for people with many years of offshore experience and a degree from a major maritime University... $175k usd?

Back in 1999, I was in the elevator and I heard the admin. staff say PeopleSoft (the software) doesn't do anything. This was @ Polytechnic Uni. in Brooklyn. This conflicted with the school's public message where (officially) adopting PeopleSoft was a big step in the right direction. I always wondered if PeopleSoft improved since. I guess not?

My employer recently switched from PeopleSoft to Oracle for HR stuff. So now instead of the system itself being slow to respond, it is plenty responsive when I put in for time off. However, if I fill out the form too quickly it rejects the input as invalid. They can't even get form input validation to work correctly.

Stuff like this reminds me of "Star Wars: A New Hope", where people noticed how the storm troopers kept missing Hans Solo despite having so many hi-tech rifles firing at him. "How can they miss him? They are highly trained and so many of them firing at him at once." These IT anecdotes might explain why.

Machine learning aim assist project gone awry.

Oracle bought PeopleSoft. Oracle HR is PeopleSoft.

> That stuff is toxic, and I pity any organization that Oracle got its claws into using that junk.

In my experience, no one is really duped by these big companies. The people writing the checks generally start their search with magic quadrant leaders and then further narrow it down to the biggest players because "we can't trust our business to some no-name." It's really a situation that companies willingly put themselves in.

The only defense I've seen for that is "You want the company to be big enough to sue when they screw it up"

Which is necessary of course because you can be sure they'll screw it up.

And it’s rare for anyone to trial or test this software.

The people who have to use it, who are probably best informed to point out it’s strengths and weaknesses are usually the last to see it.

I've yet to see any way to procure custom "Enterprise" software that isn't awful, and sadly I do believe duping is a huge part of it.

The duping starts long before the Oracles/IBMs/Whomever of the world ever reach a customer too - the "analyst" firm role is a huge problem with companies like Gartner/Forester who dream up the "Magic Quadrant" that people purchasing this stuff end up stuck relying on. I've seen in several roles the enormous gulf between what Gartner/Forester will claim about a leading player's capabilities vs the reality, which of course is a state of affairs large Enterprise software companies are all too keen to encourage.

I've never seen a sales consultant answer "no" to a question about capability in an enterprise software sale scenario, ever, across many RFPs. If your job explicitly targets you to bring in x millions of dollars of business a year, you answer "yes, of course" safe in the knowledge that when it does blow up it's someone else's problem and far away from you by that point.

Duplicity is the name of the game in Enterprisey software. Whether you're a startup struggling to add the features that a client requested, that sales said "Of course we can!" to, and that are utterly foreign to your product ("Why the hell did they ask for voice recognition in this vacation hour tracker?"), or a hugecorp like Oracle with a do-nothing platform that is essentially re-written on site to customer specifications, the time-honored practice of "managing customer expectations" is the name of the game in this market.

Falling behind? Ship half a feature, lie about it or explain how it's going to be working in the next release (and when that doesn't happen, blame the customer's requests for churn, or the height of the tide, it doesn't matter). In danger of actually completing a contract? Trod hard on the bugs, you don't want that money pump to stop. Larry Ellison needs a new dock for his yacht? Time to increment the product's version field and let a thousand consultants bloom. Ka-ching, baby.

My experience with the space, both as a developer of Enterprisey software at several start-ups and as a customer, was that this corner of the industry is ethically sick, and I won't have anything to do with it.

My father was accused of being "anti-Microsoft" sometime in the early 90s, because he kept pushing for cheaper alternatives. He was writing Windows books at the time, so he ended up bringing in a book to show them.

A lot of this purchasing is purely "cover your ass" thinking. Going with IBM, Microsoft or Oracle won't get you fired.

I don't have a lot of love for MS today, but you'd be hard-pressed to compare the MS of today to the MS of the early 90s. Find some developer magazines from that time and check out their ads - they were still scrambling to grab market share in areas other than DOS/Windows for client PCs, and had a lot of competition in the form of Borland, Lotus, Novell, and Oracle, as well as smaller fish like Watcom, Gupta Technologies, Fox Software, etc.

SAP is just as bad - at my last place there was a member of staff who worked 4 days a week - SAP could never get the rounding on their annual leave correct, often leaving them with 0.999 or 0.499 days left that they couldn't book. We had to just record it manually.

New Zealand in the 90's was trying to have an integrated policing system designed by IBM which burnt a lot of money (in the order of 100 million) and was abandoned [0].

I was working with one of the companies that came along to replace one part of the system with a much smaller component that just handled that individual piece and vividly remember the meeting with the IBM team. They had no code to show me, but they handed my a massive folder - probably verging on 10cm thick of use case diagrams. So many little stick figures staring up at me I was awestruck - but not in a positive way!

[0] https://en.wikipedia.org/wiki/INCIS

Yep, INCIS was a cockup of massive proportions.

I remember interviewing a BA (business analyst) who had just come off of the INCIS project. She said their entity model was enormous, with tens of thousands of entities.

I couldn't understand how any project could have so many entities, let alone how anyone would ever hope to code up such a thing.

She said the commandment to the analysts was to model everything so they did, just like they learnt in data modelling 101.

So (INCIS was a project for the NZ Police), e.g.:

- A police station has zero or more cells

- A cell has one or more doors

- A door has one and only one key

- A door has many bars

Another data point in INCIS - my sister at the time worked in the police. Her workplace ran out of pencils (they did a lot of note taking with pencils and paper). INCIS had so thoroughly depleted the police's budget that there was nothing left for...say.. computers with email and the like.

More recently, NZ had the Novopay debacle: https://en.wikipedia.org/wiki/Novopay

The software itself, before fixing it, cost $182mm and covered payroll for 90,000 teachers. That's over $2000 per teacher. It cost another $45mm to fix it after this.

The real achievement here is managing to run up a bill of 1B CAD while still failing to deliver. I mean, I can think of a few projects that probably cost several million/yr but 100x? Astonishing.

There's an anecdote floating around about the consultant who heard of a billion-dollar government IT project who promptly offered to do it for 20 million. When asked how, they replied that they would sit on a beach for 5 years then pick up the phone and say, "We failed". Their contact got back in touch 5 years later and said, "I wish we'd given it to you".

CityTime was budgeted by City of New York for $63M and after spending $700M, a bunch of people ended up in jail.

https://www.wnyc.org/story/143601-citytime-cautionary-tale/

This sounds like a good application of Gall's Law:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

The name is just too perfect: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...

To that point, I’m wondering if all IT initiatives codenamed “Phoenix” are doomed from the start, given that the implication is “let’s burn down everything we have that works, and rise anew from the ashes”...

Hah! I make this joke all the time with friends. I have literally been on three different failed projects named ‘Phoenix’

Elaborate?

https://en.wikipedia.org/wiki/Phoenix_(mythology)

Why did they fail?

Well, one possibility is that I am the common factor in all of my bad relationships. But another possibility is that projects tend to get named 'Phoenix' for a reason, and that just as in real life, it's pretty hard to successfully be reborn from a pile of ashes.

What are some projects that succeeded and/or impressed you?

Not just all IT initiatives - I just heard today that the project that would eventually create the Ford Pinto was also named Phoenix. (from Stuff You Should Know podcast)

The Pinto was a good car, though, even with the one design flaw that was actually the result of both best practices for the time and a lack of regulation. Except for that, the Pinto was a landmark subcompact car.

> To that point, I’m wondering if all IT initiatives codenamed “Phoenix” are doomed from the start

Firefox seems to be doing pretty well :-).

There is also the Phoenix web framework for Elixir. However, it's supposed to help people recover and deal with failure :) It's an Erlang thing.

Yeah, but only after two name changes.

I can literally do that with my DigitalOcean droplet. I can destroy the instance and run a bash script to provision a new, identical machine in around 3 minutes.

It’s a lot of hard work to get every little detail working automatically. Probably 10x as much as just manually setting things up from scratch. But in the long-term it can pay off.

It wasn't a bad book, but if you've read anything by Eli Goldratt on the theory of constraints (The Goal, for example) then Phoenix Project just looks like a watered down, IT-focused, and nearly plagiarized variation.

So, what makes a payroll system at this scale so difficult? On the face of it, there are salary employees and there are hourly employees. How many taxing bodies does Canada have? Federal, provincial, and municipal? Is that a few dozen or a few hundred? Is Canadian tax code crazy complicated that doesn't allow for a rules based system?

I'm honestly curious as to how a "simple" payroll system can go so far off the rails.

FTA: ".. had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."

I remember thinking “Why does this simple state Jury Duty site need to be down from 9PM to 9AM”

“I could write a site that did this job with 99.9% reliability in a weekend”

Turns out disability laws require a real person be available to assist with the site. Imagine bidding on a website project and accidentally promising 24/7 live support.

That’s government tech in a nutshell.

There are multiple rules based on collective bargaining (union).

"The software platform itself, a heavily modified version of Oracle’s PeopleSoft program, had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."

http://wiki.c2.com/?WhyIsPayrollHard

That's.....amazing

It almost gives the impression that an air traffic control system is actually easier (although also failed.)

Both are incredibly complex (I am a developer on an air traffic control system.) There are thousands of requirements for voice communications alone.

Interesting

In the end, it seems a bit like "why is $social_network hard" I mean, why is it hard to show my my social graph?

And the answer is: it isn't. The issue is building a system that deals with everybody's social graph. Because you have in your social feed one person that has few friends (easy), that other person that is followed by several people and they leave lots of comments (and you are loading their information when the post is shown - and of course there's no way of sharding that so your info can be read from one shard exclusively) and everybody has their list of blocked persons, other permissions than on top of that you have things like "algorithmic reordering", prioritization, paid promotions, etc

Similarly,

Gay marriage: the database engineering perspective

https://qntm.org/gay

That article doesn't seem to exist at the moment. For the sake of discussion, let's imagine there was some DB schema somewhere that kept two separate WOMEN and MEN tables. Can we not agree that was always wrong? Gay marriage cannot possibly be the first situation in which that sort of decision caused extra work or inconvenience.

> So, what makes a payroll system at this scale so difficult?

Complex pre-existing business rules of a large, sui generis employer (governments, especially sovereign ones rather than localities, often are because the laws that apply to other employees don't apply to them, and the rules that do apply often aren't coherent across-the-board policies determined by a central org focussed on HR, but often are set in special case laws applied to individual subunits of the government by a legislature then operationalized by the HR staff in that subunit) trying to be put in a COTS system make it hard, but probably what makes it most out of control is that the contractor is optimized in maximizing value extracted not efficient delivery and that governments generally suck at overseeing IT contracts (sometimes this is corruption, but it's often just lacking the knowledge and skills to do it effectively at the direct contract management staff level, often complicated by mandated IT contracting frameworks that are based on the underlying assumption that software engineering is basically structurally the same as civil engineering.)

It seems like lots of the payroll was being carried out by humans in local units. Humans are very good at holding lots of exceptions in our heads and, in the absence of clear rules, doing reasonable things. Computers are shit at both of the above.

So things like Bob negotiated with his boss to leave early on Thursday to cover childcare and, in exchange, is coming in early on Tuesdays... but got asked to come in Saturday and now has overtime and what does he get paid based on province or local rules? Does OT end at midnight or not? Do employees get paid non-required OT as a courtesy? And frankly, lots of payroll probably sometimes violates the rules in minor ways, but as long as the employee is happy and the org is happy no-one is upset. Computers are not good at letting small things like that go. And that's a pretty trivial case.

What about eg janitorial that is paid by multiple government orgs on some cost sharing arrangement because the orgs share office space, but is covering an event for a 3rd org in the 2nd org's space? And that janitor is on a particular contract, one of perhaps 5 or 6? If that has to get encoded into a central system, have fun with that. God only knows how many exceptions there are across a quarter of a million employees.

> Humans are very good at holding lots of exceptions in our heads and, in the absence of clear rules, doing reasonable things. Computers are shit at both of the above.

Which is a remarkably concise summary of my thoughts regarding self driving cars.

When did you form those thoughts? The last 10 years has had pretty monotonic progress in the opposite direction. It seems unlikely for the technology to suddenly stop and certainly not go backwards.

It's that 80/20 rule... I'm sure Canada saw monotonic progress in their payroll system over the last 10 years, too. But with cars, the 20 can kill, not just screw up paychecks.

Thanks for this, this is an excellent set of examples.

The fundamental problem is when people want software to prevent people from ever messing up instead of simply enabling people to do their jobs better. I heard someone say the other day that they wouldn’t use a simple workflow tool becuase it didn’t provide read only access for other company employees. Look, other people could walk over to your desk and throw all of your papers in the trash too. You still use paper. Why must software prevent malicious actions perfectly?

> does he get paid based on province or local rules?

I believe the answer is "neither". Federal employees are exempted from both. That case would be decided based on federal law and the union contract. It's made more complex by the fact that the union contract is often applied retroactively to events several years in the past due to how long negotiations take.

Corruption is often brought up, and I'm not saying it doesn't happen, but any time I've been consulting, it's always general issues, personal issues, technical issues, refusing to change, all sorts of standard stuff. Not corruption.

It's hard to implement even a minor thing in a large company.

Probably because you don't notice the corruption.

That vendor that has lots of technical issues and the manager that is refusing to change. Well, they know each other and the vendor is giving kickbacks for dragging it out as long as possible.

Most of the corruption in government IT contracting is more subtle; its things like government managers developing relationships with contractors and favoring them subtly without kickbacks, simply because they are comfortable and feel better working with them. The people doing it probably mostly feel like they are actually doing a public good and working around red tape, because they genuinely believe that intangibles that wouldn't be weighed properly in the formal process do weigh in favor of the contractor being a better value.

OTOH, yes, the kind with actual kickbacks absolutely exists, too.

I don't even know if I call that corruption. Humans are the key part of business, and relationships with contractors are just a reflection of that. That absolutely happens, near constantly. I don't think of that as corruption, as much as in a world of very difficult success, when someone proves they can be successful, and make you feel confident, it makes logical sense to stick with them.

Decades of different payment rules are anything but simple.

Imagine people hired over the last 50 years. Each at a different rate, pay scale, union, non-union, some people in a position have certain perks, some don't, and it all varies from region to region, province to province, department to department of the government.

I recently oversaw a payroll transformation project in Canada for an energy co at a smaller, but most complex scale according to the payroll vendor.

A few key problem makers add to the payroll mess;

- Accountants and their spreadsheets who think adding one little tweak here or there isn't a big deal. They create so much technical debt. It's not realized, until 10 or 20 simple tweaks on top of each other, and then have to interpret them relative to labor or union rules of each province. Accountants are not a vision of innovation. You will hear many just talk about having to do "manipulations" of data to get it how it should be and quickly losing vision that computers are good calculators and followers of instruction.

- HR / Payroll practitioners do not have enough digital skills on average. Either they can't do the excel work and feel threatened, or do have the excel skills and feel threatened. Either way, change management is a constant threat.

- No one knows how to successfully procure, implement and roll out software as a non-technical manager. Whether it's accounting, hr, payroll, sales, every position is becoming digital first, and the powers that be are not. It's a good case as to why CTO's are needed reporting to CEO's who get the big picture.

Payroll othherwise, is a neat problem on it's own. In my case, the project ended up implementing a larger vendor of payroll that did a decent job that was saved by preparations we made at the CTO level.

Before selecting a vendor, a payroll script capturing the rules for all employees in all regions to calculate everything was developed in-house. It took months of hardening at a few payrolls per month. Using this in-house payroll tool, and not wishing to update it, it was used to hold the screws to the payroll vendor if their output didn't match. Hilarity inevitably ensued.

If someone built the stripe of payroll, it would be something.

Let alone all the country specific rules. Just that is impossible to deal with properly.

Absolutely. You reminded me of the pain I blocked out from the project - at the end client decided to go international and now needed to pay people everywhere through separate entities after swearing they never would at the start.

Sadly the "global" payroll vendor did not have a global, multi-entity solution, and required individual accounts to be setup in each country and reconciled.

Creating a new multi-tenant, multi-entity, multi-country compatible mainframe is maybe something that needs further exploration.

The unions have crazy complex pay rules.

For example, in the private sector, if my boss goes away on a 2 week vacation, usually the boss might delegate someone to answer questions and possibly run meetings... but that delegated person doesn't get paid more.

However, the public sector unions have negotiated that the delegated person get a pay bump for those 2 weeks only while the boss is away.

> How many taxing bodies does Canada have? Federal, provincial, and municipal?

For payroll tax, only federal and provincial. Municipalities can only collect property tax, not sales or payroll tax. So, all in all, 14 tax jurisdictions (1 federal, 10 provincial, 3 territorial).

If they were also handling payroll for diplomats and other Canadian employees in other countries, this number could be significantly higher.

My understanding is its not taxing, its shift work and the thousands of edge cases across the various gov departments that have evolved their pay systems, contracts/union agreements separately

The headline is misleading. According to the content of the article, the project has been going for 10 years and has spend $460 million. The government has pledged another $431 million in the future to deal with problems going forward.

$460 million over 10 years is $46 million a year, which is still a lot of money, but if you break it down it's not nearly so mystifying. If all the money is being spent by contractors, we could assume that the loaded labour rate of the contractors is about $200K per year. It's been more than a decade since I was in Canada, but I'm guessing this is not an outrageous rate. That breaks down to 230 people working on the project. In truth, the payroll system seems to have been operational for the last 2 years and the system employs 500 employees (it was supposed to replace 1200 employees, thus saving money, but there is no indication if those 1200 were, indeed, replaced). I'm guessing that the $460 million includes those 500 employees for at least 2 years and may also include capital costs to house them (the reporter may not be interested in depreciating capital assets in their report).

So it's hard to say exactly how many contractors worked on this, but it's probably safe to say that there were more than 100. A government project with more than 100 contractors == not possible to succeed in my experience. Just the communication issues alone would be insurmountable. As time wore on and pressure to put something into use grew, the mistakes would have piled up causing absolute mayhem.

Unfortunately this is an all too common occurrence. If you go to any meetup in Ottawa and talk to government contractors I'm sure there will be a couple wiping their brow and saying, "This is a stroke of luck because it will take some of the heat off our our project that is burning even more money".

For historical perspective on another (in)famous payroll system project I encourage developers to read about the Chrysler C3 project. It ultimately failed, and yet provided the genesis of some agile development best practices that most of us use today.

http://wiki.c2.com/?WasChryslerComprehensiveCompensationSucc...

https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...

These things are hard to get right in any complex enterprise.

The software platform itself, a heavily modified version of Oracle’s PeopleSoft program ...

What implementation of Peoplesoft isn't heavily modified?

Adapting business processes to COTS software packages is extremely challenging. Anyone remember the absolute bank that Peoplesoft consultants made back in the day when ERP and Business Process Re-engineering was all the rage?

When I last "dealt with" PeopleSoft, the standard operating procedure was:

1) Pay PeopleSoft/Oracle a metric f--kton of money.

2) A CD arrives in the mail. (This little envelope cost $800,000?)

3) Your PeopleSoft consultants show up, laugh at that CD (tossing it into the microwave) and pull out their own distros with a zillion customizations that they proceed to spray over your servers. You have no idea what they are actually installing.

4) You get out a firehose, fill it with money, and keep showering the consultants with cash until you are sure they won't succeed in delivering anything worthwhile, and you fire them.

I was only installing a local test instance of PeopleSoft (to do some integration work with another product). Somehow the PS consultants in the area got wind that someone in my company was doing an install, and I started getting all kinds of cold calls offering customization and optimization, with some of the most condescending and customer-hostile attitudes I've encountered. PS had pretty a pretty shitty ecosystem 15+ years ago, and it doesn't sound improved. I'd hate for anyone nontechnical to be at their mercy.

I was privvy to discussions surrounding a search for a university president. One of the candidates dropped out when he heard that the university was replacing a perfectly functional home brewed accounting system with PS - it had brought his previous university to its knees and he didn't want to ever deal with it again.

That candidate showed remarkable common sense.

That's so depressing. What really gets me is just how bad the actual Peoplesoft product is: in my experience it's unreliable, poorly designed and unpleasant to interact with.

My company recently switched away from Peoplesoft fortunately.

Cross-posting a comment I made from 7 years ago when NYC's payroll project blew up: https://news.ycombinator.com/item?id=2716319 which I last cross-posted to the thread about Phoenix's payroll project blowing up: https://news.ycombinator.com/item?id=15303555

Could this be the first time someone gets fired for hiring IBM? (Sorry could not resist after having heard of that quip so many times).

Nope noone gets fired for choosing IBM

Probably not. IBM will return a token amount of money so politicians save face. Someone might take the revolving door either direction.

Nobody gets fired, but nobody gets promoted either.

and that's why governments should hire software engineers that can be in the loop and understand not only the technical but the policy side. Hiring an army of contractors earning 300k/year to deliver "something" in the waterfall model is a recipe for disaster.

Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic.

Software engineers who are not just technical but SMEs as well as having excellent skills in stakeholder engagement are incredibly rare. And most of them know how good they are and are contracting at, you guessed it, around 300k/year. Talented engineers are usually quite savvy when it comes to money.

Also for most projects like this it is Waterfall for the overall project (Requirements, Design, Development, UAT, Production) and Agile for the Design/Development parts. It's really the only way since the client does Requirements/UAT/Production and the vendor does Design/Development.

> Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic.

I've been in places where it's done. In fact, I've been involved for a long time in a public sector environment with three key systems, one of which is a central claims adjudication system, and two of which are accounting and tracking systems that interface with it, performing similar functions to each other but for two separate programs that share the adjudication system. Those three systems, for historical reasons, have three separate models:

The adjudication is state-owned (that is, the code belongs to the state) and operated, developed almost entirely by a team of contractors with some involvement by state programming staff and a state development lead and also tight integration state-staff IT team that does requirements analysis and acceptance testing and manages most of the interface with state business users.

One of the accounting/tracking systems is, and has been from the beginning, developed and maintained almost entirely in-house by state staff (and almost solely with programming staff.) Because of subsequent organizational consolidation, the state team working on this system overlaps the one working on the prior system.

The other is a vendor-maintained MOTS system, with state IT oversight.

Each has it's strengths and weaknesses, but the MOTS system is consistently the one that is the roadblock to adapting to changing business requirements, and both before and after consolidation the all-internal one was the one with the quickest best-case requirementd to delivery speed, and by far the least expensive for the value delivered.

Was this at the city, state, or federal level? Or in another country?

Large state.

You're absolutely correct, it's nigh impossible to find SMEs you can even put in front of a stakeholder, and not totally jeopardize something.

I've been on the end of jeopardizing, and on the end of bringing in SMEs... It's a nightmare.

This is what the US Digital Service is doing. Recruiting these folks /is/ damn hard, but one thing that helps is being able to offer the opportunity to make truly meaningful large-scale impacts. It's a pretty unique environment with some interesting challenges...

Waterfall (or waterfall with agile loops) is not bad for known solutions and scopes.

Agile is much better when no one has figured out a way to pay people money

Unless you are the contractor making $300k

If you're not part of the solution, there's good money in prolonging the problem.

"Understand" != "Empowered"

One of the things people grossly underestimate when dealing with government is that nobody ever really has enough power to make unilateral decisions.

I'm not even putting the bar at a correct decision, simply A decision.

Just try terminating a person or a contract with the government. Good luck getting the 12 signatures you need.

Even IF IBM screwed up this badly, who with sufficient power in the government is going to sign off on the order to fire IBM? It's a small world, and if you piss off somebody more powerful than you, your position is toast.

The cardinal rule of bureaucracy is to never sign off on anything unless you absolutely have to.

So it doesn't have to be this way.

I work for Alberta Health Services, we have 120,000 employees across the province. There are likely 5 separate unions plus out of scope staff (management). This organization was created after the merger of 5 separate health care organizations all with separate systems etc. They over time rolled everyone into the same pay system and I didn't hear about a single person missing their pay. The federal government transition has resulted in tons of people being not paid at all, being paid too little or being paid too much.

"there is reputational risk for IBM in not helping us fix this,”" - not true. IBM has a long history of exactly this kind of problem. Canada didn't care about their existing poor reputation and the next government to hire them won't either.

The Canadian government attempted to build a gun registry, originally predicting it to cost $12M, then $85M, then $1000M, then $2000M, and then it was scrapped. While it's easy to blame vendors like IBM, I have to imagine feature-creep and "what about"-isms lead to such monstrosities.

quick correction, Canada still has a gun registry for handguns. In one of the several elections that they won, the Harper Conservatives campaigned on removing the long gun registry (rifles and shotguns). Which they successfully did, putting hundreds of federal employees out of work in a small town part of New Brunswick. The political outcry from these unemployed people and the economic impact sent them on a new trip back to the pork barrel, centralizing all of the Phoenix payroll system clerical operations in... you guessed it... the same small town part of New Brunswick.

How many long guns are there in Canada anyway. Could you not use sheets.google.com? I'm serious.

There are quite a lot, actually, though canada has a greater proportion of traditional type long guns like bird hunting shotguns, bolt action rifles, etc. Less people who want to buy a consumer-legal version of a HK416.

You jest but really it shouldn't cost 2B for a database.

2B$ for a gun registry? Seriously???

This is sad. IBM used to be the watch word for "It just works!" I started at IBM many years ago. When I went to my 1st ISP (back in the early 90s) and things broke I was like "what do you mean it broke? How is this okay?" It was quite a shock to move from a company where we had systems with 20+ years of uptime to having reboot the USENET server every 5 days.

The comments made about the government about excluding IBM from making future systems for the government stood out.

The reality is payroll should be executed by payroll specialists. I recently had to research payroll providers and there are players who service 20-30% of the Canadian population already.

This is another sign of folks not knowing how to source, design, oversee or implement software implementations.

This is also why Ceridian is still around in spite of all of their faults (or so I hear from any small/mid-size business practitioners I know).

I take it that's who you're referring to?

ADP too.

There's actually a few at that size, Ceridian is one, ADP is another...

Most have their pros and cons but ultimately can handle complex union type payrolls.

Payroll, also is ripe for disruption.

>Payroll, also is ripe for disruption.

From everything I hear about working with those companies—new and suitable replacements would be welcomed wholeheartedly.

That is, if they can maintain the same level of professionalism and ability to handle situations like you mentioned. Reliability is of immense importance, so it's not as easy to just hack away at it and sell people an MVP and iterate.

Ripe for disruption, except that when you look at the laundry list of requirements, the thing that needs disrupting is the need for a billion different custom requirements, not a different payroll system.

I thought that too, until I learnt how all the payroll rules are fed into a mainframe for companies like ADP and Ceridian.

A mainframe stood between a client and their need.

Maybe we're after a better mainframe that doesn't need so much workarounds to do what has become payroll today.

I'm choosing to be optimistic that if a Stripe can navigate the payments industry, a payments startup could allow a better mainframe to be built through a rules engine that allows the degree of flexibility and data interoperability required in today's world.

Your comment made me think of two things: Accurate payroll, and auditing of data.

ACCURACY: One of the biggest things I learned is that a correct payroll isn't a correctly executed payroll, but one that the employee is paid correctly. What's the difference? A lot of payrolls have a ton of manual adjustments to fill the gaps, which never easily get codified. If the barrier to codifying rules could be improved, it would pretty easily start to push around mainframes.

AUDITING: Existing platforms like Ceridian and ADP appear to do an okay job with their auditing - any calculation set to be done by a computer, rule, or human is forever recorded on the same leger. For what I saw, it was a healthy paranoia of tracking how everything is calculated for liability's sake. Faving this available, for repairing data and fixing payrolls is more important than reliably executing incorrect rules, or data. It creates a great audit trail for weeding out behavior.

I'm in no way endorsing one product, but I was reasonably surprised by ADP's newish workforce platform. Of course, once we got into the weeds, things broke and and processes needed to be hacked or split up, but it was only because the ancient mainframe underpinning could not be touched. Which sucked.

If Teyve said in Fiddler on the Roof, all traditions were new once. Maybe all new mainframes were new once.

It's been a long time since I've heard about any successful projects coming out of IBM's consulting and services businesses. Mostly it's one headline after another like this- "Billion dollar project scrapped". How much longer can they keep coasting on the reputation of what was, if we're being honest, a completely different business that just happened to have the same name?

It's incredible, isn't it? I can't believe any govt still hires them. They were behind the Australian census debacle too.

You don't hear about successful projects. They don't make news stories.

I've done a ton of projects in consulting, not IBM, but you've never heard of any of them, because they worked, and the customer was happy (although maybe some delays, or other randomness.)

It's self selecting that news is about massive failures.

Canadian here. This has become a huge political football. People are blaming the current Liberal government, when the project was started by the Harper Conservatives. The full extent of just how fucked up it is has just now become public knowledge.

It is also a political pork barrel project.

> The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs.

What's not written there is that the decommissioned long gun registry was located in a rural part of New Brunswick. When it was killed by the conservatives, they had a big political issue on their hands because suddenly hundreds of federal government clerical workers were out of a job in this area. So what was the solution? Re-hire the same people and put them on the Phoenix payroll project. Never mind that not one of them had any experience with IBM, Oracle, PeopleSoft, payroll, finances, or anything else related. They wanted butts in chairs in front of keyboards and to keep those people on the federal workforce payroll.

https://www.google.com/search?num=100&client=ubuntu&hs=ssW&c...

quick edit, history lesson:

early 1990s, Chretien Liberals: we need a gun registry

liberal party: okay we passed a gun registry law

liberal party: we need a pork barrel project to make these people in small town new brunswick happy, let's hire 500 people in miramichi

liberal party: this gun registry has now cost 400% of what was originally budgeted

liberal party, paul martin government loses an election to stephen harper and the conservative party

harper conservatives: we promise to scrap the long gun registry

harper conservatives: we've removed the long gun registry but now all these people are out of work and pissed off

harper conservatives: we need to overhaul our old payroll system that runs on a bunch of old minicomputers, let's hire ibm

harper conservatives: let's re-hire those people we fired and put the payroll center in small town new brunswick!

harper conservatives lose the 2015 elections

trudeau liberals: wtf is going on with this contract the previous government signed. wtf is going on with this clusterfuck of oracle, ibm, peoplesoft and porkbarrel politics of employing people who are not payroll specialists.

trudeau liberals, later on: seriously this is broken beyond repair

It has been fascinating reading as an example of when Bureaucrats strike to cover their own tails and deadlines.

The way this article was written was really weasely.

> The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors

This is assigning blame to Trudeau in the weirdest way.

It sounds like it's blaming his predecessors?

Still mentions his full name and title front and center. I think that is what gp was referencing.

Oddly enough I read it the same way. It's blaming the conservative predecessors, but on first pass I got the impression they were blaming Trudeau.

You're missing at least two pieces of incompetence:

1: the long gun registry was also a huge failed massively over budget project, so using the same people for Phoenix...

2: the long gun registry project was relocated from Ottawa to rural New Brunswick. Any competent civil servants switched to other departments to avoid the move. Only the civil servants without other options made the move...

yes, it is sort of like a russian nesting doll of incompetence. the fuckups are nested hierarchically inside the earlier fuckups.

This isn't really a political issue. Implementation was done by the civil service and was and is managed by non-political deputy ministers. There was a similar disaster with the centralization of web servers in the federal civil service a couple of years ago.

It's possible to infer from this that CA ordered an output from IBM, rather than an outcome.

IBM can claim that they have indeed delivered the thing that was ordered, while CA remain disappointed that the effect they had in mind is missing.

In Canada's defense though, it's sometimes hard to find someone who will even take on contracts of this magnitude. Getting good terms, even more so.

These large government software projects end up costing way more than they should pretty much every time. The process is just broken. There's way too much design by committee and tying the hands of the people who are doing the actual work, preventing them from implementing the correct solution and instead making them implement the one the disconnected-from-reality specification writers decided on.

Then they have to go back an iterate the correct solution, which involves more committee meetings at huge expense and rewriting documents and disseminating them to everybody...

This is completely wrong.

The CA government engaged the vendor, IBM, after they had decided on a list of requirements from what they wanted from a solution. In your argument you say that the hands of the people doing the actual work were tied. Well that's the vendor, and so no, their hands weren't tied.

And not sure if you've ever worked at a project like this but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.

Committee meeting take the time of multiple people. They are absolutely not free. People's time is by far the biggest expense in a development project like this.

In the long run they can save money (getting everybody on the same page to avoid conflicts in the future), but you're paying up front for that. In practice the value of the meetings decreases with frequency (in the reducto ad absurdum case 100% of your time is spent in committee meetings and nothing gets done).

Only engaging with the vendor after the requirements were drawn up is what I'm talking about. They contract with IBM and hand over a 2,000 page document describing their requirements, then discover that it's impossible to implement and end up spending many years and millions of dollars on trying to fix it.

> but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.

https://en.wiktionary.org/wiki/bikeshedding

> The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed, while neglecting the design of the power plant itself, which is far more important but also far more difficult to criticize constructively. It was popularized in the Berkeley Software Distribution community by Poul-Henning Kamp[1] and has spread from there to the software industry at large.

http://bikeshed.com/

Often multiple smaller/medium-sized systems are better for the particular task than one large. Even more so for taxation and payroll systems where taxation/regulation varies greatly.

Hard to have an opinion without knowing the details but from my little experience I feel like the reason why these big project fail is not so much because they are too complex, I am sure a good (not excellent) developper could come up with an algorithm and data structure generic enough to accomodate all corner cases. But because these big projects start from a preexisting solution (PeopleSoft in this case), that wasn’t designed for this particular complexity, so it requires orders more energy to go around its limitations, and the people staffed by the likes of IBM on these projects are usually project managers with little to no programming skills and little domain knowledge. So you end up with a disaster for things that aren’t that complicated (custom shifts for coast guards and prison guards doesn’t look like a hard computer science problem).

> The government will spend C$431 million to keep the program running in the near term, on top of C$460 million already spent to put Phoenix in place and fix the problems it generated.

So for each of the individual salaries Phoenix was supposed to eliminate, they spent C$600k ($500k USD) in up-front costs, and it will be another C$600k to clean up the mess. Turns out, software takes a lot of time and money to build. Who knew? I guess IBM didn't. Or maybe they didn't care?

Maybe there should be some kind of advisory board system for large software projects, made up of disinterested-yet-well-informed techies, similar to what exists for corporations.

> Maybe there should be some kind of advisory board system

Committees are a bad way to solve a problem of mis-aligned interests.

Perhaps we (developers) could improve the industry as a whole if we can provide a solution to this misalignment of interests.

> I guess IBM didn't. Or maybe they didn't care?

I think from the perspective of IBM, this contract has been good. They made way more from the Canadian government than they initially estimated the contract to be.

Turns out, software takes a lot of time and money to build. Who knew? I guess IBM didn't. Or maybe they didn't care?

Oh they knew, and they cared... about the money

There is some missing math. They likely have 30 or 40 payroll systems all up for renewal/replacement. The old systems are probably loaded with features and forgotten requirements that would have been discovered during deployment.

Govt.office "We need to open these X year old databases/files for tax purposes once a year"

Programmer.ibm "What the hell format is this?!"

"Turns out, software takes a lot of time and money to build. Who knew? I guess IBM didn't. Or maybe they didn't care?"

Well they do make a lot of money while building said software, large expensive projects are the kinds IBM gravitates towards. Most likely because it's easier to extract large amounts of money from them and it's hard to rein in the costs with oversight.

The whole big 4 IT consulting industry needs to die a swift death. It's basically institutionalized corruption. Overbill the client, under deliver with shoddy products made overseas, get that vendor lock in and watch the money flow in. The cost savings are largely an illusion sold by false promises of a better world.

You are mistaken, I work for one of these. I have 12 years experience and I have only one failure to deliver and it was stopped in the design phase because it was too complex and the client wasn't ready.

I worked for one too and I can't say the same thing. Entire projects were poorly thought out, designed and improperly executed. The people working for the company were incompetent and their "hiring bar" was if you had a pulse. And this was a big well known firm.

You're both right, I've seen both sides.

The reality is, in all of them there are teams of amazing, insanely talented people, and teams of complete train wrecks.

It's just life. In any large company, there's a massive variance in ability.

Same experience with me. I have mostly only consulted with large companies. There are some companies that insist on solid code and design. There are others that just live from quarter to quarter and the whims of the CEO and just throw out whatever is done at the end of the quarter. Having a great lawyer team chomping at the bit for litigation helps their confidence in living from quarter to quarter.

Do some research into government contracting and you will realize that companies like Oracle are tame compared to many of the straight up scams being pulled.

You will see countless examples of companies with no assets, no employees, virtual offices, etc, pulling down multi-million contracts.

I agree. I worked for the Fed for 6 years. I used software that often didn't work. I'd look in the HTML comments, and oftentimes they were not in English.

These big IT firms sell a premium on top of what the very best US-based development shops would charge, and then outsource it to the cheapest bidder overseas. It's a scam. I don't blame the overseas teams. They are getting paid a fraction of what they should be. So there's probably a lot of junior developers working on this stuff.

Sometimes I wonder how these projects are started; is it really that they go straight to “here’s a binding contract for millions”, without having them do any smaller projects in between?

If it were me, at the very least I would start with something small and say: “deliver this, let me see how it works, give me every last line of code for further inspection, then we’ll talk more”. I mean, have them build a mini-site or something and make sure that they’re competent before unleashing them on the entire payroll system, right? And you don’t need some grand familiarity with the software industry to plan that way, it just makes sense to start smaller and make sure you know who you’re really dealing with (and also spending less money if it fails).

I hope to someday read a comprehensive post-mortem but I doubt there's anyone involved who can/would write one without a lot of bias against the other parties involved. This will go down in an annals of massive software project failures.

Actually, Australia went trough the same payroll procurement fiasco with IBM. You can read the report http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...

I poked around to read more about this and the gov't did commission a report to look at what went wrong:

http://www.cbc.ca/news/canada/ottawa/phoenix-federal-governm...

IBM?! Oracle?! And you got away with only losing $780 million! Consider yourself lucky.

We really need to rethink the metasystems around these kinds of projects. A $1bn government payroll project is like the canonical "project that is guaranteed to suck" but realistically this extends pretty widely.

These custom software builds, built around specific corporations, with all the sales and account management and expectation planning and making sure that *when" this fails, it is the other party's fault....

We need to find better ways of doing this stuff.

To add to the list of failed (at least initially) payroll systems, we have New Zealand's Ministry of Education and their Novopay system: https://en.wikipedia.org/wiki/Novopay

It cost $180mm and was borderline non-functional, and then cost another $40mm to fix to its now functional state.

The same company that did Novopay also did the payroll for NZ Post and had similar issues.

I have payroll issues with IBM still. They overpaid me when I left, then they got my end date incorrect and wanted more back than they deserve. This has been ongoing for 7 months, they can't seem to figure it out... just check the email logs, if they had any... They sent it to collections some time ago, now trying to get them to negotiate a payoff rather than dealing with IBM Employee "services"

Whoa, doesn’t your state have an agency that handles that stuff? I was terminated once by a company that claimed they went bankrupt, so I filled out one form with the labor commission (had to in order to claim unemployment benefits - I was 22 and had no savings), and they got ALL up in my former employers business and straightened it out quickly with no extra effort in my part.

Wow, collections..

I have a similar story: as an IBM employee I had a company Amex card. It works like this: you buy something on the card (in your name, against your personal credit), but you also fill out an expense reimbursement form (a horrible Java web-app) and IBM pays the card.

All fine and good, but if you return something or if the vendor gives you a discount later or anything like that, you end up with a negative balance on the card. Now you leave the company, but you have an Amex card with a negative balance.

I received Amex statements for six months before they figured out how to deal with it.

This reminds me of the part in the Robert Moses biography when he was made responsible over implementing a more meritocratic employee review system for the government. Just like this story, that one also ended in flames, though Moses made it out all right in the end. I wonder who the Moses character in this story was, and if we'll read about it in a biography in a few decades.

> Canada to scrap IBM payroll plan gone awry costing $1B. The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors 10 years ago to centralize the government payroll.

Is it me or is this wording terrible?

Trudeau is not a Conservative. It makes it sound like it is Trudeau's party that caused the problem.

Yeah why is his name there at all?

And why isn't "The Harper Government"(TM) not named?

I'm not a fan of the convservatives but it's still unfair to place the blame of a bad choice of client on a political party. A lot of non-partisans were involved in getting the project running.

Sure, but it would make more sense to refer to them by name than as "Justin Trudeau's Conservative predecessors" or whatever.

So: for anybody who is curious what's actually going on here:

1. Problems are far bigger than the specific payroll software system. There were tens of thousands of outstanding cases in 2015 before the new system went live. There were tens of thousands of outstanding cases in 2012 when the project started. And there were tens of thousands of outstanding cases every year before that on the "old great system". (why? read on)

2. IBM didn't create a new system from scratch - Govt chose PeopleSoft HCM, an industry-leading system which reliably pays people day in and day out at thousands of customers. Base solution is not the core issue.

3. When you buy PeopleSoft (or SAP, or JDE, or whatever), ideally you make your business fit the system's best practices, customize as little as possible. No client fully does it, but govt doesn't even try. IBM, together with govt employees, spent five years trying to hammer several tens of thousands of complex time and labor rules from more than hundred labour agreements into the system.

3. Implementing this system was half of the initiative, other half was reducing workforce for compensation advisers and moving them all to NB, hiring new grads as replacements. Who were surprisingly not exactly intimately familiar with these hundred labor agreements and tens of thousands of time and labour rules. We're talking one third of the support force at one tenth of experience.

4. So, the billion wasn't spent on "the system". It was mostly spent scrabling afterwards to re-hire old CA's as expensive consultants and trying to catch up the damage.

5. Because guess what, garbage in, garbage out. If your manager doesn't submit or approve your acting gig or leave of absence or maternity leave for several weeks, or months, or more, system won't magically know about it. Not the old one, not the new one, not some magical future one.

If your HR person doesn't create you as a new employee, you won't get paid.

If you don't submit your overtime slips on time, but instead collect them for 10 months so you can have "Xmas bonus", that also MAY just may create an issue. Ditto with retirement applications. Secondments to other departments. And if through this all employee groups perform denial of service attacks on the system because this is all a political game, that may also be a problem.

There is a pervasive cultural issue that boggles mind as to why nobody does their HR things on time or correctly there. The system just provides useful target of the moment, but issues were there before and without cultural change they'll be there in the future.

There are real problems and there are real people suffering because of them, and they have for decades, and yes it's gotten worse. Grandstanding over some magical new systems won't fix it though.

This is a burner account, I had the dubious privilege of working on this few years back and am aware of some of the horror stories - from the present and the past. I'll stay the dear heck away from Federal Govt / Public Sector project in the future if I can.

As a manager who is on the receiving end of GoC payroll, you make some excellent points. The key ones are: 3 - reduction of corporate knowledge. My pre-phoenix comp advisor could make shit happen. Now, I can't even get a return phone call.

Likewise, pt 5 (and 3). Example; pre phoenix, our summer students would always have a pay glitch end of July - why, comp advisor on holidays, didn't properly brief their replacement, stuff fell through the cracks.

I think the scale of the problem and suffering is much greater with this implementation than it was under the disparate departmental systems.

Not to be political, but, really that plays into this while conversations is that what I don't think can be ignored in all of this though is that the Harper mentality of consolidate to save money hasn't worked in any regard; just look at Shared Services - try getting timely procurement, and oh, BTW, no spares allowed to be kept... complete clusterf... as well. That's a whole other mess that affects people in a different way - oh, your phone line is down; tech will be there in 2-3 months.

"ADP tech chief Stuart Sackman says his company is under constant attack globally and is hoping to encrypt more of its data using the new technology. A sixty year customer of IBM, ADP currently runs its largest domestic payroll engine on the systems, calculating 20 million paychecks per pay period, while also running its money movement, tax filing systems and cloud-based software suite on versions of the Z-series mainframes."

https://www.forbes.com/sites/alexkonrad/2017/07/17/ibm-seeks...

As someone working in ifosec, i loathe how vague that article is about what exactly is going on in the chips.

Does anyone know what the real problem was here?

> Some people were paid too much, others not at all. Those issues snowballed with the deluge of requests to fix incorrect paychecks.

> IBM said it’s “fulfilling its obligations on the Phoenix contract, and the software is functioning as intended.”

Okay, so the cost is coming from incorrect paycheck values, but the software is working correctly?

Does that mean it's human error? Can these issues be traced back to one or a few major instances of human error? Is this a widespread issue caused by a confusing or misleading UX design?

The article is quick to talk about the projects costs and failures as well as the politicians involved, but I can't learn anything or blame anyone until I understand what went wrong.

My guess is that IBM believes the software works according to specification. And maybe it does. Maybe the specifications are wrong. Most likely, it's a combination of bad specs and programmed logic errors that don't always follow the specs that are correct.

Of course, this requires testing and time to hash out. It would be a major mistake to scrap a project in it's final phase of refinement. They should have switched over slowly, department by department, handling and resolving issues along the way. But scrapping the entire project now only means another $1B investment and repeat down the road. Trudeau would join the ranks of the FAA in flushing multi-billion dollar software projects. It's a waste. I don't know about this project, but software engineers died on-the-job working on the FAA's AAS that never launched.

From the anecdotes I've heard, it sounds like the support staff is just wildly insufficient to handle all the issues that crop up with a major change like this. Initially the government switched over a trial group of (iirc) something like 40k employees. There were a bunch of problems with people not being paid or being paid the wrong amounts, but for some incomprehensible reason (perhaps they wanted to ditch the old system to focus on fixing the new one or something crazy like that?) they switched over everyone before fixing those.

Now, I personally know people who've been getting incorrect pay for over a year, who submit a complaint to their HR, who pass it up the chain, where... nothing seems to happen. Presumably because the 500 people or whatever it is tasked to deal with this are literally dealing with tens of thousands of issues, many of which aren't trivial changes (and so block the many that actually are trivial changes, but aren't even being seen).

Basically it was released before it was ready, initial signs of issues were ignored, and now the human side of the system is overwhelmed. It's kind of like when a web server gets overwhelmed with traffic, and it isn't set to reject at a reasonable threshold, so things start getting queued and timing out, people start refreshing pages, and it just snowballs until the whole thing goes down. Which apparently is what has now happened.

> a trial group of (iirc) something like 40k employees

Oh boy.

Don't quote me on that number; it's from memory. Point is, it was a fraction of the total, and then they released it to everyone without solving the problems in the trial group.

Totally OT but

    The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors 10 years ago

seems an odd way to refer to Stephen Harper's government.

The only solution to tax / accounting law and software problems is to create a reference accounting system and set it as a applicable law. In other words, to describe the law in unambiguous language.

Do we really have to write a law in the same way as Romans did? Math as a science made progress after it invented a new precise language for itself.

I'm not talking about the whole law. Only about that according to which the software is built. Since no one calculates taxes today by hand, it is worth to write the law so that it can be easily applied.

Ibm global services, Accenture, computer associates, Oracle. See a company sign a big contact with them or their ilk, short that co. Their whole raison d'etre is to rip off old assholes in the c suite who can't find the on switch of the company or government money they're meant to be protecting.

Hiring these consulting firms should get you fired every single time. It's a wonderful indicator of total incompetence. Notice how they're all deep into Washington? Yeah. Makes me sick these companies exist.

Two common factors in a lot of these stories are "IBM GS" and "Oracle"...

Would you sign to build a project who's MVP is 260,000 salary calculations with rules poorly documented and spread other hundreds of agencies?

Maybe if they started with a subset and slowly built up..

This problem was likely better served by many smaller systems that better fit the specific needs.

Government is way too complex to try to cram all rules into a single monolith like Oracle's PeopleSoft.

If what I've read is accurate, the various departments were already served by smaller systems that fit their specific needs.

The intention with Phoenix was to save money by consolidating them all into one. Oops.

At least part of the problem is that projects like this don't end up being awarded to those who are most capable of completing them successfully. Instead, they end up being awarded to companies and consultancies who are experts at shepherding proposals through the RFP process.

Anybody have any insight on what it feels like working as consultant on such a huge failure? I (or my department, but honestly mostly me) once failed, as a consultant, to successfully deliver on a $50k contract and I felt like absolute shit for weeks and had trouble sleeping. I'd hate to think how it would feel to fail on this scale. Or is it a situation where the whole thing is so large and diffuse that no one can honestly be said to be responsible.

I'm surprised there weren't any financial penalties for the IBM practice of over promising and delivering a broken system that generates more problems than it solves. They underbid and embellish proposals with BS to get the contract, staff if with overseas workers to get the maximum return and then hope for the best. This is how IBM does business and it's unfortunate that they continually get away with it by profiting on their incompetence.

The Big Rewrite. What can go wrong?

I am guessing one of the reasons is probably the software selected... oracle

Oracle Peoplesoft is used successfully at a number of organisations so blanket statements aren't helpful.

But I would question whether it is better/cheaper in these cases to just write bespoke software from scratch rather than trying to modify a vendor's existing product.

Oracle/PeopleSoft can do payroll for a defense contractor with tens of thousands of employees across multiple nations.

I think proper configuration is what's lacking in this case.

Oracle costs a lot. It infects your entire stack, from hardware all the way up to personnel, and drastically inflates the cost from end to end.

With no real utility advantage, either.

That's not relevant to whether or not the software is capable of performing the task.

Everything infects your entire stack. If I hire Java, Ruby, PHP etc people then everything from hardware up to personnel is going to be selected to optimise the delivery and execution of software. Kind of common sense that you would do this.

I don't disagree that Oracle costs a lot and that they inflate margins pretty poorly but to say they have no real utility advantage is a bit ridiculous. Most of the world's enterprises are running core business systems off Oracle and if they move to the cloud it's usually to Oracle instances in the cloud not any other system.

Proper configuration, and proper business needs mapped, and no moving targets, and probably a lot more, we'd have to be on the project to know.

Likely not true.

Software implementations usually fail because of the implementors failing to understand and map the business needs prior to beginning in detail, vs launching with whatever and making twice as much money fixing it after the fact "as you go".

I have lead a few full cycle ERP implementations, and replaced vendors along the way if they were not performing.

The challenge remains is software vendors are not sufficiently motivated enough to learn the business in detail before implementing software to help run the process.

It's a mix, I've lead many enterprise implementations, erp, crm, etc. It's not just the implementors, it's all of the above, from the lack of the customer even being able to communicate the business needs.

It'd be lovely if you were right, because the problem is then fixable, but you can take the most motivated, best team of SMEs into a project, and be unsuccessful given the client.

It's all a mess trying to get things done, and every single company is different. Going onsite that first few weeks is an exploration into in what exact ways the mess is going to manifest itself, and how we're going to fight our way through it.

I agree. To clarify, I'd like to believe any client in the position to spend money on an ERP size solution (including payroll) has earned the right to do so - be they got large enough to have a problem by luck, manually inefficient processes, outgrowing existing systems, etc.

There is still a responsibility in my mind on the implementors part to be experts who can assess the maturity in place at a clients operation, as well as the degree and potential depth of unknowns, as well as their bench's ability to be experts of closing the unknown.

Too often, implementors are focused on landing clients, more than passing on work beyond their pay scale due to trivializing or outright ignoring things.

Every single company is totally different - a desire for implementors to want to templatize as much as they can may be part of the issue.

I'm being a bit harsh on implementors despite being one, I think it boils down to if you don't understand a business in detail, including it's data, before beginning, one should probably not touch it, whether its a client or the other side.

I'm certain one of the reasons is that all kinds of things were selected, before getting to the meet of the requirements. Which is typical of large government IT projects, where the order is often: choose some of the technology stack, write an contract opportunity (RFO, etc.) around the chosen technology and a vague (even if stated in terms that seem precise) concept of the requirements, select a contractor, who then pre-determines more of the tech stack based on their own offerings, business relationships, or competencies, then starts gathering real implementable requirements and building the customer-specific elements of the software (and even the last two are sometimes in the reverse order.)

..and then start backing all that in to existing long standing customer business processes which aren't going to budge.

And then you end up with the OP.

I wish governments would just get on with it and open source their functionality for those of us who want to "serve our country"

Passively wishing does not tend to influence government policy.

Don't mistake my HN comment as being my _only_ action.

Have you heard of Open Source? Government of France has. Enjoy the article. https://www.enterprisedb.com/blog/france-takes-lead-new-open...

I've never seen a PeopleSoft implementation go well. It's always underbaked and overpriced.

I am genuine curious - Are there any recent examples of big, long but _successful_ projects that involves these 2 companies? (IBM and Oracle are mentioned in the article)

There seems to be a gap to fill for companies/organizations building software for government. Wondering which reputable companies are filling this gap?

Former ibm employees are even toxic. The cmo of openlink is a former ibm cloud marketer that has effectively run out the entire marketing staff

I wrote my own payroll system for my house keeper, and it is surprising hard to get taxes right... I am not surprised.

Reply on news.ycombinator.com