IBM's reputation at risk in wake of census bungle
smh.com.au
smh.com.au
The big problem is that it's not beneficial for IBM/Oracle/Accenture/$consultancy to quickly deliver a stable solution that works well. They make much more money by having seven layers of middle management committees vote together a crappy solution that will need five years of support/upgrades and then a replacement.
In other industries, like construction, the same fundamental problem exists, but there human lives are at stake, standards bodies have been formed to prevent it, and non-software engineers aren't outsorced to people in India who have as little knowledge as they have pride in their workmanship.
If a contractor built apartment buildings the way IBM and friends build software, they'd be sued out of existence. I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all? It happens relatively often with software consultancies, usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice.
Construction projects go over budget and are late all the time. Remember the big dig? https://en.wikipedia.org/wiki/Big_Dig
The Sydney Opera house was 10 years late and 1475% over budget: https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_...
Anyone who's had renovations done to their house can also probably sympathize.
Private construction? Not so much, certainly nothing like 3x the estimated cost.
https://en.wikipedia.org/wiki/Plaza_Rakyat
http://gizmodo.com/the-sad-fate-of-the-worlds-six-tallest-un...
But rather than split hairs: I interpreted the underlying assertion being that construction is more predictable and reliable than software. I've heard this in other contexts as well. And that didn't pass the gut check for me, because I can think anecdotally of many construction projects (and home renovations) that were way over budget and took longer than expected.
Also something to consider in a circumstance like public transport stuff is that the multiple approval levels and meddling happens on the Government side as well. Like a perfect storm of bureaucratic bullshit that ends up making an enormous bloated mess that does everything but what was originally intended.
1) financial implications (you were paid to develop a solution. we, the customer, have discovered that you were donated time & code. did you disclose your potential status as receiver-of-charity during your initial bid? do we need to hold a recompete because other bidders did not know to use free labor in their bid? how much code was donated? does this reduce the average wage of developers on the project below minimum acceptable standards? can you prove it?)
2) product liability implications (this free code you used is not working properly. people are dying or becoming impoverished as a result. we are going to the donor-developer for doing this, his employer for allowing it, and you for failing to provide sufficiently rigorous integration testing.)
3) support implications (your free code broke, and nobody can get ahold of the rando who wrote it, and the company he worked for at the time disavows responsibility. everyone is going to get sued.)
4) due diligence implications (did you use this code because it was free or because it was correct? can you prove it? was this a cost-cutting measure that may have deprived me, the customer, of a superior product? can you prove it? don't answer me; answer my legal team.)
I know it sounds extreme, but I've seen each of these things occur, at differing scales, over the years, and the reason companies like IBM exist is to deliver defensible solutions. This is not startup-land; it is not sufficient (or even, often, necessary) to deliver a working product -- the product must be able to withstand a Congressional investigation. You don't have to prove it correct, but you do have to prove you adhered to legislative guidelines intended to control the process.
This is also the reason behind the headline regarding IBM's "reputation." Nobody at IBM gives a shit what any of us think about their technological prowess or even their project management skills. IBM's reputation is an ability to operate at a massive scale without getting nailed to the wall by some government or another. If they get their ears pinned back in this census situation, the takeaway is not "IBM tech sucks" or "IBM PM sucks" -- the takeaway is "IBM is not a safe company for politicians to hire"... and for an IBM, that is fatal.
Again, I agree the whole system by which IBM operates is disgusting and wasteful, but to compete in this space you must become this space, as it were.
I don't think their process is heavy because it's rigorous, I think it's heavy because it's a reflection of their corporate structure (which is large).
The process rigor encourages checking stuff off of lists, but discourages the little ad-hoc fixes that make a product good. Nobody's going to fix the minor bug in the date entry field if it takes 4 hours of "process" to write the 4 minutes of code.
I work for IBM and believe me, the process and paperwork gets super tedious, but I understand why. The number of times I've had a client threaten to sue for breach of contract is staggering. If we didn't have a ton of process and paperwork showing a trail of evidence and every communication that led us to the final result, we'd be out of business so damn fast. Unfortunately some people really like to take unfair advantage of their vendors.
Let me give you an example. Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted. The real event was more complicated, but for the sake of anonymity, let's say they wanted a weekly report as a bar graph, and when we delivered, they realized a pie chart made more sense. Now, being the good consultant I am, I switched the bar graph to a pie chart. It wasn't in the statement of work, but it only took a couple of hours and I like to keep my clients happy. After they got the pie chart, they said they liked the format but they wanted it switched from a weekly to a monthly report. Sure, it only takes a couple of hours. And then they wanted the monthly report, but the data resolution broken out into weeks instead of days. Only a couple of hours. And then they wanted the name of the report changed. A couple more hours. Bear in mind, this is all 100% custom development, from scratch.
Eventually we ran over budget and over time doing these minor tasks, because what the client asked for wasn't what they actually wanted. They demanded we keep going until they were happy. They hadn't signed off on this work being completed, so they claimed it wasn't done according to the contract.
A few weeks later we got a letter from their legal department, and in return we sent them the statement of work with everything checked off, and all of the emails we had sent back and forth between us and the client, showing where they acknowledged the scope was changing and we had said we would try our best, but couldn't guarantee anything other than what was in the SOW. We also showed where they refused to sign an extension of the project timeline and refused to sign a SOW change to formally add the new scope to the project. And that was the end of that story.
Every step of the way when something changed, even if we discussed it on the phone, we had a follow-up email to confirm. Everything we did got documented, every change was clearly spelled out as to what we changed and what was the original agreed-upon scope. Every scope change was approved by the PM and sometimes, one of my managers. And at the end of the project, we deliver all of this (sometimes hundreds of pages) in a nice, neat report detailing to the client what they hired us for and what we actually did. It takes forever and it doesn't guarantee the end result will be perfect. What it guarantees is that if we get sued, we have all the paperwork proving we did exactly what was agreed to in the contract.
This is essentially always the pain point for "enterprise" software, and essentially always it's because the actual programmers communicate with the actual users through seven layers of middle management at both client and consultancy. Because middle management has to justify their existence, and are mortally afraid of being cut out of the information loop.
That's why I always give a little leeway if I can. If they want their report changed from monthly to weekly, I'll do it. But sometimes it becomes clear they're just trying to take advantage of me and get extra work done for free. After all, why would they pay for a bar graph AND a pie chart, with both of them being shown as monthly and weekly reports, when they could just pay for one and then say "oh that's not what we meant, can you change that?"
I certainly will change that, no problem. But only if I have hours left after I complete everything listed in the original SOW.
"Securing revenue through enterprise development contracts" is a different discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)
Process meant to dissect problem to the right size for predictable execution. Right size does not translate to always the same dissection for vastly different sized problems.
For a tool as simple as the parent post describes, what IBM does is not process rigor.
My recommendation is that if you ever think about using an IBM product, hire a developer instead. You'll pay less money for a developer than you would pay for an IBM product, and after they're done implementing a the product better whatever unreliable enterprise clusterfuck IBM would shoehorn into your "solution", you have a developer who can work on other things.
Whoever suggests using it should be fired on the spot.
If I see any mention of it on a person's CV (because I only mention what I am ok with), is enough for me to second-guess this person's ability
I'm so glad git is replacing it.
The only "problem" with developers is they usually don't throw golf days for their best customers :)
I can't really speak for other IBM products, but I suspect they generally align with your perception here.
I found this headline kind of amazing because I was like, "reputation at risk? This has been my impression of their reputation for decades."
""If you think about something like what goes on in Nauru, if a child is injured it's not the government that's responsible, it's Serco, Transfield or whoever else," she said, referring to the private contractors that run Australia's offshore detention centres."
Private offshore detention centers for children?
What's going on down there?
http://www.abc.net.au/news/2016-07-25/four-corners-evidence-...
https://www.theguardian.com/australia-news/2016/jul/25/teena...
https://www.theguardian.com/australia-news/2016/jul/26/abu-g...
https://www.theguardian.com/australia-news/2016/aug/10/the-n...
http://www.npr.org/2016/08/11/489584342/claims-probed-of-bru...
https://en.wikipedia.org/wiki/Immigration_detention_in_Austr...
Australia essentially pays Nauru to host an immigration detention centre, and sends asylum seekers who tried to reach Australia by boat there, regardless of age or legitimacy of claims or anything else.
Yes, plenty of countries let you turn up at the airport and obtain a visa on arrival - but if you look like you're moving in (no onward tickets, etc) you're at the very least going to have an interesting discussion, and maybe get a free flight back.
This particular situation is undeniably complex.
> ... sends asylum seekers who tried to reach Australia by
> boat there, regardless of age or legitimacy of claims
> or anything else.
I think a big part of the problem is language. Like the life/choice descriptors used by people with strong opinions on abortion in the US -- hey, who doesn't love life and choice?
In AU one side will call these people asylum seekers, or refugees - the other will call them illegal immigrants. In many cases, both appellations are technically correct, but I think in almost all cases both appellations are non-helpful.
The 1951 UN Refugee Convention (Australia is a signatory) recognises that refugees have a lawful right to enter a country for the purposes of seeking asylum, regardless of how they arrive and whether or not they hold valid travel or identity documents.
http://www.asrc.org.au/wp-content/uploads/2013/07/it-not-ill...
https://www.theguardian.com/world/2016/aug/10/a-short-histor...
It's more reflective of poor procurement discipline than anything else -- whomever put out the bid let lobbyists or high level people do their thing without appropriate governance or penalty. It's a relatively small contract, and without clear penalties for poor performance this type of large contractor will do as little as possible.
From a purely technical perspective, there's no way to know if something "works" if it hasn't been defined or - worse! - is constantly being redefined on the fly.
That is true for oh so many government IT projects. The part that I still struggle to comprehend is "Why on earth aren't the IT companies, like IBM, telling governments that what they're asking isn't actually possible?"
The Danish government have an insane amount of failed IT projects. Most of them fails do to the complexity of laws, rules and requirements. We even know that they'll fail before they even start. Most good developers are able to predict this with great accuracy. So why aren't IBM and other telling governments that what they're asking isn't possible?
My best guess: Because their lawyers will ensure that they're paid anyway, and because they have absolutely no pride in their work.
You can't hurt the reputation of IBM, or at least it won't matter, because their competitors have equally poor reputation, and no one else is going to bid on the jobs they take on.
The problem is not lowest bid (which few government contracts are any more, AFAIK), but the fact that government contracting is its own domain of expertise (actually, often a separate domain for each state plus the federal government, and even many local governments) separate from the kind of work actually being contracted for, and there are in any field a very small number of players (compared to the field as a whole) that have developed expertise in the "winning government contracts" domain (and, in IT, IBM is one of them, at essentially every level of government.)
Of course you can pay more.
-- OracleLast minute changes to a law with a looming implementation deadline so that someone can score political points = everyone is non-compliant for some period
You can't rewrite 30-year old COBOL-backed business processes overnight.
As an Australian, I hope we can move far away from such contractors. The amount of waste is insane. It reminds me strongly of the initial Healthcare.gov contracting failures.
[1]: http://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-4...
On Healthcare.gov, there was no primary contractor who had authority to manage the other contractors building various pieces. Solution: hire an IBM.
On the Australian census in the article, it seems like they only contracted IBM to both manage and build the project.
My takeaway is that you can get a decent result in large software projects, but (1) hire a party whose sole role it is to manage / conduct ongoing audits of work, (2) set the contract such that the managing party has a mildly adversarial relationship with the implementing parties, & (3) cover off cost overruns in the initial contract.
The Healthcare.gov debacle and resulting "start-up culture" resulted in the US Digital Service, which terms themselves as "a startup at the White House".
http://www.theatlantic.com/technology/archive/2015/07/the-se...
Having to be in a DC alone makes them much more vulnerable to a DDoS (I remember our provider would just cut us off first and then ask questions later is they spotted bad traffic).
Really?
The specs: The online census form was a pretty simple client side app. I'm going to say, I could whack something together in much less than 200k of JS.
For each household the site needs to:
1. Serve the client side app. The app had no images beside the ABS header. I'll be generous and say 200k of static assets including the client side JS.
2. Accept & validate the code households received on their postal forms
3. Save the final result (a single record) back to the databases. A single append operation. Being generous, 20k of JSON data per household.
When I filled in the census on Thursday night I noticed it made no effort to normalize addresses or anything else. Umless I'm missing something (or IBM turned off half the features after the DOS) the site was that simple.
1. 200k of static assets * 10 million households = 2TB of traffic. AWS would serve 2TB of traffic out of cloudfront for a whopping $200. If you don't trust 'the public cloud' your self-run data center could serve a tiny payload which downloaded the JS from cloudfront and verified its hash before running it. Or just serve all 2TB of traffic directly.
2. Accepting the unique code on the posted form is easy - just have a few servers which store the database of all posted codes in RAM and respond with 'yes' or 'no'. The codes for 10M households will fit in less than a gig of ram.
2. Saving the final result is pretty trivial. You need to accept 200 gigs of JSON data in an append-only journal. The peak write load would be around 4 million requests / hour, or cumulative ~1000 writes per second, or ~20MB/second of data. For an append-only store backed by an SSD this isn't much load at all. If the numbers weren't so low I'd suggest using a hand-written journal, but honestly most off the shelf databases could handle this write load without breaking a sweat. And you could easily shard it if you wanted to, and just aggregate back the data after its been collected. (You might want to actively geo-replicate the data immediately too, while its being collected just in case something happens).
In total, I'd feel comfortable running this entire site out of a few dedicated machines in just about any data center in the country. If you're feeling paranoid geo-replicate it, and actively stream received census submissions to a couple offsite backup locations. Total whiteboarding time: about 20 minutes. Might make a good interview question.
Of course IBM was also paid for consultancy time, and engineering time to design & implement the site itself, but even factoring in healthy margins I can't help but feel our government was ripped off by a factor of ~10-20x.
First off is project mgmt and coordination. On the government side there's going to be a small host of people involved, not just a single "customer". Meetings and more meetings. Somewhere between 50% and 100% of someone's time is going to be spent just dealing with them. And that's on top of the normal PM effort (so this could easily be 2 full time people).
You've also left off the generation of the mail outs. This means a data load, an id assignment, and a "print file" for the printer/mailer. High coordination costs (PM now dealing with the gov't and the print shop).
Security. Ouch. Need to make some effort to ensure the mail out codes can't be guessed easily... but are still small enough/readable enough that folks can use them (i.e. a GUID would be perfect except for the low usability). Further there ought to be some sort of post processing that attempts to ensure the data is good (i.e. no one played silly buggers and guessed codes).
On the application side "just in case something happens" isn't good enough. Once the incoming census data is accepted it must not be lost. The statistician would go ballistic (what do you mean you don't know how much data you lost?) and there's really no way to recover lost data without sending out new mailings.
On the back end is the final "give the stats guy the data" step. Conceptually just a giant CSV file... but how likely is that really? Also somewhere/somehow someone needs to build a list of households that have not completed the survey (and reporting on how many/what percentage and eventually producing lists of households for census staff to visit... which is probably handled internally by the existing systems... but it's another entire interface).
And all the design back and forth of how the website should look (and it has to be properly accessible). Not really hard to do on IBM's side... but the gov't side is going to be a nightmare of micromanaging UX specialists.
Oh and general security... it's a web app so there's all that. Hopefully the testing would be handled by a separate organization... which means another organization to coordinate with (PM is gonna be busy).
And of course all the QA you can stand.
Plus... there's legal costs (that contract ain't gonna review itself).
Plus... there's the upfront costs of the actual bidding process. Can't bill for it... but this does mean that every successful bid by IBM needs to be higher to cover these sunk costs (i.e. if IBM wins 20% of all its bids then the costs for 80% of the failed bids need to be covered by the winning 20%... if you see what I mean).
And now IBM needs to pay for some PR/damage control.
In the end I'm not so sure that $9 million is so very out of whack.
And damn it all if I didn't forget to include something for planing/supporting defending against denial of service attacks...
For the database, you'd want to use caching. When the form is submitted you write the data locally and queue it for replication to a central server. To redisplay you'd try to read it locally and if you don't find it you'd read from the central server. You'd probably want to use the household's code for sharding so that you're going to the same local database each time.
I can't imagine it'd be necessary to spread the load out geographically. I'd have a single datacenter / cloud equivalent set up with load-balanced machines for the front end and the database tier, a full fail-over backup in case of critical failure of the main datacenter, and a separate database server for the master which the others replicate to and clone from.
This is a six-figure project, for sure, except maybe for time spent dealing with governmental red-tape.
Corp sector in Australia is on the tail end ridding themselves of IBM nonsense after a decade of utter incompetence.
Unfortunately the gov sector catches on slowly.
Sadly a lot of places don't learn though and just replace IBM with TCS/TechM which is somehow even worse.
And don't get me started on TCS. Fucking shitshow.
I miss this IBM.
There's been a bunch of discussion in Australia that there likely had been no "attack" and that it was simply just terrible planning around expected loads the night of the census.
This happened in canada. https://www.google.ca/webhp?sourceid=chrome-instant&ion=1&es...
Too soon?