IBM to pay more than $30m in compensation for Australian census fail
abc.net.au
abc.net.au
Unfortunately no one seems to have considered that the majority of users would do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the time frame and it was never going to work with a 1 million per hour capacity.
What is interesting is with IBM paying out they must have failed to deliver the required capacity. I thought it more likely the government had not tendered it properly and requested a lower capacity than needed but obviously IBM just tried to cut costs and under resource the operation.
Which brings us back to another question: was there even a DOS attack? I've seen any little technical details on the attacks, and I know myself, like a lot of other Australians logged on behind vpns. When the site wouldn't load, of course we hit refresh...
The report calls the DoS an "event" rather than an "attack", carefully worded so that tech journalists would pick up on the wrong thing, but clear that it was not malicious intentional traffic.
The DoS was the Australian public attempting to use an online government service.
https://www.righttoknow.org.au/request/abs_preparedness_of_2...
http://www.businessinsider.com.au/ibm-layoffs-1-month-severa...
I mean how do you get away with hiring IBM (or accenture, PwC, didata, deloitte or, may God help us, Tata or something like that, where cost of paycheck is literally the only factor in hiring).
They also know the people judging them are in the same boat.
Thus they fall back to the only thing understandable to themselves and people who will judge them: big brand and prestige/shininess. If the project succeeds, you take credit. If it fails, you can't be blamed for "buying the biggest and the best" (even if the biggest and the best are just a bunch of stary-eyed ignorant grads with a salesman/project manager at the head).
/that's the good version. The bad version points out how many of these firms gets business through networks, personal connections and "you scratch my back I'll scratch yours" type dealings. //it's not a rant if it's true.
I cynically suspect that whoever drafts these contracts might even believe that by agreeing on an excessively expensive failure clause, failure will thus not occur — after all, it is legally covered.
We see this in The Netherlands as well (and I suspect it is a global phenomenon). Government needs software, big-name consultancy firm gets the job, builds stuff that won't work or barely functions, project implodes or is delivered in a half-baked state. Repeat ad nauseam. These companies (Cap Gemini comes to mind) thrive by these projects, even though most seem to fail or end up underwhelming. There appears to be some degree of corruption there on the side of these contractors — investigative journalism has uncovered several questionable practices.
Some part of my wonders if having a well-funded government department dedicated to these projects that creates and maintains the (free!) software needed is not a more tenable (and cheaper) approach. On the other hand, large government-driven software projects may just be impossible to deliver at this point; we simply lack the know-how to prevent the inevitable implosion of such complex projects.
I've heard reports that they spent >AUD$5Million on just the load testing aspect, but I hope someone can verify that. And I trust that they have improved comprehensive testing methodologies that actually work to improve reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus Notes??</sarcasm>)
I guess we can be thankful that is was 'merely' a census and not something like electronic voting!
See: http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...
For as much as I've heard about this census issue, I thought it was a bigger (monetary) deal.
It was embarrassingly insecure.
(Doesn't change the annoying niggle of conflating a state with an entire country, but I guess Americans are all Alabamans in that regard)
I don't see how IBM thought they could pull if off for $9.6 million [0], especially with the ABS requirements for information handling during the census.
"Key measures to safeguard information include strong encryption of data, restricted access on a need-to-know basis and monitoring of all staff, including regular audits. After data collection and processing, the ABS removes names and addresses from other personal and household information. Names and addresses will be stored securely and separate from one another. No one working with Census data will be able to view your personal information (name or address) at the same time as your other Census responses (such as age, sex, occupation, level of education or income). Stored separately and securely, individuals names will also be substituted with a linkage key, a computer generated code, completely anonymising the personal information. Only these anonymous linkage keys will be used by the ABS to bring data sets together." [1]
Currently, it seems IBM is blaming their subcontractors, "While IBM issued a mea culpa, they were then quick to lay the blame at the feet of Australian communications company NextGen Networks and its upstream provider Vocus, blaming them for failing to implement a geoblocking service called Island Australia to prevent traffic reaching the site from outside Australia." [2]
And considering IBM's has a revenue stream of $20.24 billion this quarter [3], the fine won't really impact them.
As to their reputation, IBM doesn't have a brilliant name already:
* Failure to meet deadlines [4]
* In Queensland they were overbudget, overtime, and under scope [5] resulting in their permanent ban from working with the Queensland government.
* They have a great habit of building things that they call incredibly fast, but never perform up to spec in the real world [6].
Anecdotally, National Mutual contracted out to IBM for some "supercomputers" for their COBOL applications, to speed up processing time. The machines were half the speed IBM claimed. IBM's solution? Double the number of machines delivered, and ignore the extra costs of power, maintenance and network load.
[0] http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...
[1] http://www.abs.gov.au/websitedbs/censushome.nsf/home/privacy
[2] http://www.news.com.au/technology/online/security/ibm-faces-...
[3] http://www.businessinsider.com.au/ibm-surprises-with-q2-2016...
[4] http://www.abc.net.au/news/2016-09-27/ibm-unlikely-to-hit-cu...
[5] http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...
There must be a big penalty looming for such a massive level of incompetence.
Phoenix seems like a rather awkward name for an application.
http://www.nytimes.com/2016/11/18/world/canada/government-wo...
>Though many government employees work in 9-to-5 office jobs, the public payroll also includes a wide array of people like Mr. Ryan with complicated work schedules and pay rules.
My father used to be in the Coast Guard he worked 28 days on and had 28 days off. If you were injured that meant the days you were off were really two days?? Try explain that to some bureaucrat in Ottawa who can thinks they are speaking English (really French), they're 2,000 miles away and doesn't understand the system either.
For instance, IIRC the census asked for my and my wife's country of birth and nationality, but later on when filling in the info for our children, it asked for the nationality of the both parents again. Simple logic checking back through the data should have shown that our (the parent's) information was already supplied, and this information for our children could be pre-filled or skipped and auto filled in the database.
Just a few small things that would have made the filling out of the forms a lot smoother, faster, and less complicated for lay people who were already nervous about doing the data entry.
You might argue that it would work for the "most common scenario". In which case why not pre-fill all the data to the statistically most common values? (Hint: the census's purpose is to collect those statistics.)
Also, pre-filling might cause confusion as to which fields have already been filled out by the user; if a value is present, they might wrongly assume no action is required on their part.
(Having said all that, I haven't seen the online census.)
I am relying on my memory here from few months ago, so I could be wrong, but I did feel at several points during the session that quite a few questions could quite possibly be chained together in better fashion. Perhaps not so much a failing of the actual site UX designer, but the census analysts who designed the questions in the first place?
The questions are largely derivative and the same each year: cost to develop and test while maintaining statistical validity and capture of all possible scenarios, while still being interpreted correctly by the population, is enormous.
Secondly, the number of permutations and combinations of answers is almost universally underestimated by everyone I've ever met outside of this process. Nationality is not the same as birthplace is not the same as parent is not the same as adults in the household. The amount of diversity is staggering.
Additionally, although those two are the primary reasons for the question designs (and I assure you, people will be doing deductions and logical inferences on the back-end once the data is collected), being exhaustive at the point of data entry rather than deductive allows for further meta analysis on the data entry process and survey responding experience itself, whereas if you decide to attempt to deduce, you miss out on a dimension of typo, misskey, misinterpretation, and other analysis data that you would forgo in the former case.
It seemed to be a single page application because once you got started it was fast to get through the process. I suspect the SPA nature was a deliberate decision to offload workload from the servers. Imagine how much more traffic would been needed if every page/section on the form required another call to the server.
Design consideration has to be given to how/what people will be using to access said questions.
If it looks like a web form from the 1990s , ask yourself what kind of technology some Australians might be using to try to fill out their census form, and how standard we have to try to keep that interface/experience.
Based on non-cloud setups like http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...
A rack or two would have been able to handle the load.
SO can work on such a small amount of hardware because they've got people who either know the entire infrastructure from ground up, or have the skillset and interest to figure it out. Plus they have very tight coordination between the folks doing the development, and running it day to day (possibly because it's the same people).
Your average developer working on a government contract through $BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset or motivation to learn it. Even in the unlikely scenario that they did, they're not going to be in a situation where they can act on it. It'll be a case of developing to specification, throwing stuff over a wall to some sort of QA, and then throwing the result of that over to someone to run it. The whole thing will be run by project managers who have opex budgets to meet, on infrastructure that was specified well before the thing was ever built.
However it was all non-cloud self-hosted hardware.
Because it's bloody hard to DDoS paper forms. Widely distributed atoms are curiously disinterested in IP packets.
It says the damage was estimated at 30 million. It says IBM paid something. It didn't explicitly say IBM paid $30 million.
"Computer giant IBM will pay more than $30 million in compensation for its role in the bungled census"