Public servants sit idle as Australian Tax Office's IT melts down
smh.com.au
smh.com.au
1) http://www.intermedium.com.au/article/hp-enterprise-services...
2) https://delimiter.com.au/2012/06/08/hp-starts-cutting-aussie...
3) http://www.theage.com.au/national/public-service/thousands-o...
This might be a bit of a cheap shot, not knowing the exact failure, however my bias is that they are related.
it's pretty irresponsible and downright dumb.
http://3parug.com/viewtopic.php?f=18&t=2309
http://3parug.com/viewtopic.php?f=18&t=1975&p=9092
Let me put this to you: If a "firmware update" involves copy pasting a several KB bash script into a putty session, how surprised are you to see software bugs?
[1] http://www8.hp.com/us/en/hp-news/press-release.html?id=94895...
[2]http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr...
Edit: A newer press release (Nov/2016), showing the HPE 3PAR deployment. If the comments below are correct, and it was 3PAR that failed, they sure weren't in for long. http://www.crn.com.au/news/ato-ripping-out-emc-storage-and-m...
"The new solution introduces many improvements, including faster processing, reduced costs and significant simplification to the storage environment."
Ouch.
HP are apparently calling it the first unit in the world to fail, but apparently one in Kings College London kicked the bucket too. [3]
[1] http://www.itnews.com.au/news/hpe-storage-crash-killed-ato-o... [2] https://www.hpe.com/au/en/storage/3par.html [3] http://www.theregister.co.uk/2016/10/19/kcls_strand_data_cen...
Ended up returning to a large keepalived+iscsi+softraid linux storage cluster.
edit: typo
http://www.userfriendly.org/cartoons/archives/97nov/uf17a199...
Sometime, somewhere, someone's had to resort to this recovery method, I am sure.
Right before they went out of business. Years ago (like 15-20) I heard a stat about companies that suffered a catastrophic data loss (lost the drives, no usable backup). I don't recall the exact number, but something like 75% of them were out of business within 12 months.
Tangentially related anecdote - years ago, the smallish company I worked for had one - ONE - person booking our travels.
Seeing as we were basically living out of toolbags at the time, trying to cash in as much as we could on a boom in a market in which we enjoyed a near monopoly, the office was often manned by the travel booker, our IT guy and an accountant.
All good, until the IT guy figures he can use his script-fu to automate configuration of new computers.
He made some mistake in the line having the computer decide whether it was due for a reinstall or not, leading EVERY computer checking in on the intranet to proceed with formatting its /usr partition.
So - travel booker arrives at work, boots computer, loses data. And, as a consequence, has no idea where anyone are anymore, much less any idea when she ought to get us back home or onto the next job.
Anyone checking in via VPN found their laptop bricked in seconds.
She claims it was the most surreal experience of her life, getting concerned phone calls from all over the world from field grunts asking what had happened - oh, and could they please be allowed to go home?
I was one of the lucky ones; I was on Guam with an agreeable climate, sitting on a crate of beer (parting gift from the vessel I'd just disembarked) and in no hurry to get home.
A colleague who was stuck in Nuuk was somewhat less impressed.
By no means a catastrophic data loss, but still...
The founder knew the file layout and could recover the data by manually walking the metadata.
They dumped the product.
Hah...we had a catastrophic 3par failure at a company I worked for.
maybe they'll learn from this.
High end storage doesn't have quite the margins that it used to.
Still, impressive that they can restore that inside of two days (according to their estimates).
But I doubt HP did follow the policies to the letter.
I hope the ATO are incentivised by the loss of revenue to push for high penalties, but I see them paying HP to fix something that they likely could have anticipated: back pressure on the backup machine.
Since the article indicates that the failover system was also affected they may have triggered a bug in the storage systems or it could be something completely different like "someone let the magic smoke out of our bespoke networking equipment and we didn't realize the closest spare was disassembled in HP's network engineering lab in California."
No taxpayer's private data was compromised though, since third parties can not compromise the data because it is corrupted.
And you're correct, their audit programmes all produce shitloads of data, all of which must be retained for legal record keeping (Archives Act).
EDIT: come to think of it, the biggest space eater would be scans of documents sent in by folks who have been audited...