Attack on Australian Census site didn’t register on global DDoS sensors | Hacker News Reader

Attack on Australian Census site didn’t register on global DDoS sensors

cso.com.au

27 pointsRiverTam37 comments

Turnbull is now hanging it on IBM:

https://www.theguardian.com/australia-news/2016/aug/11/censu...

Having worked at IBM on a bunch of terrible Aus Govt projects, I'm so glad they're finally catching some heat.

I've had the displeasure of working with IBM Global Services as a partner in several occasions. In one instance where we had a connectivity issue, it took weeks of discussions before we finally got everyone on a conference call including some of their senior engineers. We were simply trying to make a call to a web service as part of an integration with another company. IBM managed their network. We had supplied our IP addresses and the client had put through a firewall change request to the IBM support staff.

We just couldn't get our requests through. Finally on that last conference call, one of their engineers piped up. "Is this white list rule supposed to be bidirectional? All traffic going back to you is blocked.".

"No shit Sherlock, or how can a HTTP request get a response back", was my answer, or words to that effect in a more polite manner.

"But the change request form only mentioned incoming traffic" said the IBM project manger. I internally screamed.

This kind of thing has now happened several times with IBM on projects I've worked on. I know they charge amazingly large amounts of money for what they do. They appear to have an brilliant sales team, and a bunch of under qualified junior engineers that can't deliver. Over sell and under deliver. Maybe I've just been unlucky, but I don't have a high opinion of them.

>"No shit Sherlock, or how can a HTTP request get a response back", was my answer.

Aren't they configured on the basis of inbound/outbound connections? If they allowed inbound traffic you should still be able to get a response back.

I wasn't privy to their firewall configuration. But that's what their engineer said, made a change, and everything started working. I'm guessing the firewall configuration was non standard.

More likely they forgot or screwed up and tried to put the blame back on you.

God only knows. Whatever it was, my opinion of them remains the same. They're generally incompetent.

Patrick Gray from the Risky Business podcast appears to have the inside scoop on what actually happened. Here's his latest sequence of tweets (link references below):

"The things I'm hearing about #CensusFail are absolutely mind boggling. I'll be on @theprojecttv tonight talking about it.

Geez, ya'know, rebooting your firewall when you haven't sync'd your ruleset to your secondary is kinda dumb, ABS. Also, relying exclusively on geoblocking from your ISP instead of actual, you know, REAL DDoS mitigations is also pretty fail.

As is declining your upstream provider's offer to help with said mitigations to save money.

The funniest part? They detected exfil, thought the DDoS was a distraction. That's when they pulled the pin. Was a false positive.

This also explains why ASD are involved -- they're running incident response on a fucking false positive! This can't get any more hilarious!

.@OaaSvc There was no exfil. It was their own reporting traffic to offshore that tripped the alerts. You can't make it up.

There's more: that "router" that went down for a few minutes? That was likely the firewall reboot. Secondary didn't kick in b/c no rules.

They didn't need to fix it.They needed to clear the state table because they had a whopping 2gbps of ICMP and DNS reflection traffic inbound

Please, oh please, read this sequence of Tweets all at once. Then sit back and let your mind BOGGLE."

https://twitter.com/riskybusiness/status/763581261487091712

http://risky.biz/censusfail // https://twitter.com/riskybusiness/status/763605906047107073

The current running joke is that the DDoS was not detected as all the traffic originated from inside of Australia, primarily via home connections.

So much fail from the Australian government technology wise recently. This is all despite supposedly having a "tech focused Prime Minister".

Ditching FTTP for FTTN while increasing in price and managing to slow down the rollout (copper is the future!). 25 mbit for everyone by 2016! No upgrade path! Actually make that 2020, and we don't care if you only get 25mit for 2 seconds at 3am and otherwise get 1kb that's good enough for us! (for the record, I believe FTTP would have been late and over budget as well, but at least you delivered FTTP's highway to FTTN's goat path)

An innovation push which cut funds to organisation's such as the CSIRO (invented usable WiFi etc...) and failed to mention the afore mentioned rollout at all, presumably because they know it is a waste of taxpayer money.

Followed by a promise that the Census website would not go down, which promptly crashed and burned instantly.

Then the blame game started with claims of a hack, followed by DDOS, followed by faulty hardware, followed by claims of all three. All the while, the Minister in charge, a security expert and the ABS all contradicting each other over what had happened. This for the low low price of 20+ million dollars https://pbs.twimg.com/media/Cph9c4aUkAAyyuZ.jpg:large

In short the technology sector in Australia has been wrecked for the next 20 years along with the reputation of the ABS which was held in high regard by many. But its OK! We have a plan to dig more stuff out of the ground and sell it to China/Vietnam/Indonesia, and we promise next time it will work, especially when we move to online voting!

> Ditching FTTP for FTTN while increasing in price

This claim is very misleading when you consider that FTTP NBN's price was never properly realized. In their last progress report they were over budget and behind schedule despite hiring contractors to work overtime in the months before the election.

When you also consider they already moved the goalposts back a few times, it would be ludicrous to believe that FTTP would have been completed without going well over budget.

Actually I agree with you, on the basis that all large government projects are over budget and late.

However, when you sell an election on 29+ billion, fully costed ready to go, everyone online with 25 mbit by 2016 and then once in power, adjust to 56+ billion by 2020 you should be hit with the same laws that apply to bait and switch advertising.

Updated my original comment to reflect this.

FTTP would have been cheaper long term, they bought millions of dollars worth of old copper from Telstra and Optus which needs replacing and has maintenance cost's far greater than anything fibre.

They also bought Optus's old HFC network (cable) to rebrand it as NBN ( Which Optus was previously sued for branding as NBN tech).

We just need a lot more of those skilled 457 visa IT workers that have been flooding the market because we don't have enough IT people and IT graduates. Oh, wait...

Hey, I'm a 457 worker! What did I do wrong?

They designed the system to support 1,000,000 users per hour over 24 hours.

Of course the census is Tuesday evening so it's more like 20,000,000 users in one hour.

Politicians would argue that this sounds like a good test, but technology doesn't do political word games.

It's not every citizen (~23M) but rather per household (~10M) that would use the system. Not everyone is online or choose the online option so it'd be less than 10M users total but even a conservative back of a napkin estimate would show that in the after dinner hours on census night they'd have to be dealing with closer to 3M to 4M users per hour peaks.

Very badly planned. They even gloated that the 1M/hour peak the system was designed to deal with was twice what they needed and it'd all be fine even when lots of experts were pointing out that that wasn't going to cope.

I think every Australian has two weeks to complete the Census. Why they advertised August 9 as "Census Night" I don't know.

You're supposed to answer with respect to August 9 - it's a snapshot of every person in Australia as of that night.

http://www.abs.gov.au/websitedbs/censushome.nsf/home/getonli...

"Get online on August 9"

Yeah I think since the fiasco they've suddenly been bandying about "Oh you have until August 23rd, there's no rush guy just chill okay" but that was after months of "If you don't fill it in on August 9th we'll fine you $180/day" to try and shut down people calling for boycotts after the changes they shoved through the week before Christmas that allows them to keep the data linked to everyone's name for 4 years.

That was certainly one mistake. How they overlooked the fact that almost everyone was going to get home, eat, and then hit the Census site afterwards, beats me. It was a prevailing theme for everyone I know, sitting there refreshing and then making fun of it all on Twitter.

August 9 is the 'snapshot' night so you respond as from that night no matter when you fill it in.

Benefit of being overseas: didn't have to get caught up in the disaster

It is actually every citizen. I live in two share houses and everyone has to fill it out.

Although I'm away at the moment so didn't get counted!

Shared houses are an edge case, yes if you don't want to share your private data (income, health, etc.) with other members of the share you can do your own census form but for the vast majority of Australia all members of a household go onto a single census entry. In your example you'd only be putting in an entry for the share house you were staying in on the night of the 9th, it doesn't matter one bit that you live in two share houses - the census is just a snapshot of August the 9th.

Slightly off-topic but I've been having an argument with my friend around the security of census data. I wonder if anybody on HN might be able to shed some light...

I made the comment that, potentially, a data breach could result in a flat table containing names, addresses, household incomes, etc being released online. This has obvious potential for easy identity theft, stalking, etc. Mainly due to the existence of name and address (since without these fields an abuser cannot target individuals or individual addresses).

My friend insists that it is impossible for a breach to release all of the census information in a denormalised form, as the names and addresses are "unlinked" from the rest of the data and stored in a separate table in a separate system.

This does not make sense to me. Do the ABS seriously remove the address column from their data? Wouldn't this cripple their ability to do spatial analysis on anything more granular than a suburb level? Security concerns aside, do they seriously pay 300m for data and then throw it away?

I wish the ABS would be a little more clear about exactly what they do with the data...

Not sure how it's done these days, but many moons ago (round 1996 from memory) I worked for ABS on the census side of things.

What your friend says was utterly correct at that point in time.

Further more, you couldn't even dial up a post code and get meaningful data of that precise post code.

The more specific the geographic area the more "representative" the data got. Privacy was taken very seriously.

Yep, that sounds just like what he was describing.

I can see the advantages for legitimate policy development from this new system... But one can only hope they don't let everything get leaked.

My recollection was that https://spacetimeresearch.com/ (ex ABS guys setting up a plum contract) did all the data processing of the raw data, providing a representative but anonymous set to ABS masses for reporting.

The real data was locked away (no electronic access) and I recall destroyed after a certain point of time. There were no names attached to the real data, even though the real data was not accessible by any mortal.

The representative set had the property that if you say queried a statistic at the national level it would be correct, but if you queried that statistic on a narrower area it get more "representative" and less applicable to the precise area queried. The means etc would however all pan out.

The data was accessible at 7 granularities of geographic areas. Some of the areas may have only had 100s of residents, so you can see why the data was fuzzed.

We were co-located in a building with the ATO, but there was a strict protocol and security personnel denying any access by non ABS staff. We had no internet access, no floppy drives, (no USB back then from memory), no CD burners no easy way of getting data in or out.

The pposts above give a general high level view of most ABS publications and data sets that are released to the public/institutions, with some minor deviation depending on context.

Yes, we separated out and destroyed name and address data after processing was finished for the census. Indeed, most stats, data and calculations are released to the public at various levels of statistical aggregation, not at unit record level. Almost all ABS data sets treat privacy and naive identifiability far more seriously than most lay people understand, and we had a variety of techniques to test for such, and ways to change the underlying data to produce good higher level stats while destroying the underlying unit record level/value.

Census is a little bit different: there is a pre-processing/pre-consumable data set stage where various techniques and research is taking place to try to produce various improvements and information from the raw data to produce a finished product and think about its future use. Nothing nefarious when I was doing it, think ocr, error correction and analysis, value distribution, imputation, comparisons to dress rehearsal and calibration. Removal of the string "fuck off census collector" and every swear and curse word and smart arse answer imaginable from every field, as a fun example the public don't think much about...

Security at the time I was working on it was OK. Better than a lot of private industry. There was some network access (this shouldn't be surprising, it's an e census, it's a tautology there's some network access to bits of the data somewhere in the entire process) but ideally not to the wider internet outside of specific bits of the internal ABS network once the raw data is in. Only specific people working with such data are given access at this stage and in this way.

Nothing is perfect, I can't stress that enough in the area of IT security and my particular area of experience and statistical analysis. A mature analyst (so, not tabloid headlines) understands that all security and identifiability is based on risk/reward/work/information trade offs: how much can we change things while keeping the analysis valuable, and pushing the amount of work required to gain access to the information that remains to be greater than the effort required to do so and the likely information/ability an attacker has.

I can't speak for conditions after I left, or how this census has been run, and I believe I've been honest without giving anything important away or painting them in an inaccurate light.

I won't make a comment here about the general question of name address collection, suffice to say that on a personal note, possibly because of my profession/ability, I'm closer to the tin foil hat end of the spectrum than most of the population, and posting this even makes me uncomfortable...

> I'm closer to the tin foil hat end of the spectrum than most of the population, ...

You and me both! I only work with data for machines and reliability, so I can only guess at what nefarious stuff might happen with this data.

But I think it's important that society has people worrying about information security. Many (most?) people just don't care.

My understanding was that that was what they _used_ to do - address information would be used to aggregate data to Statistical Area 1, 2, 3 etc levels (see http://nationalmap.gov.au/#share=s-fg7dcWhKqOxxyqyfbS43YsSKE...) which would then be kept. Starting around 2005 they started keeping address-specific data if you opted in, then from this census onwards they stopped giving you the choice.

http://www.abs.gov.au/websitedbs/censushome.nsf/home/privacy

>then from this census onwards they stopped giving you the choice.

The still allowed spaces as your address though...

Right, that makes more sense now. Great links, thanks.

It's an obvious lie. Par for the course for the totally dysfunctional Coalition government, which hasn't even managed to pass a budget in three years.

As opposed to the previous, completely dysfunctional Labor/Green government, which turned a sizeable surplus into a huge deficit within the space of 5 years?

Should add "Like most other attacks, " in front of the title.

Reply on news.ycombinator.com

​Attack on Australian Census site didn’t register on global DDoS sensors

Attack on Australian Census site didn’t register on global DDoS sensors