BTW, this looks like neat program name, "Bernd is studying Creative Industries and Information Technology".
I'm not saying their overall architecture wasn't better than what the official census had, but the solution is not comparable without knowing all the requirements.
- Scalable to the entire population of Australia (not less than 1 million submissions per hour)
- Significant attempts to combat expected DDoS attacks (the original census site decided not to take the offered help from upstream DDoS protection services)
- Data Security needs to be paramount (seems like the one thing they did right as nothing has been leaked yet)
- Just because it is a government contract doesn't mean the cost is irrelevant. [$9.8m + (testing costs exceeding .5m and .5m of OFFICE plants)]
These students achieved this in a caffeine filled weekend hackathon for around $500 and it achieves at least my first point as they benchmarked it at 4mil req/h. Bantering on about how it isn't an adequate solution just isn't productive here. The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
Credit where it is due guys.
Of course the kids didn't rebuild the ABS census system in 50-some hours, and I don't think the sensationalism in this article is meant to be taken so literally. The purpose of the hackathon project and of the article is more likely to help the average person understand that the ABS could have built this in a fraction of the time, at a fraction of the cost, and given the technically trivial task at hand, it shouldn't have failed.
Sure, they made a nice little prototype -- or MVP, if you will. I'll certainly give them credit for that.
But did they really do it "better and cheaper ... in 24ish hours" when they didn't come anywhere near meeting all of the requirements of the project (most of which probably aren't even public/published)?
That's the trivial part. You seem to imply ABS spent 5 years doing just the web form and that's a false premise just there. Once they match the privacy requirements you can read about here:
http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3...
Let me know how much time it took them (including getting security clearance to do it, all the post-processing, hardware management, etc.)
This exercise is about performance not things like security clearances.
> The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
Neither did ABS spend 5 years on the webform, nor the guys implemented everything that IBM and ABS did in 24h. They did something completely different. These guys got to skip all of the complexity of the solution and did the easy part in a weekend. You're also presenting the easy parts only. Even "add to queue => process later" is easy only if you want to show notification on a small site. Add all the assurances you need for the census and it's a multi-day task on its own.
Basically we're reliving the times of "why is Twitter going down all the time, I implemented Twitter in RoR in a weekend, look"
I have all sympathy for the argument that you could do things better and cheaper than the government and IBM, but while banter isn't productive, neither is a benchmark of a bare hosted web form.
Edit: and as another poster below me pointed out, lower downs in the ABS are well aware of AWS. Inability to get new tools and methods in was one of the reasons I left (also the philosophy that tech pay basically tops out at the APS six level, and they don't want tech experts at EL1 / EL2 levels, they want "managers". Also, the government has been pretty explicit that they don't value or respect the ABS: efficiency dividends, job cuts, left without a head for a long time, moving jobs too Geelong!?!, and low levels of pay for technical experts).
Edit 2: also I just want to stare that I in no way think AWS is necessarily the right platform on which to be conducting a national census.
http://www.itnews.com.au/news/ibm-wins-96m-to-host-ecensus-i...
http://www.abs.gov.au/websitedbs/d3310114.nsf/4a256353001af3...
Karl Stefanovic on HN, really? ~ https://news.ycombinator.com/submitted?id=codeka
But you can extrapolate a lot of the build into a more "Government approved" environment, and you still come in demonstrating that a modern type of environment can be done a lot cheaper than census, which was their goal.
They've said a lot to the average non-tech about what you can achieve.
I'd think I'd have read about that somewhere because it seems like a fairly serious limitation.
https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL
> Microsoft Azure and Microsoft Office 365 were among the first cloud services to achieve this certification for the storage and processing of unclassified (DLM) data. The certification recognizes the successful completion, review, and acceptance of a comprehensive assessment undertaken by an Information Security Registered Assessor. This certification can be leveraged by all Australian and New Zealand government agencies.
> ...certification provides assurance to public sector customers in government and their partners that Microsoft has appropriate and effective security controls in place for the processing, storage, and transmission of unclassified sensitive data (the majority of government, healthcare, and education data in Australia).
It's not a yes/no question. It depends on the information: https://aws.amazon.com/compliance/
But as far as census goes, no. I don't expect any country to be stupid enough to trust a foreign third-party with census information. Even in the US, AWS has separate zones for federal applications. This includes people who specifically can touch the data involved.
Are two Junior engineers capable of producing an actually scalable, secure, fault tolerant system with adequate consistency, integrity, availability? Probably not. In a couple of days? Almost certainly not.
Could it have been done in less time and for less cost than a government did it for? Almost certainly yes.
A server is a server right?
Problem could be cultural and generational. Some old school types like the idea of a server without redundancies and fire prevention and fault tolerance and high security clearance in a data center. They want it to be blinking lights at them in a cupboard on premises. They feel it is safe due to physical proximity, even though their firewalls are like cottage cheese.
We're noticing that issue with health data for medical practises. Older people are suspicious of cloud. The younger generation are more savvy.
An Australian census is never going to take place in hardware controlled by a foreign national on foreign soil, especially not in a country where cloud servers have been seized in the past for simply having the possibility of containing data associated with a person associated with a crime.
One thing you will learn as you gain real world experience is that there is no such thing as "too paranoid" when it comes to IT projects handling sensitive data.
Look at the actual rules and what risks they want to protect from: http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3...
Here are some of them:
2. an audited linking environment, involving staff activity being logged, monitored and, if
inappropriate activity is found, investigated. Any misuse would result in immediate
termination of access for the staff member, with further sanctions imposed if necessary;
3. ABS staff and in-posted officers sign legally binding Undertakings of Fidelity and Secrecy to
ensure they are aware of their obligation to protect confidential information, and the
consequences of disclosure (which include criminal penalties);
4. enforcement of the clear desks and clear screen policy;
5. access on a ‘need to know’ basis;
7. Vulnerability Assessments are carried out on all new IT Systems by specialised staff in IT
Security trained in the field of Ethical Hacking;
At that point, it's just easier to stuff some more servers in the government datacentre - completely isolated from other projects. And whether it's cloud or not is a completely separate thing - it may as well be on OpenStack.
> Older people are suspicious of cloud. The younger generation are more savvy.
So no, this is complete bullshit. (and so is putting this as us-vs-them - I'm part of a fairly young generation in the CS, working on cloud infrastructure, and I still would not want to see census data processed anywhere outside of "no phones in the building" government environment)