The conduct of the Australian Census 2016 is a worrying demonstration of government incompetence. Here are my observations on the debacle.
Privacy concerns
In the lead up to the Australia's 2016 Census serious concerns were raised relating to a change in the retention of personal data that allows respondents to be identified and what this would be used for.
I didn't see an adequate response from the Australian Government or the Australian Bureau of Statistics about why this was being done and what the personal identification data links would be used for.
I have concerns that government departments and other authorities could access this data and use it to track down issues with individuals, which is not the purpose of census data.
Funds stripped from ABS
The Rudd and Abbott governments both cut funding for the ABS. This set the scene for the debacle that was to follow. The ministers previously involved in this were Kelly O'Dwyer and Joseph Ciobo.
Online data collection outsourced to IBM
In a apparent attempt to save money a decision was made - its not clear to me by which minister - to shift the majority of data collection to an online process using the Internet, and to outsource the technology solution to IBM.
With this outsourcing, it was vital that requirements for the solution be specified by government, including:
Website meltdown and failure
During the evening of Tuesday 9 August 2016 people experienced problems accessing the website. Some people completed forms but were not able to submit them. Others couldn't access the website at all. Messages via Twitter from the ABS were confusing - they said to "try again later". Millions of people gave up trying to submit their response and were left wondering what had happened.
The morning after - claims of hacking
There was intense interest on Wednesday morning on what had happened. Claims emerged on ABC morning radio that
"the website had been hacked"
"no data has been compromised"
"there was a DDOS attack" (no evidence has been provided for this)
"DDOS is not actually an attack because no data was accessed" (by Minister McCormack)
"the website was taken down by the ABS due to a false positive alert from IBM"
"a hardware router failed that prevent people accessing the website"
"the website was tested for up to 1 million concurrent users"
It is not possible to determine the veracity of any of the above claims as no information is available to validate them.
More recently, it has been claimed that:
"access to the website is geo blocked" (you must be in Australia to access the website)
"DNS servers outside Australia were blocked (preventing them routing access requests to the website)
What these claims highlight is rank incompetence by the ABS, the Government and IBM.
Solutions
I work in IT. The following solutions were possible, but apparently neglected.
Robust security: Encrypted sessions to secure data (appears to have been implemented)
DDOS protection: Mechanisms are available to identify and avoid DDOS attacks, which are quite common.
8 million concurrent users: Website performance should have been scaled to meet up to 8 million concurrent users as the majority of the population is on Eastern Standard Time and therefore was trying to access the online form at the same timee.
Drop the extended data retention: In the absence of valid reasons for retaining identification data longer, this should be dropped.
Hardware failure: Redundancy and fail over is required, preferably via virtual devices rather than physical ones.
Increase scalability: If the application has been written properly (it may not have been) and the solution is cloud hosted, then performance can be scaled up (e.g. by instantiating more virtual servers) as required, and scaled down when not required.
Unfortunately, recent comments from Prime Minister Turnbull, Minister McCormack and the ABS only amount to misinformation, blame shifting and finger pointing.
The Australian online census meltdown is a failure of outsourcing and reveals gross incompetence of the Australian government.
Turnbull has said that "heads will roll". Perhaps Turnbull and McCormack are the ones who should be sacked for rank incompetence?
This debacle makes a complete mockery of Turnbull's "innovation agenda".
Misinformation, blame shifting and finger pointing won't fix the problems.
See also
Privacy concerns
In the lead up to the Australia's 2016 Census serious concerns were raised relating to a change in the retention of personal data that allows respondents to be identified and what this would be used for.
I didn't see an adequate response from the Australian Government or the Australian Bureau of Statistics about why this was being done and what the personal identification data links would be used for.
I have concerns that government departments and other authorities could access this data and use it to track down issues with individuals, which is not the purpose of census data.
Funds stripped from ABS
The Rudd and Abbott governments both cut funding for the ABS. This set the scene for the debacle that was to follow. The ministers previously involved in this were Kelly O'Dwyer and Joseph Ciobo.
Online data collection outsourced to IBM
In a apparent attempt to save money a decision was made - its not clear to me by which minister - to shift the majority of data collection to an online process using the Internet, and to outsource the technology solution to IBM.
With this outsourcing, it was vital that requirements for the solution be specified by government, including:
- Appropriate security for online submission of forms
- Appropriate security for data retained
- Performance - the maximum number of concurrent users supported
- Website availability - protection for Distributed Denial Of Service (DDOS) attacks
Website meltdown and failure
During the evening of Tuesday 9 August 2016 people experienced problems accessing the website. Some people completed forms but were not able to submit them. Others couldn't access the website at all. Messages via Twitter from the ABS were confusing - they said to "try again later". Millions of people gave up trying to submit their response and were left wondering what had happened.
The morning after - claims of hacking
There was intense interest on Wednesday morning on what had happened. Claims emerged on ABC morning radio that
"the website had been hacked"
"no data has been compromised"
"there was a DDOS attack" (no evidence has been provided for this)
"DDOS is not actually an attack because no data was accessed" (by Minister McCormack)
"the website was taken down by the ABS due to a false positive alert from IBM"
"a hardware router failed that prevent people accessing the website"
"the website was tested for up to 1 million concurrent users"
It is not possible to determine the veracity of any of the above claims as no information is available to validate them.
More recently, it has been claimed that:
"access to the website is geo blocked" (you must be in Australia to access the website)
"DNS servers outside Australia were blocked (preventing them routing access requests to the website)
What these claims highlight is rank incompetence by the ABS, the Government and IBM.
Solutions
I work in IT. The following solutions were possible, but apparently neglected.
Robust security: Encrypted sessions to secure data (appears to have been implemented)
DDOS protection: Mechanisms are available to identify and avoid DDOS attacks, which are quite common.
8 million concurrent users: Website performance should have been scaled to meet up to 8 million concurrent users as the majority of the population is on Eastern Standard Time and therefore was trying to access the online form at the same timee.
Drop the extended data retention: In the absence of valid reasons for retaining identification data longer, this should be dropped.
Hardware failure: Redundancy and fail over is required, preferably via virtual devices rather than physical ones.
Increase scalability: If the application has been written properly (it may not have been) and the solution is cloud hosted, then performance can be scaled up (e.g. by instantiating more virtual servers) as required, and scaled down when not required.
Unfortunately, recent comments from Prime Minister Turnbull, Minister McCormack and the ABS only amount to misinformation, blame shifting and finger pointing.
The Australian online census meltdown is a failure of outsourcing and reveals gross incompetence of the Australian government.
Turnbull has said that "heads will roll". Perhaps Turnbull and McCormack are the ones who should be sacked for rank incompetence?
This debacle makes a complete mockery of Turnbull's "innovation agenda".
Misinformation, blame shifting and finger pointing won't fix the problems.
See also
- IBM And The ABS Census: Let The Blame Games Begin | Lifehacker Australia
- Lesson of #CensusFail: continued funding cuts mean agencies can’t do their job | The Guardian
- Census debacle puts the Prime Minister on notice | Sydney Morning Herald
No comments:
Post a Comment