Australian Census 2016 - rank incompetence, lies and a failure of outsourcing

The conduct of the Australian Census 2016 is a worrying demonstration of government incompetence.  Here are my observations on the debacle.

Privacy concerns

In the lead up to the Australia's 2016 Census serious concerns were raised relating to a change in the retention of personal data that allows respondents to be identified and what this would be used for.

I didn't see an adequate response from the Australian Government or the Australian Bureau of Statistics about why this was being done and what the personal identification data links would be used for.

I have concerns that government departments and other authorities could access this data and use it to track down issues with individuals, which is not the purpose of census data.


Funds stripped from ABS
The Rudd and Abbott governments both cut funding for the ABS.  This set the scene for the debacle that was to follow.  The ministers previously involved in this were Kelly O'Dwyer and Joseph Ciobo.


Online data collection outsourced to IBM
In a apparent attempt to save money a decision was made - its not clear to me by which minister - to shift the majority of data collection to an online process using the Internet, and to outsource the technology solution to IBM.

With this outsourcing, it was vital that requirements for the solution be specified by government, including:

• Appropriate security for online submission of forms
• Appropriate security for data retained
• Performance - the maximum number of concurrent users supported
• Website availability - protection for Distributed Denial Of Service (DDOS) attacks

Statements were made by the ABS and the newly appointed minster Micheal McCormack that everything was in order, data would be secure and the website would handle the number of user that would use it on the night Tuesday 9 August 2016 (data collection night).


Website meltdown and failure
During the evening of Tuesday 9 August 2016 people experienced problems accessing the website.  Some people completed forms but were not able to submit them.  Others couldn't access the website at all.  Messages via Twitter from the ABS were confusing - they said to "try again later".  Millions of people gave up trying to submit their response and were left wondering what had happened.


The morning after - claims of hacking
There was intense interest on Wednesday morning on what had happened. Claims emerged on ABC morning radio that

"the website had been hacked"

"no data has been compromised"

"there was a DDOS attack" (no evidence has been provided for this)

"DDOS is not actually an attack because no data was accessed" (by Minister McCormack)

"the website was taken down by the ABS due to a false positive alert from IBM"

"a hardware router failed that prevent people accessing the website"

"the website was tested for up to 1 million concurrent users"

It is not possible to determine the veracity of any of the above claims as no information is available to validate them.

More recently, it has been claimed that:

"access to the website is geo blocked" (you must be in Australia to access the website)

"DNS servers outside Australia were blocked (preventing them routing access requests to the website)

What these claims highlight is rank incompetence by the ABS, the Government and IBM.


Solutions
I work in IT.  The following solutions were possible, but apparently neglected.

Robust security:  Encrypted sessions to secure data (appears to have been implemented)

DDOS protection:  Mechanisms are available to identify and avoid DDOS attacks, which are quite common.

8 million concurrent users:  Website performance should have been scaled to meet up to 8 million concurrent users as the majority of the population is on Eastern Standard Time and therefore was trying to access the online form at the same timee.

Drop the extended data retention:  In the absence of valid reasons for retaining identification data longer, this should be dropped.

Hardware failure: Redundancy and fail over is required, preferably via virtual devices rather than physical ones.

Increase scalability: If the application has been written properly (it may not have been) and the solution is cloud hosted, then performance can be scaled up (e.g. by instantiating more virtual servers) as required, and scaled down when not required.

Unfortunately, recent comments from Prime Minister Turnbull, Minister McCormack and the ABS only amount to misinformation, blame shifting and finger pointing.

The Australian online census meltdown is a failure of outsourcing and reveals gross incompetence of the Australian government.

Turnbull has said that "heads will roll". Perhaps Turnbull and McCormack are the ones who should be sacked for rank incompetence?

This debacle makes a complete mockery of Turnbull's "innovation agenda".

Misinformation, blame shifting and finger pointing won't fix the problems.

See also

• IBM And The ABS Census: Let The Blame Games Begin | Lifehacker Australia
• Lesson of #CensusFail: continued funding cuts mean agencies can’t do their job | The Guardian 
• Census debacle puts the Prime Minister on notice | Sydney Morning Herald 

Tasmanian fires and damage to natural environment - enquiry is needed

To: The Hon. Greg Hunt, Federal Minister for the Environment Greg.Hunt.MP@aph.gov.au

The Hon Will Hodgman, Premier of Tasmania  will.hodgman@parliament.tas.gov.au

I note that areas in excess of 95,000 hectares have been burnt recently by out of control bushfires in Tasmania.  Many of these areas have very significant native vegetation and animals that have been destroyed.

Hectares of burnt cushion plant. Photo: Rob Blakers

I have previously visited and bushwalked in the Tarkine wilderness, the Central Plateau, The Overland Track, the South Coast and South West Cape tracks and the Denison and King William Ranges regions.

Could you please initiate an enquiry to determine:

1. Whether the Tasmanian Fire Service has sufficient resources to control bushfires in remote areas in a timely fashion in order to limit their spread.

2. Whether the federal government needs to provide resources to enhance air support, training, and resources for remote area fire fighters.

3. Whether current resourcing and fire management practices are sufficient to deal with conditions expected from the extended fire seasons that are now anticipated under climate-change scenarios

4. What restoration programs will be required after the 2016 fires, and what resources will be needed to ensure the best possible restoration of fire affected vegetation, especially high altitude mountain environments and cool-temperate rainforests.

Yours faithfully,

Peter Campbell

Links

• ‘Like losing the thylacine’: Fire burns Tasmanian Wilderness World Heritage Area, Sydney Morning Herald
• Tasmanian bushfires: Damage to Wilderness World Heritage Area to be assessed by specialists - ABC News (Australian Broadcasting Corporation) 
• Talking Point: Wilderness fires ignite need for better protection, The Mercury