In the wake of the Census debacle that happened this week, there's been a lot of finger-pointing as to who was to blame. Prime Minister Malcolm Turnbull has put the Australian Bureau of Statistics (ABS) and IBM, the company hosting and managing the Census website, on notice, expressing his disappointment over Tuesday's website meltdown. Well, he's going to be even more disappointed today as the Census website went down again last night. It's looking more likely that IBM will be shouldering the majority of the blame for the Census disaster. Read on to find out more.
I've been told to use the word "clusterfuck" sparingly in my writing, but I feel like Census 2016 is definitely the right time to use that term. It has been an absolute clusterfuck. Controversy over its decision to collect and retain name and address information for up to four years and that it wanted to tag Australians with a unique identification number so they can be tracked throughout subsequent Census surveys had been brewing. The ABS just urged everybody to keep calm and carry on.
Then on D-Day, the Census website was supposedly hit with a number of DDoS attacks, resulting in a hardware failure that caused the ABS to pull the plug on the site. The ABS at first described the incident as a "hack" then changed its tune and described it as an "attack" before adding to the confusion by implying that it wasn't an "attack" per se. Regardless of what you want to call it, the website was down for nearly 48 hours.
It was during this time that speculation mounted that the outage wasn't due to DDoS attacks, but simply that the ABS was completely unprepared for the influx of users logging onto the Census website and the servers were overloaded. The ABS still maintains that the shutdown was a result of DDoS attacks, but according to security and IT experts who spoke with Lifehacker Australia, the agency should still have been prepared for these kinds of incidents, especially when it's something so important, like the Census.
A spotlight was put on the technology providers that were paid to ensure the Census website would be up for the job. Revolution IT was paid $469,000 to load test the servers supporting the website. The company has now come out to say it only performance tested the website based on the requirements of IBM and the ABS.
IBM was paid $9.6 million by the ABS for the Census contract. Its responsibility was to host and manage the Census 2016 online website. You'd expect that Big Blue, a company with a long history of working on the IT infrastructure of high profile enterprises and government departments around the world, would get it right.
But if veteran IT security reporter Patrick Gray's sources are correct, IBM may have shot itself in the foot by taking a "she'll be alright" attitude towards DDoS attacks. According to Gray's blogpost on RiskyBusiness, here is how the events unfolded:
- IBM and the ABS were offered DDoS prevention services from their upstream provider, NextGen Networks, and said they didn't need it.
- Their plan was to just ask NextGen to geoblock all traffic outside of Australia in the event of an attack.
- This plan was activated when there was a small-scale attack against the census website.
- Unfortunately another attack hit them from inside Australia. This was a straight up DNS reflection attack with a bit of ICMP thrown in for good measure. It filled up their firewall's state tables. Their solution was to reboot their firewall, which was operating in a pair.
- They hadn't synced the ruleset when they rebooted the firewall so the secondary was essentially operating as a very expensive paperweight. This resulted in a short outage.
- Some time later IBM's monitoring equipment spat out some alerts that were interpreted by the people receiving them as data exfiltration. Already jittery from the DDoS disaster and wonky firewalls, they became convinced they'd been owned and the DDoS attack was a distraction to draw their focus away from the [data exfiltration].
- They pulled the pin and ASD [Australian Signal Directorate] was called in.
- The IBM alerts were false positives incorrectly characterising offshore-bound system information/logs as [data exfiltration].
- ASD still needs to roll incident response before they can send the website live again. Even though it was false positives that triggered the investigation, there still needs to be an investigation.
But the question of culpability still remains and the answer will hinge on whether IBM advised the ABS to refuse the DDoS protection that was put on the table or not.
Lifehacker Australia sought comment from both IBM and NextGen Networks regarding these claims, specifically that IBM and the ABS had refused DDoS protection that was offered. NextGen Networks declined to comment. IBM returned to us with the following statement:
"We genuinely regret the inconvenience that has occurred. We want to thank the ABS, the Australian Signal Directorate and Alastair MacGibbon for their continued support. IBM’s priority over the last two days was to work with the ABS to restore the Census site. We are committed to our role in the delivery of this project. Continuing to maintain the privacy and security of personal information is paramount. The Australian Signals Directorate has confirmed no data was compromised. Our cyber-security experts are partnering with national intelligence agencies to ensure the ongoing integrity of the site."
As IBM states, it is still working on the Census online page. Sure enough, the website was up again late afternoon yesterday, only to go back down again like a yoyo a few hours later. It might be another DDoS attack. Who knows?
For now, we have been assured by the Australian Signals Directorate that the Census DDoS attacks did not result in any data leakage.
It's highly unlikely we will know any time soon whether IBM gave the advice to spurn DDoS protection, but it probably wouldn't matter. When you're paid a large sum of money for an outsourced job, you're most likely going to be the scapegoat when things don't work out; that's the nature of the service business. As an IT professional once told me: "It's common knowledge that you always throw the vendor under the bus to save yourself."
What Happens Next?
IBM's already precarious reputation in the public sector will no doubt take a beating after this Census epic fail. In 2013, the company was sued by the Queensland Government in a high profile case concerning IBM's failed delivery of the Queensland Health payroll system. The contract was worth $6 million. The case dragged on for years and finally ended in April when the Supreme Court in Brisbane formally dismissed the case. It wasn't entirely IBM's fault that the payroll system failed -- public servants who screwed up the management of the project were partly responsible -- but the company's name was dragged through the mud anyway.
Whether IBM will be sued again over the failings of the Census website is unclear. Publicly, IBM will probably be burnt at the stake. Prime Minister Turnbull has already said that there were "clearly very big issues for IBM -- the provider of the systems -- and the ABS itself"; note that he mentioned IBM first. Also, to put things in perspective, a $9.6 million dollar contract is relatively small fry for IBM given that it has agreements worth hundreds of millions of dollars with a number of organisations around the world. It may not have put the best talent to work on the Census.
But realistically, it'd be difficult to prove that the company was the main culprit for all the Census woes. The ABS ignored the outcry over the Census' privacy concerns and inevitably painted a huge target on its own back for DDoS attacks, which aren't the most difficult attacks to launch. The ABS also experienced budget cuts, which might have meant they were forced to cut corners.
The Community and Public Sector Union has since come out to express its anger over the Federal Government's treatment of the ABS in light of the Census debacle. The union's National Secretary Nadine Flood said in a statement:
"Our members working in the ABS have slugged their guts out for months to make this Census work despite multiple Government decisions that have caused major problems. They know how critical the information collected in the Census is to the nation and they’re absolutely gutted at the damage done to the ABS's reputation and the Census itself. "Staff saw these problems coming a mile off. There are 700 fewer staff at the ABS now than when the last Census was conducted five years ago and as a result staff are suffering under massive workloads. Critical planning time was lost as the Government foolishly considered axing the Census, chopped and changed ministers three times and dilly-dallied for nearly a year in appointing a new chief statistician." "It's shameful that Prime Minister Malcolm Turnbull has said 'heads will roll' at the ABS over the Census while taking no responsibility for the real cause of this debacle, the decisions made by his Government"
Right now there is much speculation as to the root cause of the failure, and who is to blame for the decisions that led to it occurring. The eyes of the public will be watching this space closely.