Last night’s Census was a debacle for Australians trying to fill the survey out online. According to the Australian Bureau of Statistics (ABS), the website suffered repeated distributed denial of service (DDoS) attacks. The ABS ended up pulling the plug on it. People were furious that they weren’t able to get online to submit their forms and worried about the fines they would face for not completing the Census. But there are now doubts as to whether yesterday’s Census website fail was really a result of DDoS attacks. Here are the details.
Article was last updated at 12:46pm
The drama never stops with Census 2016. Last night, the online Census form website was hit by four DDoS attacks, according to the ABS. After the fourth attack, the ABS decided to shut down the site to “ensure the integrity of the data”. The ABS still managed to collect two million forms online.[related title=”More Stories on Census 2016″ tag=”census” items=”3″]
The ABS is working with the Australian Signals Directorate to determine the source of attacks but they already know it was launched from overseas. At the time of writing, the Census form website was still down but is expected to be back up this morning.
The ABS took precautions to ensure that the Census website wouldn’t crash as millions of Australians were expected to log on to complete their survey.
However, some security experts have come out to say that there were no big DDoS attacks occurring in Australia last night.
hmmm. nothing unusual DDoS wise for australia and yesterday #censusfail pic.twitter.com/x7rQ0jzI1F
— Matthew Hackling (@mhackling) August 9, 2016
This is the DDOS for yesterday (site is US-based hence date). Brazil obviously, usual Asia/Europe/US. pic.twitter.com/VgOgF7VEBM
— Gordy irl (@GordyPls) August 9, 2016
While the ABS has come out to reassure the public that cyberattacks didn’t compromise the data it held, DDos attacks are often used as diversions for data theft attempts. It is possible the hackers were using the DDoS attacks as a red herring for something more sinister.
In light of what happened last night, Australian Privacy Commissioner Timothy Pilgrim is launching an investigation on the ABS in regards to the cyberattacks. He said:
“My first priority is to ensure that no personal information has been compromised as a result of these attacks. ABS have confirmed that a decision was taken last night to shut down the website in order to protect personal data.
Yesterday I noted that the Office of the Australian Information Commissioner has been briefed by the ABS on the privacy protections put in place for the Census. My office will continue to work with the ABS to ensure they are taking appropriate steps to protect the personal information collected through the Census.”
On the other hand… maybe the ABS just weren’t adequately prepared for the influx of traffic on the Census 2016 website but are too embarrassed to admit it.
Greens Senator Scott Ludlam has demanded that the ABS and the Government “[p]rovide the community with sufficient detail about the systems failure on census night so that claims of a denial of service attack can be independently verified.”
The Government has since responded by saying that it was actually a hardware failure caused by one of the DDoS attacks that led to the ABS shutting down the Census website. Small Business Minister Michael McCormack said at a press conference this morning:
“A router became overloaded. After this, what is known as a false positive occurred. This is essentially a false alarm in some of the system monitoring information. As a result, the ABS employed a cautious strategy which was to shut down the online Census form to ensure the integrity of the data already submitted was protected.”
QUT privacy expert Professor Matthew Rimmer told News.com.au:
“I’m sure the ABS would like to externalise the blame and say it’s not us it’s someone else, but ultimately given the claims that they were making about absolute privacy and security they do have to take responsibility for privacy and security for their method of delivery…
“I really question the wisdom of the claims by the ABS and the government that everything would be OK, that there would be absolute protection in relation to privacy and security when obviously they were painting a bullseye on their back making those sorts of claims. It underlines we need better privacy protection in an age of big data, cloud computing and hacking.”
The ABS has said that people have until September 23 to fill out the online Census so nobody will be fined for yesterday’s website fail.
Did you manage to complete the Census online last night? Let us know in the comments.
Comments
23 responses to “Census 2016 Website Crash: DDoS Attack, Incompetence Or Something More Sinister?”
Two possibilities; the census website was attacked by unidentified “hackers” groups using a DDOS attack undetected by any external security monitoring …
… or the ABS and IBM screwed up.
I know the one I’m inclined to believe.
I also know that the website in the screenshot can’t actually work. That requires infrastructure in every country, in a thousand different locations just to be able to operate effectively. Just pinging a service doesn’t mean much when you are only hitting a load balancer.
That website makes no sense in how it operates with my only feasible option being that it’s monitoring social networks for reports of DDoS attacks and not actually monitoring traffic because it can’t legally or even at all because of the infrastructure for it.
Love people stressing about the fact the census data could be potentially breached.
Umm , you fill out your tax every year right with the same information and more. Does the tax office servers offer better security. Does your accountants IT infrastructure give you a much safer level of security than the census. Please.
Better yet does your instagram , twitter and Facebook feeds hide details about you. What about that online purchase you made with the small shop in Penrith , your details secure including credit card information.
Talk about overkill on the paranoia for things you are vastly exposed to already.
You’re overlooking a key point about this Census: It is not an anonymous national snapshot, or the kind of information you might provide Facebook, Twitter and the like. It is a tool for detailed data linking on individuals, using name address, date of birth, and other personal details. The information can be used to link health records, criminal records, education, tax etc etc. It is the Australia Card, by stealth. That’s what makes people nervous about providing their information.
It may be but these questions are not very personal, sure if they are stolen it would be bad but this info is filled. Anyway what is so bad about having the government have a better way to put things together.
Actually yes, the ATO ranked first of the government agencies tested by the National Audit Office in 2014. They scored top marks. The tests, undertaken by ASD against the standards defined im the Australian Government Information Security Manual (ISM), scored the ABS dead last with a mark so low they barely scraped a pass.
They specifically failed in database security. This is piblic record on the NAO website and is actually quoted by ABS as “conforming to stringent government standards” and being “certified by the ASD”. In reality they we very poor.
The Tax Department offers better security than the ABS. It iss essentially the central bank of the whole entire government… trillions of dollars, it has better security than most banks, and is protected by a lot of financial grade security systems (which in some cases are more secure than military systems), they are dealing with money and with that comes a lot of security.
ABS is a scientific research department, and cause of that, while it has experience in collecting data its record on keeping private information… until this day… has been “We dont collect private information”. When they changed that policy, the question was ask why the change to policy and they just gave us the paraphrased answer “You can trust us”
…. well they failed last night. If you were sceptical before, but said okay I trust the ABS, today your asking yourself why did I trust the ABS.
what makes it worse is that we are still in the dark at 11.15am the next day . Supposed to be kept informed according to the blurb.
Wouldn’t 16 million people attempting to log in at the same time look like a DDOS?
They tested the server for 1 million per hour. There’s 7.8 million households in Australia. If only one person per household logged on then that’s 7.8 million connections. The crash happened around peak hour and in the past they have encouraged people to do the census at the same time while watching a TV show so it wouldn’t surprise me if at least 2 million tried to access the site at the same time.
Also, it cost nearly $500k of taxpayer money to pay a company to test the servers under load. Money well spent.
$54k actually.
https://pbs.twimg.com/media/ConLvQdUIAASRt3.jpg
Which was probably not enough. I don’t believe it was a DDOS. Just 10 million people
all trying to log in at once.
Odd. The papers are reporting it as $469,000
http://www.theaustralian.com.au/national-affairs/census-2016-online-hacking-was-inevitable-say-it-experts/news-story/fc5cb8630c0a7dda93601c2fb0da2cb3
They had multiple contracts like that – I saw screenshots on Twitter yesterday of others as well, though I didn’t bother trying to add them up
It was something else…
Background: I’m a Solution/Enterprise Architect and Consultant, and I didn’t see anything last night that indicated a DDoS either.
I feel it is disingenuous to claim a DDoS attack, and should be backed up by proof – Namely provide the Web server logs, aggregated by the number of requests to each prefix of the IP address so as to protect privacy – Say a Class B subnet). I’m sure most connections came from within Australia. i.e. The Australian population trying to access the site.
I think the failures were the following:
(It’s a bit technical, but should be understood by most network/web developers)
1. Insufficient DNS Server capacity
The DNS server that serves abs.gov.au is a SINGLE server! (do nslookup for SOA=abs.gov.au records) census.abs.gov.au has 4 servers sharing the load.
They remembered to scale the census.abs.gov.au DNS servers, but NOT the up line servers (which would typically be going to abs.gov.au before doing the census? No One! – so that means the entire load of some 8 Million connections will be resolving the census.abs.gov.au domain via the abs.gov.au server (DNS operates like a hierarchical lookup), and you’ll get DNS timeouts (which is exactly what was seen)
This is an oversight on their scaling, and probably should have been spotted in the design phase. The solution/enterprise architect wasn’t on the ball.
2. Insufficient CDN Capacity
The Content Distribution Network (CDN) that hosts common files cannot serve the web page JavaScript and CSS files quick enough. the CDN seems to be coming from “softlayer.net”. Whilst I cannot be sure, I suspect that when the servers were load tested, the load testing that was claimed was probably run from a smaller number of load-test servers. DNS lookups would certainly be cached by those small number of servers (say 50 Servers) and it is indeed possible that the CDN content was also cached on the load-test servers.
If this were the case, then the load test would represent (say 50 users getting the shared CDN content) and a small number of DNS lookups (say 50 users again).
Even if there were hundreds of load-test servers, then this still is NOT representative of a real world load test, where there are 8Million retrievals. The load testing would have tested effectively only the final submission performance, and inter-page performance (being a JSP app).
Even at the time I am writing this, the CDN is still failing.
3. Underestimating number of simultaneous users.
There was not enough capacity. As also though by gunnermcdagget above, the capacity for 1Million simultaneous requests is far lower than it should be. It was claimed that this was (from memory) twice the expected capacity. Lets look at this claim in more detail.
Census has always been “Census night” in the past, so it would be safe to assume that most will do the Census when they get home. Lets be generous and say 25% of the 8 Million households (7.6 last Census, found a projected number that says 9+Million for 2016), that means 6Million to lodge between say 6pm and 12pm on Census night. That means that for each hour, we are already 2x their expected capacity for the whole 6 hour period. Of coarse populations don’t behave that way, the follow a normal distribution! So there will be a peak in the middle and lower access as the sides, and it is safe to assume that the peak is much higher than the average.
We can go into this further, specifically using an Erlang calculator. I haven’t done that simply because it’s obvious that the peak capacity is much higher than even the claimed maximum expected of 500,000 users per hour.
There appears to be a failure in basic capacity planning here.
4. Just implemented wrong for such a high capacity parallel usage scenario.
The Census application appeared to implement be implemented in a server side technology known as Java Server Pages (JSP). Whilst the issue I see isn’t related to it being JSP (or any other server side processing technology), the choice of server side processing IN GENERAL is a poor choice here.
If the application needed to connect to backend data stores throughout the completion of the application, fair enough. The Census is an online form. As such, the entire Census process could have been performed inside a modern browser (which was a requirement anyway) and stored the data locally inside the LocalDB that is provided by the browser. When submitting the final data was needed, then data could be sent ONCE to the set of target servers. A set of target servers are the end of the process is FAR simpler than having pools of complex App servers running for each request.
If the CDN was leveraged better, and more client side technology was utilised, then almost the entire Census form could live inside the persons browser, and pretty much everyone would have a good experience last night.
If the load testing had addressed the CDN failure by having sufficient capacity (which I think was probably preventable), then the whole app could basically be served from CDN and everyone would get their Census loaded in their browser. If would be fault tolerant (as the data could be resubmitted) and sent securely over HTTPS/TLS. All that is required here is that the Client Side Census form can send the data once to a sufficiently large enough server.
By doing the application this way, then the Census processing on the sever side would be drastically reduced (say 1/60th of the requests…assuming 1 request per question under server side processing).
There were much better ways to build this app so that it scaled to the required capacity.
In summary, I think there were a number of failures. They could have been preventable, and even if it does turn out there was some sort of DDoS, you can still address this even if solution was build in a more scalable way. Until there is proof provided of a DDoS attack, then I don’t see any reason to believe that there was one, as I can see plenty of reasons for the failure with some quick checking, and observations.
1. The errors people were experiencing were not DNS errors. Having only one DNS server is probably not prudent, but I would be surprised if it were a significant factor – especially when most ISP’s DNS servers would have cached the response almost immediately. The TTL for the
census.abs.gov.aurecord was also about 3.5 hours, so it isn’t like the caches would have being refreshed every 10 seconds.2. Requests to resources on the CDN are indeed returning 504s. It is _possible_ that the CDN configuration is a request-and-cache, in which case this response could occur if the servers from which the CDN retrieves the original files has been taken offline. However, you may be quite correct.
3. I think this is very likely
4. I believe (based on a friend for whom the census submission failed, and who had his devtools open) that the form was essentially static and the complete content was sent to the server as a single chunk of JSON. On that basis, the actual per-page load on the server _should_ have been trivial – especially if they did the sensible thing and inlined all the content into individual JSP files (rather than trying to retrieve it from a DB in realtime or something equally dumb for such a high-volume site). Really, the only thing they actually would have needed to retrieve from a data store on the server at all was the unique code everyone had. Even then, with decent design that wouldn’t all need to be stored in a single place – you could distribute users to one of a set of data stores based on (for example) the last three digits of the code. That way any one data store only has to handle a relatively small subset of data.
Unless they did some profoundly silly things, a DDoS is the only _reasonable_ reason for the census site to go down. That isn’t to say it was a DDoS; having seen several proposals by IBM, I find it just as plausible they just didn’t do a great job.
100% agree with 1-3 but 4 is debatable. There are many pros and cons for client side vs server side rendering as posted in this
http://openmymind.net/2012/5/30/Client-Side-vs-Server-Side-Rendering/
Even Twitter have adopted to use server side rendering.
So set 20 / 30 mins aside last night to complete my obligation of filling out my census… boo
site unavailabe… with a classic dumb I.T. error message
OH no its a Code 9 ..
“Thank you for participating in the Census. The system is very busy at the moment. Please wait for 15 minutes before trying again. Your patience and cooperation are appreciated. [code 9]”
thanks ABS for clearing that up…
try again today and same stupid non informative message .. that ol’ code 9 error…..
why didnt they use the classic I.T. error message
I.D. Ten t
(id10t) …
how many times do we have to waste our time trying to do this 3, 5, 10
at what point are we to give up due to lack or service from the ABS (spending our tax dollar..)
I say that enough … u had you chance ABS .. c u again in 5yrs…..
my time is mine and ABS have failed all of us….
as the head of the ABS stated yesterday in a radio interview: “We even pay ethical hackers to try and break the system”
really they didn’t think to stress test it too?
Any noob can purchase booters/servers fairly cheaply. Although with millions trying to get onto the site it wouldn’t of taken much to bring it down…
anyone been on anon IRC lately? there was clearly an #OpCensus running.
The question I have most important one is given the importance the ABS Census is, why on Census Night was the Prime Minister and the Chief Minister for the department… Why were they at home and only reachable by playing phone tag and leaving voice mail messages.
We saw them burn the midnight oil leading up to the election cause every vote counts… but when it comes to Census, close of business lets go home.
Chief Minister, I have only been in this job a couple of weeks, yeah good fact finding mission for you actually be in the building of a once in a 5 year event to see how it works (or fails to work).
It is an interesting situation but at the current time it is hard to tell what this was. It was most likely either an actual DDOS or it could have been a lie to cover up the websites inability to handle the traffic.
A DDOS is possible and it is most likely from Brazil which is indicated by the above graphs. It is especially likely because of the amount of DDOSing lately. However there is also a good chance of this being a cover up to stop the anger at the census and the website.
As said at the start it is to hard to call what this is and i am not sure. However I am leaning towards the belief it was a DDOS because of the graphs showing how busy Brazil has been and with no other website claiming to have been DDOSed it does make it an easier decision.
Filled my online form last week,way before the census night, because I know the server can’t handle this large traffic.