Government and Public Sector
Helping to build a world-class digital solution to capture critical data online.
Amazon Web Services
Share this article
Where human meets digital. Helping the ABS capture Census data critical to informing decisions that improve the lives of Australians.
The Australian Bureau of Statistics (ABS), PwC Australia (PwC), and Amazon Web Services (AWS) came together to deliver a successful 2021 Census Digital Service (CDS).
Trust has never been more critical, or easier to lose, in our increasingly digital world. Nowhere is this more evident than in the relationship between a government and its citizens. For the 2021 Census, the ability to deliver an experience with security, accessibility, simplicity and ease of use at its core was essential to its success. The insights gained from the Census help inform decisions for the essential services of today for Australians, while helping to shape the Australia of tomorrow.
As Australia’s national statistical agency, the ABS plays a vital role in collecting and making available data to the benefit of society by informing some of the most important decisions impacting the lives of all Australians. The statistics provided by the ABS tell the nation’s story — and the most important input is the Census.
Underpinning these success factors was the evolution of the relationship “from one of a more traditional, supplier or vendor type relationship to where all parties were aligned and collaborated towards a common goal.”
A powerful combination of human-led, tech-powered.
For the 2021 Australian Census, the ABS worked with PwC Australia and AWS to build a world-class digital solution to capture the data completed online. It was essential that this solution had robust cyber security to keep our nation’s data confidential and secure, whilst maintaining high levels of availability and performance for all participants. The ABS brought its strong focus on designing and delivering a contemporary level of usability and accessibility across a range of devices and browsers.
The Census - more than a single night.
A national exercise to record data relevant to Australia’s population, the Census is conducted every five years, counting every household and person in Australia. The Census captures a point in time snapshot of Australia.
The ABS delivered three key success factors for the CDS. The first was ensuring the smooth running of the operation, so that everyone had an experience with the CDS that was easy, secure and simple. The second was ensuring it had the confidence of government, business and the community given the sheer scale of participation required. The third was that the data delivered would have to be of the highest quality.
Innovative by default.
Given people’s expectations and previous experience, the ABS expected the public would largely complete the 2021 Census online, and sought a cloud-based solution to support an online completion target of above 75%, which represented over 18 million people. To support this, PwC proposed to build a solution leveraging the latest innovative cloud capabilities of AWS. PwC’s proposed solution was to maximise the use of cloud services “to design, implement, test and operate a solution to fulfil a demanding set of security and performance requirements, using the scale and resilience of AWS's cloud native services,” says Scott.
Scalability, performance, and resilience.
When the ABS came to market in August 2018, it was looking for a knowledgeable and experienced organisation, and a solution that would be easy for Australians to use and allow it to mitigate the risk of cyber attack. This was especially important given the significance of the Census, the sensitivity of the data captured, and the security incidents that disrupted the previous Census in 2016.
Focused on the technology solution, “a key focus of the 2021 Census was to ensure the digital service had redundancy, performance and protection against cyber threats,” says Gwil Davies, Partner, Digital Innovation and Cloud Engineering at PwC Australia. As the solution Lead, Gwil focused on the architecture, engineering and operationalisation in readiness for the Census. The end-to-end solution architected and developed by the team would be custom built using a variety of technologies to address these specific needs and would leverage 55 services from AWS.
"Trust doesn't happen by accident— it is earned and developed through collaboration, respect for diverse views and inputs and brought to life through commitments to a joint and shared outcome. All parties committed to working together to support the ABS."
A community of over 140 solvers.
Building trust at the intersection of human and digital.
Robert Di Pietro, Partner, PwC Australia Cybersecurity & Digital Trust, was the Cyber Lead for the Census project. “From tender to delivery, we knew that cyber security would have to be embedded in the solution and throughout all phases of project delivery,” says Robert.
Given the profile of the project, and the accompanying risk of cyber-attack, the ABS embedded experts from the relevant Government cyber agencies.
“It was incredibly important to see a strong working relationship with the ABS develop,” says Robert. "Trust doesn't happen by accident— it is earned and developed through collaboration, respect for diverse views and inputs and brought to life through commitments to a joint and shared outcome. All parties committed to a strong spirit of working together to support the ABS."
The PwC Cyber team also ran regular security awareness sessions and cyber threat briefings for the broader project team. Cyber became everyone’s responsibility on the project, and not the role of one team to get right.
“Success from a security perspective is based on having people who are not only technically smart, but understand people and human behaviour,” says Robert. “That meant knowing how to communicate potential security risks and issues, as well as recommending mitigations, while still balancing other priorities such as user experience and performance” he says.
“The Census is the largest statistical collection undertaken by the Government,” says Robert. “This information must not be lost or stolen, or be interrupted, and it has to be accurate, given the decisions the Census informs.”
A sustained focus on security.
In solving this important challenge, the PwC cyber team was responsible for managing security operations to defend the Census from cyber attack. “This wasn’t just design and architecture, but the front line of operations and cyber defence,” says Robert. Given the nature of the solution it was a matter of “when, not if, it would be targeted”. The PwC cyber team were responsible for leading 24/7 security incident response efforts and working with multiple stakeholders - including the ABS, AWS and government cyber agencies.
Robert explains that the uniqueness of the Census is that not only is it a significant exercise in collecting data, but also a time-sensitive activity that cyber attackers could be drawn to. Attackers knew not only what their target was, but when they should strike to maximise disruption, particularly for Distributed Denial-of-Service (DDoS) attacks which were a key concern given the challenges encountered by the 2016 Census. “This posed a significant cyber challenge that the team had to defend against from day one,” he says. This combination of the availability of the system, the confidentiality of the data, and the importance of its accuracy meant the platform had to be highly secure and resilient. Often in cyber security, one of these three factors would be prioritised over the other, but in the case of the Census, all were of equal importance.
The team had a year working together before the pandemic hit. “Working hard through the tender process, we were clear on our preferred way of working, co-located with agile hubs and delivery sessions,” says Scott. “When the pandemic hit, and lockdowns were introduced, the team had to pivot to a working-from-home model. The upfront investment in security meant we handled the transition smoothly,” says Scott, speaking to the flexibility built in from the start.
The team maintained its velocity, even with the pivot. “With a large multi-disciplinary delivery team, we’re proud of the resilience that they all showed to pivot and operate remotely, pretty much overnight,” says Gwil.
“With our investments in security, tools, capabilities and risk management, the project barely missed a beat during the pandemic. We were able to adapt quickly to keep the show on the road, and the foot on the pedal.”
The Census is held every five years, which helps reinforce the importance of keeping to schedule, and the pressure that comes with it. As an example of the scale of the endeavour, the ABS were recruiting 35,000 people trained in COVID-safe protocols to knock on doors, while the digital solution was being built.
An important milestone.
In building, earning, and sharing trust the project reached an important milestone when it went live with a Census Test in October 2020. “We had a number of simulation events, and rehearsed cyber incidents to test processes and ensure we had the muscle memory developed for the main event,” says Scott. “This was in addition to multiple rounds of security code reviews as well as penetration tests designed to emulate the tactics of a real attacker.”
Running for two months, 100,000 households were asked to participate in the Census Test. “This test provided an important proving ground for the project, embedding trust in the relationship,” says Scott.
At the same time, there was still a significant way to go before the launch day. However, the success of these tests meant that by the time Census night came along, all parties – PwC, AWS, ABS, government cyber agencies, and other important third parties – knew how they’d respond to an incident.
“We hope that this success has gone a long way to help build confidence in both cloud and IT delivery, for the Government and citizens alike.”
Succeeding on Census night.
This year the ABS made it clearer to the public that they could respond over a period of time rather than focusing on Census night as was the case in 2016. This allowed for households and individuals to submit their responses ahead of Census night.
By Census night, confidence and trust in the solution was strong. With over three million Census submissions before August 10 2021, the team had been proactive in operations and mitigated risks as they emerged, and on the night everything went smoothly and as planned.
"We were confident going into Census night given the extensive preparations and hard work by everyone involved, and were delighted with the outcome for the ABS," says Gwil. “We had a number of operational dashboards, to monitor the service and watch the level of submissions. On the night, volumes peaked at over 270 logins per second, and 142 form submissions per second, and it was great to see a total of 2.8 million forms submitted on Census day.”
The system remained live after Census night until the end of September, and continued to receive thousands of forms per day throughout the period. The team stood side-by-side throughout the event with the ABS and other providers involved.
A resilient outcome.
“When it came to the solution, we did everything we said we would,” says Scott. “As a cloud native solution, available to anyone online, we built it for maximum security from the start.” The proof of this is substantial, the solution was successful in blocking around 130,000 malicious IP addresses on the system across the lifetime of the CDS. “We built a resilient solution, capable of withstanding attacks and with no interruption to service,” says Robert.
“The investments made into security, mitigated and effectively stopped in their tracks anyone attempting to find a way in. It did not mean they didn’t try, they did, but the solution was able to withstand these attempts,” says Scott.
The relationship between human and digital has evolved significantly over recent years, and people expect more from technology than ever before. “We hope that this success has gone a long way to help build confidence in both cloud and IT delivery, for the Government and people alike,” says Gwil.
A new benchmark for digital delivery in Government.
While the ABS do much more than the Census, it’s the most publicly visible of their activities. “A successful Census not only meant a smooth, resilient, and available digital solution for the ABS, but one that people could trust,” says Scott.
Given the target user base comprises the whole of Australia, the ABS brought a keen focus on ensuring usability and accessibility of the CDS. The CDS was able to operate on 95% market share of physical devices, browsers and operating systems. The CDS has been heralded as a "world-class" service in support of accessibility for deaf and vision-impaired people, through its expansive application of Web Content Accessibility Guidelines (WCAG 2.0 AA). In terms of building trust and delivering sustained outcomes, the project successfully used AWS for critically sensitive Government data, at a PROTECTED level of Government classification. The successful delivery of this project provides a valuable example of what can be done with the use of innovative digital technology. “This sets a blueprint for public-facing digital services going forward,” says Scott. “The Government can embrace cloud for critical and sensitive workloads, and realise the benefits it can bring. “It’s not only scalable and resilient, but also more secure and as a result builds trust.”
Recognising that Australia now has a leading example of how the Government has worked across agencies with PwC to deliver a successful digital outcome, the team view this as “a standout achievement”. The project sets the standard, as a global reference that the Government can be proud of on a global scale.
Solving through The New Equation.
Instilling trust in the Government’s digital abilities, as well as taking care of the data of people, the scale of the project represents significant social impact, instilling confidence in the digital services of Government, as well as data and the security behind it. In addition, significant energy was invested by the ABS and PwC working closely together to develop a highly accessible and engaging user experience, with end user feedback being overwhelmingly positive.
The approach of multi-disciplinary end-to-end delivery is one that has been embraced within PwC’s Digital Innovation and Cloud Engineering team more broadly. “We brought hands-on solution delivery, software engineering, end-to-end cyber, platform engineering and operations,” says Gwil. Gwil continues, a culture of collaboration is in our DNA at PwC and is a key element of successful digital delivery at pace and at scale.
“The golden thread of PwC’s The New Equation is about bringing communities of problem solvers together, with a new level of working together and deep specialist disciplines, that brings the best of PwC and other technology partners, like AWS, together,” says Gwil.
Where human meets digital, we solved it as a community, for the community, and that's something PwC is very proud of delivering.
Chief Information Officer (CIO) for the ABS
Census 2021 by the numbers
Innovation with cloud native technologies.
Given the hyperscale and highly variable nature of the workload, the architectural choices were key to success. PwC chose to use AWS Lambda as the core computing platform for the cloud native application. Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. In the design phase, PwC’s solution architecture and engineering teams carefully modelled the volumetrics and business inputs to identify potential ‘hot spots’, and took a rigorous approach with validation through automated performance tests, to refine key design principles. These included approaches to optimise execution time including optimised memory allocations to reflect the characteristics of various application components, and techniques to maintain front-end application performance for end users, and system throughput under a highly variable workload, with a combination of Provisioned Concurrency and Application Autoscaling.
The importance of observability.
In modern cloud operations, it is vital that insights on business and technical metrics can be accessed at near real-time. Due to the stringent requirements within the solution for the separation of duties and data confidentiality, the operations engineering team built extensive automation and application logging frameworks. These involved a streamlined approach to log shipping and secure replication, and a variety of techniques to drive insight via analytics into solution performance and availability. In addition, as this solution made extensive use of cloud native services, a tailored approach was developed with a complementary combination of custom solutions and native AWS tools.
Continuous compliance monitoring for additional guardrails.
Security and compliance of the solution was paramount throughout, not only during the live operations but also right from the outset of the development phase. Whilst various vendor tools can cover aspects of these requirements, the specific demands of this project also benefited from an enhanced and extended set of capabilities. Throughout the delivery phase of the project, PwC deployed its Hardened Cloud asset, a continuous compliance checking framework built on serverless technologies. Hardened Cloud implements customised compliance checks for additional guardrails, in addition to those configurable in a native cloud platform. Hardened Cloud helped to make sure developers had the appropriate level of permissions they needed to perform their roles, and helped make sure that the settings of deployed services were compliant with our best practices and regulatory requirements on an ongoing basis, and not just at deployment time. With extensive automation, and integration into the team’s ITSM (IT Service Management) ticketing system, Hardened Cloud also supported the rapid notification and resolution of any issues if they occurred.
Operational rehearsals and readiness.
Operating a large-scale publicly accessible online application can be challenging, and as such, ‘game days’ and simulations are an important part of readiness preparations. Prior to the solution going online to the public, PwC worked with AWS architects and engineers to define scenarios of the most critical potential incidents covering security, reliability, operational excellence, performance, and cost optimisation, following the AWS Well-Architected Framework. The team was then divided into two groups: the Red Team to design the injection of errors to simulate the occurrence of an incident, and the Blue team, who had to use the monitoring and alarming mechanisms developed to detect, analyse what had happened, and then resolve the incident. These simulations, done both within the PwC operations team, and done in conjunction with the downstream ABS operational teams, were an important step in readying the team for potentially known scenarios and to build the troubleshooting muscle memory if required.
Defending at the edge and mitigating DDoS attacks.
The highly publicised challenges of the 2016 Census meant that defending at the edge and mitigating DDoS attacks was a primary area of focus in designing a highly resilient and secure Digital Census for 2021. The solution leveraged the DDoS protection mechanisms offered by AWS to protect against large-scale volumetric DDoS attacks at both application and infrastructure layers. The PwC team undertook extensive configuration and tuning of Web Application Firewall (WAF) rules, which were finely balanced to allow large spikes of legitimate Census traffic whilst also blocking potentially unwanted DDoS traffic. This process included ABS-led large-scale DDoS tests which simulated massive amounts of traffic targeting the CDS and its supporting AWS infrastructure, equivalent to being in the top 1% of DDoS attacks observed by AWS globally. These successful tests were a key step in providing the confidence that the solution would remain available and perform on Census night.