The NZ Transport Agency avoided a $70 million rebuild of its driver and vehicle registration system via a carefully-planned migration that moved ageing mainframe apps onto a modern Windows server environment. Its experience provides 10 useful lessons in how to manage large-scale app migrations while minimising risks and costs.
Programme manager Christian Hayes gave a presentation on the 16-month migration project at Gartner Symposium on the Gold Coast this week. The migration of systems for driver registrations, vehicle registrations, road user charges and vehicle inspections was part of a broader business continuity programme, which in turn fitted into a broader five-year IS strategic plan. But that one element was quite complex enough: it involved 3 million licences, 4 million vehicle records and a total of more than one billion database records.
Why switch such a large-scale application? "We had a legacy system end of life risk," Hayes explained. "We had a mainframe platform and it reached end of life in August 2014, and the LINC software running on it ran out in December 2015. We Could have upgraded the mainframe for August 2014, but that would only be a temporary fix [given the December deadline], and there were significant costs for extending the contract in its current form."
Operating cost and inflexibility was also a major concern. "The ongoing maintenance costs for running the two registers were significant and increasing at an alarming rate," Hayes said, noting the annual running costs were north of $12 million. (All figures in this story are in NZ dollars.)
"On the mainframe platform it was very difficult to change the system and it took a long time to make those changes." Rolling out alterations required by legislation typically required 12 months lead time "and often much longer," Hayes said. "That was a real inhibitor for legislative improvements. We wanted the system to become a transformational enabler, and at the time the mainframe was far from that."
NZTA eventually decided to migrate the existing applications (built up from 500 batch programs, 20 COBOL programs, 1.5 million lines of COBOL and 6 million lines of LINC) into a web-based system, replicating closely the existing functional but running in a Windows/Intel environment. "We developed 700 screens to provide our existing applications through a web interface," Hayes said.
Lesson 1: Don't try to do it all at once
Before the main migration, NZTA did a proof-of-concept, which was "fundamental" to future plans, Hayes said. "We took one of the small components of the register and migrated it and stood it up and run it and learnt a considerable amount from that process. It led to having two additional performance test windows."
Lesson 2: Planning for training needs is vital
Training was always going to be a challenge. NZTA uses third parties such as NZ Post and the New Zealand Automobile Association to deliver most services, across a total of up to 3000 sites. "At any one time there would roughly 6500 users of the system," Hayes said. "Our management of the changeover of the old text-based interface to a new web-based one was a key part of the program of work."
"If you look at our text-based interface, we had two options. We could go to a brand new browser interface that looked completely different, but the cost of retraining and the cost of mistakes was too high."
"We first deployed the user interface running on the old mainframe system. We ran the two in parallel, so we had both the old green screen and the web browser running at the same time." That enabled an 18 month migration of agents, rather than requiring concentrated training over a brief period.
Lesson 3: Migration will be a slow process
"In terms of actual data, when we did the migration of motor vehicle registers there were just over a billion records migrated," Hayes said. "The migrations themselves for the drivers licence register took 24 hours to migrate from the old platform into the new platform, and over 38 hours for our motor vehicle register migration."
Lesson 4: Cost justifications will come from multiple areas
"The annual operating cost for the old system was $12 million a year," Hayes explained. The new system cost $8.2 million to deploy, with an annual operating cost of $9.1 million — considerably lower, and ensuring a reasonable period for project payback.
"When we started the modernisation process, the preferred approach was a complete replacement with a potential cost of $70 million. The financial crisis and other IT project failures and cost increases provided a trigger for the business to reconsider the approach. The role that we played was to provide leadership to demonstrate alternatives to a total system replacement."
Other savings occurred because of the migration. A requirement to purchase additional transaction processing capacity around September 2012 was avoided, saving $2.2 million. "We avoided that cost by completing this migration approach."
Lesson 5: Not everything will be documented
"There was no documentation for some code," Hayes explained — unsurprising given its diversity and age. "We had to do reverse engineering in some instances to understand how to redeploy the application in the new environment."
Lesson 6: Despite planning, scope creep always happens
"One of the key principles that we set from the start was that it was a like-for-like migration," Hayes said. "All business rules and processes would be migrated and work in the same way. This was very effective in focusing the thinking of the team and constraining scope. We had to make sure we didn't get distracted by increasing the scope."
That principle didn't entirely eliminate scope creep. A major change to legislation three months changed the way road usage charges were calculated. "We had to do the unthinkable and run a modernisation programme of work alongside with and in parallel to a significant legislative change to our motor vehicle registry," Hayes said.
Lesson 7: Super-tight deadlines aren't helpful
While there was a fixed deadline for when the project needed to be completed, milestones were more fluid. "The schedule was realistic. Rather than set specific planned release dates, we established a window for each of them," Hayes said. "Typically, this was three months."
Lesson 8: Project sponsorship is vital but variable
The NZTA project had executive sponsorship on multiple levels. "We'd established board engagement at the very start and that was maintained with monthly reports." One obvious benefit? "In terms of resources it was a high priority within the organisation and it was resourced to be successful."
"We had a CIO with a clear vision and he took total ownership," Hayes said. "For me it meant I had a clear line of sight and I had clear communication."
That didn't mean there weren't glitches. "During the delivery of the program of work the second major challenge we faced was that there was a significant reorganisation of the business. We lost that key focus and our executive sponsorship was no longer there. That required the IT department to step up and temporarily own the delivery of the programme of work. We really backed the project team to take on more responsibilities outside their normal role and take the program forward."
Lesson 9: Not all your technology plans will work out
"A key element of our initial approach was the use of automated test tools for performance and regression testing," Hates noted. While performance testing worked well, regression testing didn't function effectively and was eventually abandoned. "We spent a lot of time trying to make it work before we reverted to a more traditional approach around iterative and risk based testing. The key element was making the decision to switch in order to make the original deadlines.
Lesson 10: Engage with your users
Looking back, Hayes says more input could have been sourced from the end users. "Most of what we did in that space was just-in-time or when required. A lot of our agents and partners felt that we downplayed the size and complexity of the project. But from all of our stakeholders, their feedback was they never knew how big this was and they felt they could have participated far more if they'd understood the size of the project.
Evolve is a weekly column at Lifehacker looking at trends and technologies IT workers need to know about to stay employed and improve their careers.