Review of Transport Canada’s Cloud Project

Internal audit report outlining results of the assessment of the Review of Transport Canada’s Cloud Project

On this page

Statement of conformance

This Review conforms to the Government of Canada’s Policy on Internal Audit and to the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing, as supported by the results of an external assessment of Internal Audit's Quality Assurance and Improvement Program.

Background

  • The Internal Audit (IA) Risk-Based Audit Plan 2020-2021 included a review of Information Security in the Cloud to provide Transport Canada (TC) with real time assurance on the current state of governance, operations, and security of data as the Department migrates to the cloud computing environment.
  • Due to various delays to the cloud project, IA was unable to review the information security elements of cloud migration, and instead focused its initial efforts on performing a current state assessment of the Cloud project, leveraging Microsoft’s maturity benchmark assessment of TC’s Cloud environment.
  • The Government of Canada (GC) launched a Cloud First strategy in 2016 making the cloud the preferred option for delivering Information Management/ Information Technology (IM/IT) services. TC’s IT Modernization project was launched with the goal of delivering a modern infrastructure environment for TC applications in the Cloud.
  • TC’s cloud project includes:
    • building a cloud foundation (consisting of the technical components needed to be in place prior to migrating applications to the cloud)
    • planning and executing a workload migration project (WLM)
    • implementing a Disaster Recovery (DR) Management Program (currently fully outsourced)
  • A Transport Canada Cloud Centre of Excellence (TC3oE) was set up to deliver the project.
  • Around 370 business applications currently hosted at the Macdonald-Cartier Data Centre were supposed to migrate to the cloud originally by 2023, now pushed back to 2025 – the planned closure of the data center.
  • TC self-funded the cloud project from 2017-2018 to 2019-2020. Treasury Board Secretariat (TBS) started providing cloud funding in 2020-2021 and will fund the cloud project for the next two fiscal years (2021-2022 and 2022-2023).
  • There are various reasons causing a delay of the cloud project:
    • the delayed Shared Services Canada (SSC) delivery of Secure Cloud Enablement & Defence (SCED)
    • resources have been concentrated on applications related to the COVID-19 response as these have been given top priority for migration
    • lack of skilled technical cloud resources and departures of some key skilled technical staff
  • With this delay of the TC Cloud project, the Department has the opportunity to take stock to better understand the key risks/factors that are impacting the successful implementation of the project.
  • IA performed an initial assessment and highlighted the need for TC to determine how best to move the Cloud project forward in the current environment given the unknowns and uncertainties surrounding its partners, skills, costs, and timelines. The assessment highlights opportunities to improve the implementation of the Department’s Cloud project.
    • IA held discussions with multiple stakeholders internal (Digital Services Directorate (DSD), Human Resources (HR), Finance) and external to TC (TBS, Statistics Canada, Shared Service Canada (SSC), Agriculture and Agri-Food Canada, Employment and Social Development Canada, Microsoft) to identify risk factors, best practices, and lessons learned that would benefit TC’s Cloud journey.
  • What follows is an overview of the Cloud Project Stakeholders and the functions critical to successful implementation of the Cloud Project as well as a summary of our findings that describe the current status and opportunities for improvement for each of the following areas:
    • SSC Cloud Operating Model
    • Roles and Responsibilities
    • Recruitment and Staffing Strategy
    • Cloud Project Resource Retention
    • Financial Management of the Project

TC Cloud stakeholders

For a successful cloud journey, internal and external stakeholders must be involved.

This diagram demonstrates the need for both internal and external stakeholders to be involved in TC’s Cloud journey.  The main part of the diagram shows the 4 TC Directorates and branches which report to the Assistant Deputy Minister of Transformation/the Chief Digital Officer and the Chief Information Officer. The TC Cloud Center of Excellence is comprised of 3 technical staff and 2 managers at the time of the review. It falls under the IT Service Management branch which is positioned within the Digital Services Directorate. The TC Application Development team (DevOps) work within the Solution Center which also reports into the Digital Services Directorate. Their role is key to TC’s Cloud journey. The 3 other directorates involved in TC’s Cloud journey are the Transformation and Results Directorate, the Service Innovation Directorate and the Data and Advanced Analytics Directorate. Application Portfolio Management which falls under the Enterprise Architecture branch is also key to the Cloud journey, and reports to the Transformation and Results Directorate. There is also a box on the diagram depicting TC Corporate Services (Human Resources and Finance) which are additional internal stakeholders required for TC’s Cloud journey. Another box depicts the External Stakeholders: Shared Services Canada, the Treasury Board Secretariat, the Communications and Security Establishment and Other Government Departments.Finally, there is a box entitled Partners which include Microsoft and Amazon.

Functions critical to the Cloud Project

To date, some of the Digital Services organizations primarily responsible for the functions listed below have had limited involvement in the Cloud project. These functions are critical to the success of the Cloud project.

At the center of the diagram, there is a box with the Cloud Project depicted along with the various components (Foundation, Work Load Migration and Disaster Recovery). Around the box are various functions critical to the Cloud project. These include: IT Security, Cloud Architecture, Enterprise Governance, DevOps, Strategy and Plan, HR and Finance Engagement and Solution Portfolio.

Cloud Project issues

  • Uncertainty around SSC Cloud Operating Model
  • Roles and responsibilities for Workload Migration lack clarity
  • Lack of an effective recruitment & staffing strategy
  • Challenging to retain resources within the Cloud project
  • Weak financial management of the Cloud project
Cloud Project Issues
Area Current status Opportunities for improvement
SSC Cloud Operating Model (COM)
  • Uncertain timelines, service options and costings for SSC managed service.
  • TC does not have the necessary resources, skills and knowledge to manage the COM project.
  • Leverage cloud information and support from the Direct General level SSC Committee to better understand the COM offerings, timings and updates.
  • Leverage support and advice from the GC Cloud Funding Model Committee to improve TC Cloud funding options for COM.
Roles and Responsibilities
  • Lack of a TC Cloud Strategy to clearly define the roles and responsibilities of all parties.
  • The Cloud WLM project is supposed to be dedicated to migrating applications to the cloud; however, they have assumed all other cloud related responsibilities (application portfolio management, foundation work, network/firewall, security, infrastructure, operations, architecture).
  • Lack of buy-in and governance - Cloud is not a priority even though it underpins all digital transformation and innovation. There are many competing priorities.
  • Incomplete analysis and rationalization of portfolio of applications to be migrated to the cloud.
  • Develop a cloud strategy to clearly define the roles and opportunities of various DSD stakeholders (Enterprise Architecture (EA), Solutions, Security and Programs).
  • Rebuild organizational chart – specify the positions, reporting relationships, roles and responsibilities.
  • Obtain support from senior management to establish the cloud project as a priority.
  • EA and Programs to complete the analysis, rationalization and prioritization of applications moving to the cloud.
Recruitment and Staffing Strategy
  • There is no permanent funding for hiring full-time employees (FTEs). Capital funding is used to hire casual and temporary staff. Consultants are often used for advanced technical work.
  • Hiring DSD resources for Cloud is not prioritized; the 3 other directorates prioritized ahead of cloud.
  • DSD’s HR Strategy, which outlined a need for 15 resourced positions (based on Gartner/Microsoft business case numbers), has not been achieved. Number of current staffed positions: 3 out of 6 technical resources and 2 managers.
  • Flexible recruitment processes for specialist cloud skilled resources (unadvertised positions, unilingual) are not employed.
  • Incorporate lessons learned from other government departments’ and agencies’ (OGDAs) Cloud journeys or how they built/ recruited for their Cloud teams.
  • Formulate a flexible recruitment strategy with HR.
  • If competitive processes are a requirement for certain positions, explore streamlined and flexible processes which could accelerate hiring (in alignment with internal/ external hiring requirements).
Staff Retention
  • Retention of skilled cloud FTEs is difficult without permanent funding. TC Cloud team relies on hiring and training students for a stable team.
  • 4 out of the 6 technical cloud employees in WLM left over the past few months.
  • Staff experienced stress in having to take on advanced cloud roles with limited knowledge or time to learn (“imposter syndrome”).
  • Lack of clear career progression within DSD - and no use of internal non-advertised promotions available to retain skilled staff.
  • Consider adopting a learning plan/ strategy and exploring options for allocating time for employees to learn.
  • Consider creating a Cloud Developmental Program to help junior employees stay and flourish at TC.
  • Conduct exit interviews to understand reasons for leaving to develop a more effective retention strategy.
Financial Management
  • Lack of effective financial planning and tracking – significant surplus before year-end required a lot of recoding and Payables at Year-End (PAYEs).
  • No clear plan of how to address the gap between the cloud project’s financial needs and the funding available.
  • Staff Cloud positions based on financial affordability versus actual needs – i.e. no permanent funding for FTEs.
  • Improve current financial management practices by proactively working with DSD finance and the Financial Management Advisor.
  • Review financial status – what are the priority expenditures (e.g. Amazon Web Services (AWS) work?).
  • Explore Cloud funding mechanics (Treasury Board submissions, output from GC Cloud funding exercise).
  • Determine fundamental resource needs (i.e. staff key cloud positions), allocate sufficient salary funding.

Summary of opportunities for improvement and management action plan

The following summarizes the review opportunities for improvement and management's plan to address them.

# Opportunities for improvement Management action plan Completion date
(for each action)
Office of Primary Interest (OPI)
1

Leverage cloud information and support from the Director General level Shared Services Canada (SSC) Committee to better understand the Cloud Operating Model (COM) offerings, timings and updates.

Leverage cloud information and support from the Director General level Shared Services Canada (SSC) Committee to better understand the Cloud Operating Model (COM) offerings, timings and updates.

Work with the Chief Digital Officer, SSC Client Executive and their respective teams to fully understand the SSC COM offerings, including agreeing on roles and responsibilities, funding models and timelines. September 2021 Digital Services
2 Develop a cloud strategy to clearly define the roles and opportunities of various DSD stakeholders (Enterprise Architecture (EA), Solutions, Security and Programs). The team has started to build a multi-cloud strategy. September 2021 Digital Services
Work with Digital Services and Transformation Office (DSTO), Corporate Services and business partners to develop a cloud strategy that includes internal roles and responsibilities, cost recovery mechanisms and well defined business objectives. July 2021
Obtain support from senior management to establish the cloud project as a priority. In collaboration with the Chief Financial Officer (CFO) and Transport Executive Management Committee (TMX), obtain full buy-in of cloud computing and the WLM migration project from TC's senior leadership. The Cloud Strategy will be developed first and supported with appropriate change management. September 2021
EA and Programs to complete the analysis, rationalization and prioritization of applications moving to the cloud. TC Center of Excellence (TC3oE) will work with EA (Application Portfolio Management), Solutions Center, the Business Continuity Planning (BCP) office and business sponsors to define the cloud end state of each application and prioritize this list based on mission criticality (as defined by the BCP office). August 2021
3 Incorporate lessons learned from other government departments’ and agencies’ (OGDAs) Cloud journeys or how they built/ recruited for their Cloud teams. Using lessons learned from OGDAs, the TC3oE will start building capacity and recruiting cloud team employees with specific assigned roles and responsibilities to ensure the team has the right skills and competencies to fulfill their mandate. September 2021 Digital Services
Identify an internal cloud funding model to provide sufficient permanent funding to hire and retain skilled staff. July 2021
Formulate a flexible recruitment strategy with HR. Work with HR advisors to address immediate staffing issues, identify a funding model and align the staffing prioritization with organizational prioritization. August 2021
If competitive processes are a requirement for certain positions, explore streamlined and flexible processes which could accelerate hiring (in alignment with internal/ external hiring requirements). Seek authorization from Chief Digital Officer (CDO) to use more flexible processes to fill out positions in the cloud team, including non-imperative CS-03/CS-04 technical position. August 2021
4 Consider adopting a learning plan/ strategy and exploring options for allocating time for employees to learn. Secure sufficient training funds from TC (incl. from the WLM project envelope where applicable) to provide staff with adequate training. August 2021 Digital Services
Adjust work expectations, to be built in the team's service levels to allow for 14 hours monthly of training/exploration. July 2021
Consider creating a Cloud Developmental Program to help junior employees stay and flourish at TC. Conduct an environmental scan amongst OGDAs to duplicate/adjust such programs already in place. To be developed in concert with a DSTO development program. September 2021 Digital Services
Conduct exit interviews to understand reasons for leaving to develop a more effective retention strategy. Introduce exit interviews as part of the exit procedures for all TC3oE staff. Completed:
April 2021
5

Improve current financial management practices by proactively working with DSD finance and the Financial Manage Advisor.

Review financial status – what are the priority expenditures (e.g. Amazon Web Services (AWS) work?).

Explore Cloud funding mechanics (Treasury Board submissions, output from GC Cloud funding exercise).

Determine fundamental resource needs (i.e. staff key cloud positions), allocate sufficient salary funding.

Work with the Chief Digital Officer, SSC Client Executive and their respective teams to fully understand the SSC COM offerings, including agreeing on roles and responsibilities, funding models and timelines. September 2021 Digital Services
Improve financial management practices including separating operations and capital expenditures, expense management and aligning with TC's Project Management Framework (PMF) to reduce risk. July 2021