IM/IT Projects Health Check and Governance Review

From: Transport Canada

October 2018

Table of contents

Executive Summary

Transport Canada (TC)'s Digital Services Directorate carries out around 30 application development projects annually to enable the department to deliver its mandate digitally. The development of new Information Management/ Information Technology (IM/IT) applications can take a few months to several years to complete. To manage this development, Digital Services uses an IM/IT Project Management Framework (PMF), which provides a formal structure to establish project timelines and deliverables and to ensure regular monitoring, and reporting on project status and financial health.

After several recent reviews and studies, Digital Services revised the PMF and began implementing a new PMF in 2018-19. Internal Audit assessed the new PMF to identify opportunities to refine the framework during this implementation period and to provide a foundation to monitor the performance of the governance committees over time. This assessment included a process mapping exercise to depict the project approval process under the new PMF as well as a survey and interviews with program executives to gauge their views to provide Digital Services with valuable early feedback.

The following observations were noted:

  • TC's IM/IT project governance committees are generally useful for program executives to exercise their oversight roles.
  • Progress is being made in communicating risks and issues that may have a significant impact on the delivery of projects; however, further improvement is required to meet program executives' expectations.
  • Project performance indicators currently in use are not very useful for decision making and require some revisions to meet program executives' needs.
  • Program executives' sense of accountability is wide ranging with respect to their projects, indicating misalignment between their and Digital Services' accountabilities which could lead to oversight issues for some projects.

Internal Audit identified four key PMF design principles based on the above observations and made the following specific recommendations to the Assistant Deputy Minister (ADM) of Corporate Services:

  1. Address the opportunities and weaknesses identified by the survey and interviews:
    • Develop guidance for project executives outlining their expected behaviors and roles and responsibilities in leading IM/IT projects.
    • Develop project performance indicators that are relevant and useful to a program executive's ability to make timely decisions on and have effective oversight over IM/IT projects.
    • Develop a communication plan to manage the perceived misalignment of control over decisions related to key project resources in order to keep programs informed of key resource allocation (business analysts and project managers) and changes throughout the project lifecycle.
    • Develop guidance and tools to help project teams determine when and how to escalate risks or issues relevant for executives involved in the governance of IM/IT projects-to include risk tolerance definitions or thresholds to help guide when to escalate risks or issues to a higher level.
  2. Use the survey tool to monitor and report on a regular basis the performance of TC's IM/IT project governance structure, gauging the extent to which the key principles are being met and specifically using the results to continue to improve the project management framework and governance processes.

Statement of Conformance

Although not an audit, this consulting engagement was conducted in a manner that conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of an external assessment of Internal Audit's Quality Assurance and Improvement Program.

Dave Leach (CIA, MPA) Director, Audit and Advisory Services

Martin Rubenstein (CPA, CIA, CFE) Chief Audit and Evaluation Executive

1. Introduction

1.1 Purpose

Internal Audit added this engagement to its risk-based audit plan to support the Department's Digital Services by providing independent advice on Digital Services' newly updated IM/IT Project Management Framework (PMF) and carrying out an initial assessment of the effectiveness of the recently introduced mechanisms that govern the PMF.

1.2 Background

TC uses policies, programs, legislative measures, regulations and guidelines to provide Canadians with a safe, secure, efficient, clean and sustainable transportation system. Of utmost importance is the ability to access quality data and analytical tools in order to provide advice, formulate options, develop policies and regulations, deliver programs and support decision making. Furthermore, the Government of Canada as a whole is looking to transform IM/IT in the workplace to deliver services as effectively and efficiently as possible. Therefore, a robust and cohesive approach to providing IM/IT solutions is critical.

TC's IM/IT services are delivered by Digital Services located in the National Capital Region (NCR), led by TC's Chief Information Officer (CIO) and supported by IM/IT teams in the five regional offices who functionally report to the CIO. Digital Services staff work closely with all TC program areas to “provide a reliable, secure, modern, sustainable IM/IT environment and thorough leadership and expertise, services, stewardship and solutions for clients and stakeholders in support of TC's and the Government of Canada's mandate”Footnote 1.

Internal Audit (IA) supports this effort by conducting audits and reviews within the IM/IT areas of TC and by keeping up to date with new IM/IT initiatives and departmental changes.

IM/IT Application Development Projects

The development of new IM/IT applications can take a few months to several years to complete. To manage this development, Digital Services uses an IM/IT PMF which provides a formal structure to establish project timelines and deliverables and to ensure regular monitoring and reporting on the status and financial health of projects.

Treasury Board of Canada Secretariat (TBS) has a Policy on the Management of ProjectsFootnote 2 (non-IT and IT projects) and requires periodic self-assessments of the project management capacity within departments. TBS uses these capacity self-assessments to determine the level of project approval authority delegated to each departmentFootnote 3. All of TC's current projects are within TC's delegated authority.

IM/IT Project Health Checks and Governance Review

In 2017 IA carried out an Audit of IM/IT Organization's Key Controls related to Finance, Procurement, and StaffingFootnote 4, examining the links between IM/IT projects and various administrative processes. Some of the findings highlighted issues with IM/IT project planning and management practices.

In 2017 Digital Services also completed an assessment of a specific marine IM/IT project and planned to perform similar reviews of other IM/IT projects. However, in light of a high-profile IT project failure in the federal government, Digital Services asked IA to lead a broader assessment of IM/IT projects and project governance by examining a sample of high-risk IT projects.

IA performed the assessment in two phases. The aim of the first phase of the assessment was to review key project management controls in place for a sample of five high-risk projects. At the end of the first phase, key weaknesses around project governance were highlighted across the sample of projects. These weaknesses were considered systemic and the Chief Information Officer (CIO) had started putting in place measures to address these weaknesses related to Digital Services' internal project management processes. Therefore, it was deemed little value added to expand the assessment of the remaining IM/IT projects. Instead it would be beneficial in phase 2 to concentrate on the new project governance structure with a focus on the one key area where the CIO has the least control – the role of the project sponsor (i.e. program executives) in project governance.

Further details of the two-phased approach are as follows:

Phase 1 - IM/IT Project Health Checks

In Phase 1 - IM/IT Project Health Checks, five high-risk IT projects were reviewed. The assessment used the TBS' IM/IT project review methodology which covers twelve topicsFootnote 5. IA prioritized these topics and focused on the six topics considered the key building blocks for a successful project (see Appendix A):

  • Topic 1: Business proposition – relevance of the project to the program's mandates and alignment to business strategies
  • Topic 2: Sponsorship, leadership, and governance – roles, responsibilities and level of commitment of key stakeholders
  • Topic 3: Concept and approach – strategies and approaches the project takes to deliver the expected benefits from the project
  • Topic 5: Risk management – project risk identification and mitigation processes
  • Topic 6: Project structure and mechanics – key project enablers
  • Topic 7: Business requirements – links between proposed changes in IM/IT-enabled business processes and desired outcomes

The assessment confirmed the weaknesses Digital Services identified in the marine project (i.e. ineffective project governance committees, high turnover of key project resources, and poor project management disciplines) were also pervasive in the five projects IA assessed (see Appendix B for more details). The root causes of the weaknesses in the areas examined were considered systemic and were likely present, to varying degrees, in all IM/IT projects.

The importance of project governance

Although there is no universally accepted definition of governance that is used by either the Government of Canada or TC, the Institute of Internal Auditors defines governance as:

“Policies and procedures used to direct an organization's activities to provide reasonable assurance that objectives are met and that operations are carried out in an ethical and accountable manner.”Footnote 6

Fundamentally, governance is intended to encourage behavior and activities that are aligned with an organization's mandate and priorities. To achieve effective governance, a sound structure, associated policies and processes, and people must be effectively working together to achieve common goals.

There are different layers of governance associated with management and oversight of IM/IT (see figure 1). IM/IT Project Governance is positioned under the Portfolio/Program/Project Management layer and was our focus in Phase 2.

Figure 1 - IM/IT Governance

 

Text description

The figure displays an image. The image illustrates layers of governance associated with the management and oversight of IM/IT. IM/IT Project Governance is a subset of the Governance of Enterprise IT and the Governance of Portfolio management.

The importance of the project sponsor in IM/IT project governance

The Government of Canada and industry associations such as the Project Management Institute (PMI) consistently state that executive engagement and sponsorship is the number one success factor for IM/IT projects (see Appendix C). Successful projects require active involvement of the project sponsor from initiation throughout execution to project close out. The capability of senior program executives to oversee IM/IT projects is therefore critical to the success of projects.

The results of our Phase 1 review confirmed that the level of sponsorship and engagement from senior management differed across programs. Although governance committees were in place on a project basis to drive IM/IT projects forward, the effectiveness of each committee varied due to the different interpretations and understanding of the roles, responsibilities, and accountabilities between Digital Services and the programs (project sponsors). As well, the committees did not provide a strategic lens for oversight and project management as they lacked a holistic and portfolio approach for a whole program.

The importance of a Project Management Framework (PMF)

What is a PMF?

A PMF is how a project is managed from an idea to completion. It is a set of standard project management processes, templates and tools that can be used to initiate, plan, execute, control and close a project. It provides a clear road map for a project supported by a common set of processes.

Why is it important?

A PMF is important as it clearly outlines the critical steps required for a project to achieve its objectives. In doing so, it also outlines roles and responsibilities of the project development team, management and executives and their involvement in decision making throughout the project lifecycle. Benefits of a robust project management framework which is adhered to include effective decision making, optimal resource allocation, clear expectations, controlled costs and early identification of risks and issues.

The introduction of a new PMF

The “waterfallFootnote 7” project management approach used historically at TC created challenges for IM/IT project governance – the “waterfall” approach expected an unrealistic level of precision, especially for cost estimates, and it committed both Digital Services and programs to impractical targets at the start of a project; creating unnecessary pressures and complexities into the project management environment.

To address this concern, Digital Services recently revised its PMF to align with TBS's project “gating” approach to improve senior executives' ability to make an “informed assessment of progress and issues, ultimately leading to better decisions on future plans and investments.”Footnote 8 The new PMF was soft-launched in fiscal year 2018-19 and is being applied on a case-by-case basis to on-going projects and to all new projects. Some of the guidance and tools supporting the new PMF were under development at the time of this report.

Phase 2 – IM/IT Project Governance Review

Recognizing that 2018-19 is a transition year, the CIO welcomed IA's proposal to assess the design of the new IM/IT project governance structure and establish a baseline rating for measuring the effectiveness of the new governance committees. The results of IA's review presented later in this report provide Digital Services an opportunity to refine its PMF in real-time as it is being launched as well as to provide a foundation to monitor the PMF structure over time so that the effectiveness of using these committees in support of project governance can be assessed.

1.3 Objective, Scope, and Methodologies

1.3.1 Objective

The objective of Phase 2 – Governance Review is to assess IM/IT project governance by reviewing the design of the new PMF and the effectiveness of the recently introduced governance mechanisms which are intended to assist management's oversight and decision making.

1.3.2 Scope

We assessed:

  1. The design of the oversight committee structure (see Appendix D) and the processes under the new PMF; and
  2. The level of accountability and engagement from sponsoring program executives from five of Digital Services' key client groups (Civil Aviation, Marine Safety and Security, Human Resources, Policy Group, Legislative and Oversight Modernization).

1.3.3 Methodologies

Process mapping

Since Digital Services was defining some of the structures and procedures related to the new PMF during our review, we facilitated a process mapping exercise to help Digital Services describe, clarify and refine the roles and responsibilities of the different positions and committees as the new PMF was being soft-launched. A draft process map depicting Digital Services' vision under the five-gate model was provided to Digital Services to aid its efforts on further clarifying and refining the IM/IT project governance structure and processes within the new PMF.

Document review

We reviewed the new PMF and supporting documents to identify potential gaps and weaknesses related to three critical areas:

  • The risks and issues escalation process
  • Deliverable and outcomes reporting
  • Project continue or discontinue decisions (go/no-go)
Survey and Interviews

We developed a survey to assess senior program executives' understanding of their roles and responsibilities and their level of engagement in the IM/IT project governance process. All program executives (director and above) responsible for IM/IT projects from Civil Aviation, Marine Safety and Security, Human Resources, Policy, Legislative and Oversight Modernization programs participated in the survey. One on-going 2018-19 project from each of the programs was used as a frame of reference for the survey. Face to face interviews were conducted to obtain specific examples and context for the responses received. Both the survey and the aggregated responses were communicated to Digital Services to provide them with the assessment tool and the baseline results. This enables them to measure performance over time and make future improvements to the project governance framework and processes.

1.4 Report Structure

Our report includes the summary of survey results, contextual information, and key principles that summarize the essential elements of each observation, an overall conclusion and specific recommendations for further improving IM/IT project governance.

2. Observations

Observation 1 – TC's IM/IT project governance committees are generally useful for program executives.

 

Figure 2 – Survey Statement 1

Text description

The figure displays a pie chart. The pie chart illustrates that the majority of the program executives agree with the statement “The governance committee I chair/participate in is useful and beneficial for the amount of time and effort it takes”. The breakdown of the results are as follows: 20% strongly agree, 50% agree, and 30% were neutral.

Under the new PMF, project oversight is expected to take place over three executive level committees: Executive Oversight Committees (EOC), the Director General Horizontal Committee (DGHC), and the Resource Management Committee (RMC). According to the survey conducted by IA, most program executives agreed with the statement “The governance committee I chair/participate in is useful and beneficial for the amount of time and effort it takes.” To help explain their responses to this statement, the following are brief descriptions and additional comments from executives regarding each of these committees.

Executive Oversight Committees (EOC)

EOCs were established by the current CIO in 2017 to be TC's primary oversight committees for IM/IT projects. They are to provide leadership, strategic guidance and direction on the program-related IM/IT investments. Each EOC is co-chaired by the CIO and the Director General (DG) of one of TC's major programs and acts as the main decision-making body for all the IM/IT projects within the DG's program. While normally held on a bi-monthly basis, the frequency of EOC meetings can be adjusted based on the complexity of each DG's portfolio of IM/IT projects.

Since an EOC was the only committee out of the three executive level committees that was involved in IM/IT project governance prior to the launch of the new PMF, executives' responses to our survey were based on their EOC experiences.

As illustrated in Figure 2, program executives appreciate the opportunity the EOC provides to exercise their IM/IT project oversight responsibilities. They also welcome the lessening of oversight burden from many, single-project committees used before EOC's portfolio approach to project governance.

Program executives noted that the oversight of horizontal projects is one area of improvement related to EOCs. Due to the similarity and interconnectedness of some of TC's programs (e.g. safety and security, internal services), some IM/IT projects have implications that are beyond a single TC program. However, except for a few high-profile projects, most horizontal projects are overseen by affected programs in parallel through separate EOCs. As such, program executives often make decisions related to horizontal projects in silos without the benefit of input and feedback from related programs. This in turn reduces the ability for executives to address TC's mandate holistically in the most effective and efficient manner.

DG Horizontal Committee (DGHC)

Started in January 2018, DGHC serves as a senior, cross-departmental forum for all DGs to discuss, challenge and become informed on horizontal initiatives that have the potential to affect multiple groups within the department. It is designed to maximize the use of DG talents and knowledge, strengthen the DG community, and create broader outreach while reducing duplication in consultation.

Digital Services' intent is to involve DGHC in the initiation and planning stage of a project to determine the horizontality of issues the project is aiming to address. This forum will also provide an opportunity for DGs across the department to scrutinize and improve IT project proposals by identifying potential synergies and efficiencies in solving common issues.

Digital Services' proposal for DGHC involvement in IM/IT project governance was positively viewed by most executives who agreed that there is value-added for the proposed challenge function. However, some executives questioned whether the DGHC should oversee the initiation and planning for every project since few are expected to affect the entire department. Instead, they believed the topic of horizontality should be addressed at a lower level. To this point, work is currently underway in the Software Architecture and Application Rationalization (SAAR) initiative to define TC's IM/IT requirements. The results from SAAR will eventually be used by Digital Services' Enterprise Architecture group to identify synergies and opportunities to leverage existing technologies, in an effort to deliver TC's IM/IT requirements effectively and efficiently.

Resource Management Committee (RMC)

RMC was created in December 2016 as the senior executive level forum for decision making related to financial, human and other resources across all of TC. It is co-chaired by the Associate Deputy Minister and the Chief Financial Officer and its members include Assistant Deputy Ministers, Regional Directors General, the Chief Audit and Evaluation Executive, Senior General Counsel, and Directors General in Corporate Services. It serves as a forum for the discussion, integration, management, and decisions related to the investment plan, resource management, and Grants and Contribution in order to provide recommendations to TC's Executive Management Committee (TMX).

Under the new PMF, RMC is expected to serve as the approval body (gate) for all IM/IT projects to proceed through the different project phases (i.e. initiation, planning, execution etc.). Since RMC will approve funding phase-by-phase under the new PMF, each project will now be subject to more rigorous due-diligence throughout its life cycle.

All of the executives interviewed agreed that the proposed IM/IT project governance responsibilities align with RMC's existing mandate. No additional comments or suggestions were provided.

Other comments on the new PMF

Some executives interviewed expressed concerns based on their limited experiences that the new PMF appeared to be adding hurdles to projects instead of facilitating their progress. They felt that there is too much governance focusing on planning and oversight instead of innovation and delivery. They explained that the vigor in planning is a weakness instead of a strength if it stifles agility and flexibility in developing solutions. These executives advocated for less red tape when moving projects through the gating process.

The same executives were also concerned about TC's apparent preference for in-house developed applications over commercial off-the-shelf (COTS) products. They felt that governance should focus on finding the best way to deliver on TC's mandate and to stay relevant and argued it may be worth paying more for a COTS product that allows immediate use than waiting years to develop an in-house solution.

Based on our analysis and findings, project governance committees are generally useful for executives. Without effective governance committees in which executives are able to engage in project decision making, projects are at risk of having inadequate oversight, not meeting their objectives, and of executives not being accountable or engaged in the project. To further promote and continue to strengthen the governance committees now in place, the following principle should be integrated into IT project governance and project management.

Key Principle 1

An effective IM/IT project governance structure should balance accountability and agility with the organization's culture and maturity in project oversight.

Observation 2 –Progress is being made but further improved communication of risks and issuesFootnote 9 between IM/IT project teamsFootnote 10 and program executives is needed.

According to statistics from the Standish Group, there were 1.5 decisions to be made for every $1,000 in project spendingFootnote 11. This means between 120 to over 20,000 decisions would be made during the lifetime of TC's current projects. Since executives are required to make some of the most important decisions, the timing of their involvement has a significant impact on the likelihood of the success of projects. Open and constant communication between executives and project teams is critical for project success as it enables effective risk and issue management throughout a project's life cycle.

As indicated, Gartner and IA both carried out reviews of Digital Services in 2017; both identified weak communication of projects' progress, risks and issues as a key area for improvement. Prior to the arrival of the new CIO, project leads reported feeling pressured from within the IM/IT organization to report the status of their respective projects as green (on track) regardless of their actual status. This practice was misleading and masked real project issues from program executives, thus reducing their ability to oversee IT projects and make effective decisions.

In response to these findings, the CIO stressed the importance of “speaking truth to power” to Digital Services staff and encouraged them to accurately report project status and risks in their project dashboards. Since then, more projects have been reporting yellow (need attention) or red (at risk) instead of green as their statuses, indicating to program executives potential risks/ issues that may have a significant impact on timely delivery of their projects. Most executives interviewed confirmed that project teams are now more open to discussing issues.

To assess the extent that communication has improved and to provide a baseline for measuring future progress, we asked executives to rate their level of agreement with the following two statements:

  • “I have fostered a culture of transparency with the project team.”; and
  • “Issues / Risks raised to my attention at the committees are at the right level and time.”

The following are the responses to these two statements:

 

Figure 3 – Survey Statement 2

Text description

The figure displays a pie chart. The pie chart illustrates that the majority of the program executives agree with the statement “I have fostered a culture of transparency with project team”. The breakdown of the results are as follows: 40% strongly agree, 40% agree, 10% are neutral, and 10% deemed not applicable.

 

Figure 4 – Survey Statement 3

Text description

The figure displays a pie chart. The pie chart illustrates that half of the program executives agree with the statement “Issues/ Risks raised to my attention at the committees are at the right level and time”. The breakdown of the results are as follows: 10% strongly agree, 40% agree, 20% disagree and 30% are neutral.

While 80% of the executives believe they have fostered a culture of transparency with the project team, only half of them are satisfied with how risks and issues are communicated to them. The discrepancy between the two responses in figures 3 and 4 indicates that there are still communication gaps between executives and project teams.

Executives explained that, as the reporting culture was being re-established, project officials sometimes struggled with balancing the type, granularity and timing of project information escalated up the hierarchy. In one example, executives were surprised to only be made aware at an EOC meeting that the roll-out of a high-profile application needed to be delayed due to official language requirements. The opposite was also true where EOCs were used to address issues that executives expected to be dealt with at a lower level (e.g. aspects related to the appearance of user interface). There is currently no guidance or tools available within TC to help project teams determine when and how to escalate risks or issues that might be relevant for executives involved in the governance of IT projects. There are no risk tolerance definitions or thresholds to help guide when to escalate risks or issues to a higher level.

Improved communication of risks and issues between IM/IT project teams and executives is needed based on the results from the survey and interviews held with executives. The following principle should continue to be integrated into IT project governance and project management.

Key Principle 2

Projects are more likely to succeed with open communication on risks and issues and with decisions enabled at the lowest level possible.

Observation 3 – Current project performance indicators are not very useful for decision making.

 

Figure 5 – Survey Statement 4

Text description

The figure displays a pie chart. The pie chart illustrates that only 20% of the program executives agree with the statement “The current project performance indicators help me exercise my decision making/ project oversight responsibilities”(10% strongly agree and 10% agree), and the majority of program executives either disagree (20%) or are neutral (40%) and 20% are deemed not applicable.

Project performance indicators are crucial for timely decision making. Currently, Digital Services uses a standardized dashboard to track and report project status. This dashboard includes information categories such as project health, schedule, risks, issues, financial summary, and results. However, as seen in Figure 5, executives surveyed were neutral or did not generally find that the project performance information they received helped them with their project decision making or oversight responsibilities.

As part of our survey, we asked program executives to identify the types of performance indicators that would be the most relevant to them. The following are examples of the highest-ranking indicator types:

  • Output related – positive response from end-users on work completed / functionalities developed to date
  • Scope related – functionalities defined/developed
  • Timeframe related – planned vs actual completion dates of project activities
  • Risk/Issue related – number of risks/issues identified and/or resolved

When asked for their feedback program executives expressed a preference for the performance indicators listed above because these indicators relate directly to a program executive's ability to deliver on the mandates of their programs. Many of them felt that, while the existing performance indicators are important, they are the responsibility of project officials at lower levels. Program executives rely on these project officials to manage the day-to-day performance details and expect to only be called upon when their authority is needed to resolve specific issues.

Several executives noted that financial indicators are not included in the top-ranking indicator typesFootnote 12. These executives found it odd that financial indicators were not ranked higher amongst their peers and suspected that the situation may be the result of the way that projects were funded. Under TC's current investment planning methodology, there is a general view that once an IM/IT project receives its funding, it continues to receive funding every year without exception. One executive suspected that the situation would be different (see Figure 6 below) if return on investment (ROI) was considered in TC's annual funding approval process. However, ROI is currently only considered in the project planning and approval process and is not calculated afterwards to help executives determine whether a project continues to justify its cost.

 

Figure 6 – Survey Statement 5

Text description

The figure displays a pie chart. The pie chart illustrates that majority of the program executives agree with the statement “I will stop / recommend stopping the project when I feel it is no longer viable”. The breakdown of the results are that 40% strongly agree, 50% agree and 10% are deemed not applicable.

One of the executives interviewed also pointed out the complexity of having risk/issue as a type of project performance indicator. While important, risk/issue as a category encompasses considerations from most other categories. It is therefore important to provide a structure to specify what types of risks and issues project teams should consider and establish thresholds for when to escalate them to senior executives. There are currently no guidelines available on risk and issue escalation, and it is up to individual project officials to apply their own judgement.

A final consideration related to performance indicators is that executives' information needs may change depending on the project's phase. Our analysis showed that executives whose projects were in the execution phase appeared to have a narrower focus in terms of performance indicators than those with projects in the planning and design phase. Output and timeframe became even more important for executives with projects in later rather than earlier stages.

Based on our analysis and findings, project performance indicators are not always useful in decision making. The following principle should be integrated into IT project governance and project management.

Key Principle 3

Relevant and timely performance information is an important factor for improving or maintaining executives' level of engagement.

Observation 4 – Program Executives' sense of accountability is tied to the extent of their control over key project resources.

 

Figure 7 – Survey Statement 6

Text description

The figure displays a pie chart. The pie chart illustrates that the program executives express a wide range of responses to the statement “My program is fully accountable for ensuring the project achieves its business objectives”. Less than half of them agree (10% strongly agree and 30% agree), while the remaining 60% are divided (20% disagree, 20% neutral, 10% not applicable and 10% no response).

Of the six statements assessed in the survey, program executives expressed the most diverse responses to the statement “My program is fully accountable for ensuring the project achieves its business objectives.” When asked to comment on the results, almost all program executives confirmed that they should be accountable. However, some of them did not feel they could exercise their accountability for one of two reasons: 1) the lack of relevant and timely information for decision making, and 2) misalignment of control over decisions related to key project resources.

Since the topic of relevant and timely information was already covered in the previous two observations, this section focuses on the issue related to the perceived misalignment of control.

Control over key project resources

As one executive stated, most IM/IT projects are in fact business transformation projects. This means that change management is also happening with other program aspects, notably with program staff, in order to achieve the desired results. However, while programs are in full control of most of their programs and have key resources necessary for transforming business processes, they share control for IM/IT projects with Digital Services.

For instance, Business Analysts and Project Managers are key resources because they ensure the approach of the IM/IT project is consistent with a program's overall change management strategy and that the project meets the program's needs in terms of scope and timeline. However, Digital Services is currently in charge of acquiring and assigning these resources to IM/IT projects. Many executives see this arrangement as an issue because it reduces a program's ability to properly manage these resources. It also creates uncertainty and disruptions because Digital Services sometimes reassigns high performing resources to “at-risk” projects.

Digital Services is also responsible for managing the capital budget for most IM/IT projects. Although programs are responsible for monitoring and challenging a project's financial status, program executives are somewhat removed from the project's day-to-day decision making. They do not feel that they can be held fully accountable for the progress of IM/IT projects.

One of the five programs reviewed uses an alternative structure to manage their IM/IT projects. This program is in full control of decisions related to key project resources (such as Business Analysts and IT Project Managers). In addition, it has business managers who are dedicated almost full time to IM/IT and are responsible for monitoring and reporting project status to the EOC. While still relying on Digital Services to provide technical guidance such as project management templates and cyber security requirements, executives from this program feel a higher level of accountability for their projects compared to those under the standard structure.

Based on the survey results and interviews, executives' sense of accountability for their respective projects was not always apparent. While the alternative accountability structure is showing some early successes, a one-size-fits-all approach to project governance might not be ideal at this time based on different programs' project management capabilities. Therefore, it is important to monitor the performance of each IM/IT project governance structure and make iterative improvements to bring about higher level of engagement and project management capability to TC as a whole.

Key Principle 4

An appropriate accountability structure is necessary to ensure a high level of engagement from programs.

3. Conclusion

IM/IT is one of the most immediate and important means that governments rely on to provide high quality services to their citizens. Given a recent high-profile IT project failure in the federal public service, program executives across the federal government should expect intense scrutiny on their IM/IT projects. To respond to these expectations from stakeholders, TC should monitor the performance of its IM/IT project governance structure and continue to make improvements based on the following key principles:

  • An effective IM/IT project governance structure should balance accountability and agility with an organization's culture and maturity in project oversight.
  • Projects are more likely to succeed with open communication on risks and issues and with decision making enabled at the lowest level possible.
  • Relevant and timely performance information is an important factor for improving or maintaining executives' level of engagement.
  • An appropriate accountability structure is necessary to ensure a high level of engagement from programs.

It is important to note that the Department as part of its “Transformation Initiative” Footnote 13 recognizes the importance of project management and the need to strengthen project management practices throughout TC. The Transformation Initiative cites a number of principles which are consistent with the ones we have proposed to specifically focus on improving the likelihood of success for new and existing IT projects.

Ultimately, the effectiveness of IM/IT project governance will depend on the level of its participants' engagement. We believe that, by incorporating the key principles, described in our report into the governance structure and PMF, program executives will be supported to become more engaged in IT project governance. This will in turn create a more positive project culture in TC and help improve its project management capabilities over time.

4. Recommendations

As part of on-going efforts to strengthen TC's IM/IT project governance the ADM of Corporate Services should:

  1. Address the opportunities and weaknesses identified by the survey and interviews and:
    • Develop guidance for project executives outlining their expected behaviors and roles and responsibilities in leading IM/IT projects.
    • Develop project performance indicators that are relevant and useful to a program executive's ability to make timely decisions on and have effective oversight over IM/IT projects.
    • Develop a communication plan to manage the perceived misalignment of control over decisions related to key project resources in order to keep programs informed of key resource allocation (business analysts and project managers) and changes throughout the project lifecycle.
    • Develop guidance and tools to help project teams determine when and how to escalate risks or issues relevant for executives involved in the governance of IM/IT projects- to include risk tolerance definitions or thresholds to help guide when to escalate risks or issues to a higher level.
  2. Use the survey tool to monitor and report on a regular basis the performance of TC's IM/IT project governance structure, gauging the extent to which the key principles are being met and specifically using the results to continue to improve the project management framework and governance processes.

5. Management Action Plan

The following summarizes the audit recommendations and management's plan to address them.

OPI: Office of Primary Interest; OSI: Office of Secondary Interest

-

Recommendation

Management Action Plan

Completion Date (for each action)

OPI direct report for each specific action

1a

Develop guidance for project executives outlining their expected behaviors and roles and responsibilities in leading IM/IT projects.

Digital Services will

  • create a Responsible/ Accountable/ Consulted/ Informed (RACI) chart for the various phases of the PMF, providing revised roles and responsibilities for each key project resource and governance committee,
  • as required, request that relevant committees' Terms of References are aligned to the RACI,
  • reinforce the TBS key leadership competencies for executives in the guidance provided with the RACI,
  • ensure the PMF Project Charter template includes executives' roles and responsibilities related to leading IM/IT projects, and
  • communicate the changes to stakeholders.

March 2019

OPI: Digital Services

OSI: EPMO

1b

Develop project performance indicators that are relevant and useful to a program executive's ability to make timely decisions on and have effective oversight over IM/IT projects.

Digital Services will

  • establish performance indicators that are relevant, useful, and improve decision making related to IM/IT projects,
  • integrate these into the Project Management Framework, and
  • communicate the changes to stakeholders.

March 2019

OPI: Digital Services

OSI: EPMO, DGHC

1c

Develop a communication plan to manage the perceived misalignment of control over decisions related to key project resources in order to keep programs informed of key resource allocation (business analysts and project managers) and changes throughout the project lifecycle.

Digital Services will

  • include roles and responsibilities related to project resourcing in the RACI developed for Recommendation 1a,
  • ensure the PMF Project Charter template includes roles and responsibilities related to key project resources (e.g. hiring, funding, assignment, performance assessment, and replacement) so project stakeholders clearly understand and sign-off on their level of control over key resource decisions,
  • ensure that along with performance indicators established for Recommendation 1b, key resource allocations/changes are highlighted to executives,
  • ensure instructions for completing IM/IT project dashboards include the need to identify resource allocation risks/issues so that decisions related to key project resources (business analysts and project managers) are documented and discussed as required, and
  • communicate the changes to stakeholders.

March 2019

OPI: Digital Services, DGHC

OSI: EPMO

1d

Develop guidance and tools to help project teams determine when and how to escalate risks or issues relevant for executives involved in the governance of IM/IT projects-to include risk tolerance definitions or thresholds to help guide when to escalate risks or issues to a higher level.

Digital Services will

  • develop a risks and issues escalation process and risk tolerance definitions/thresholds to help project teams determine when and how to escalate risk or issues relevant for executives involved in the governance of IM/IT projects,
  • integrate these into the Project Management Framework, and
  • communicate the changes to Stakeholders.

March 2019

OPI: Digital Services

OSI: EPMO

2

Use the survey tool to monitor and report on a regular basis the performance of TC's IM/IT project governance structure, gauging the extent to which the key principles are being met and specifically using the results to continue to improve the project management framework and governance processes.

The first survey was completed in the Spring of 2018. The next survey will take place in May of 2019.

May 2019

OPI: Digital Services

OSI: EPMO

Appendix A: Comparison of TBS Independent Review Topics with Studies on Project Success Factors

The following seven topics from TBS' independent review methodology are also identified as key to project success in other recent studies of IT projects. They were therefore the priority for IA's assessment.

TBS Independent Review Topics

CHAOS factors of successFootnote 14

Phoenix - Goss Gilroy studyFootnote 15

1. Business Proposition

X – optimization; clear vision and business objectives; realistic expectations

X – business case and outcome management

2. Sponsorship, leadership and governance

X – executive sponsorship; ownership

X – governance and oversight; change management

3. Concept and approach

-

X – initiative definition

4. Organizational readiness and capacityFootnote 16

-

X – capacity management

6. Project structure and mechanics

X – modest execution; smaller project milestones; proper planning; project management expertise

X – initiative and project management

7. Business requirements

X – standard architecture; user involvement; clear statement of requirements

-

10. Human factors

X – emotional maturity; skilled resources; agile process; hard-working, focused staff

-

Appendix B: Key issues per TBS Independent Review Topics for Phase 1

The five projects assessed in Phase 1 were: Continuing Airworthiness Web Information System (CAWIS), Multimodal Personnel Document Issuance System (MPDIS), Navigable Waters Database System (NWDS), Pleasure Craft Operator Database System Rejuvenation (PCOCDS), and Transportation of Dangerous Goods (TDG) Core. The below issues apply to all projects unless otherwise specified.

Topic 1 – Business proposition

  • Projects did not receive the executive direction and attention they needed to succeed
  • Project sponsors, stakeholders and project management did not share a common vision of the project
  • Poor governance, planning and decision making: impossible to deliver upon agreed to scope.

Topic 2 – Sponsorship, leadership, and governance

  • Level of engagement from business (Perception of IM/IT being fully accountable)
  • Governance committees and decision-making bodies not effective
  • Lack of sponsorship resulted in IM/IT “driving” the project (MPDIS, NWDS)
  • Some projects received low visibility and did not benefit from the same level of project oversight (CAWIS, PCOCDS)
  • Project dashboards circulated at governance meetings do not reflect reality. Updates to dashboards are not data driven

Topic 3 – Concept and approach

  • Rigor and quality of options analysis is not in place

Topic 5 – Risk Management

  • Risk and mitigation plans were identified but were not escalated beyond the director because the project was considered a low priority by all (PCOCDS)
  • In some cases, program and project risks were not effectively managed, and scope spiraled out of control (NWDS, MPDIS)
  • Communication of risks between IM/IT and business was not consistently performed

Topic 6 – Project structure and mechanic

  • Project documents are not living documents
  • Quality, continuity and availability of PM, BA and developers
  • Heavy reliance on contracted resources for key project positions such as project managers, developers and business analyst
  • Quality control of code

Topic 7 – Business requirements

  • Clarity of project deliverables in relation to business outcomes
  • Lack of defined business requirements
  • Not a clear understanding of business requirements between business and IM/IT (TDG CORE)

Appendix C: References on the Importance of Project Sponsor Engagement and Executive Sponsor Behaviors in Relations to Project Success

Government of Canada

Project Management Institute

The Standish Group

Harvard Business Review

Canadian Government Executive

Appendix D: IM/IT Project Governance at Transport Canada

 

Text description

Image is a flow chart showing the hierarchy of IM/IT project governance at Transport Canada

  • TMX Chaired by Deputy Minister
    • TBS’s Enterprise Architecture Review Board (EARB)
      • Resource Management Committee
      • DG Horizontal Committee
      • Executive Oversight Committees
      • Project Review Committee
        • Architecture Review Board
        • Project oversight Secretariat

 

IM/IT Projects Health Check and Governance Review
(PDF, 564 Kb)

 

Report a problem with this page

Date modified: