Audit of IM/IT Project Life Cycle Controls

EXECUTIVE SUMMARY

Introduction

Information Management (IM) is the combination of structures and processes an organization uses to manage information, from planning and systems development to disposal or long-term preservation. Information Technology (IT) is used for the implementation/delivery of policies and programs to help increase productivity and enhance service to the public. Management of IT includes planning, building (or procuring) and operating systems.

At the time of this audit, there were 43 IM/IT projects underway at Transport Canada (TC) at a Total Estimated Cost (TEC) of $97M. The majority span multiple fiscal years.

Corporate Services' Technology and Information Management Services Directorate (TIMSD) has been given responsibility for providing IM/IT support services and functional expertise and direction to TC groups and regional offices. TIMSD is headed by the Department's Chief Information Officer (CIO).

Over 90% of the 43 IM/IT projects underway are software applications that are being developed to support operational requirements. Examples include inspection reporting databases to support regulatory programs and financial systems. The majority are initiated and managed by groups outside of TIMSD (i.e., Safety and Security, Corporate Services, Policy, Programs) Footnote 1. Project sponsors, usually at the Director General (DG) level, have authority over, and are responsible for, these projects. TIMSD is responsible for setting out the policies and procedures with which project sponsors are responsible to comply. It is important to highlight that project sponsors and their respective Assistant Deputy Minister (ADM) are ultimately accountable for the successful completion of their IM/IT projects.

Audit Objectives & Scope:

The objective of this audit was to assess the adequacy of the Department's management control framework for IM/IT projects, identify control gaps, if any, and test the operating effectiveness of existing controls.

In assessing the Department's management control framework, the audit looked at the following:

the governance structure that senior management has put in place to direct, manage and monitor IM/IT projects;
the Department's risk management practices for IM/IT projects; and
the Department's controls to ensure that IM/IT projects deliver the expected benefits and are completed based on the approved budget, schedule and scope.

The audit examined 19 IM/IT projects from across the Department. The projects represent a TEC of approximately $68M and included twelve projects underway, five completed projects and two that had been cancelled.

Conclusion

Although the Department has established many polices and processes for IM/IT projects we are concerned with how burdensome many of them are for Project Sponsors to follow, thus reducing the effectiveness of the controls and making the processes inefficient. At the same time, there are some fundamental gaps in the Department's management control framework for IM/IT projects.

Overall, we found that there are many committees, processes, and procedures; however, they are not effective in ensuring the Department is optimizing its IM/IT investments or managing its projects effectively. Senior management receives insufficient information for effective decision-making and oversight. Policies and processes are complex, poorly understood and they are often not followed. Monitoring practices are inadequate to ensure that there is appropriate oversight for IM/IT projects. Projects often span several years and represent significant investments, yet there is no post-project assessment of value-for-money.

The result is a very significant risk that the Department will not have its IM/IT needs met and will not receive good value for its IM/IT investments.

Statement of assurance/reliance

It is our professional judgment that the audit has been conducted in accordance with the Internal Auditing Standards of the Government of Canada as prescribed by the Comptroller General. Satisfactory procedures for the audit have been conducted, and sufficient relevant evidence has been gathered to support the accuracy of the opinions provided in this report.

Signatures

Signed by

Dave Leach (CIA) Director,
Audit and Advisory Services

June 25, 2012

Date

Signed by

Laura Ruzzier, Chief Audit Executive

June 25, 2012

Date

1. INTRODUCTION

1.1 PURPOSE

The objective of the Internal Audit function is to support the Deputy Minister by providing independent advice on the Department's governance, controls, and risk management processes. The Internal Audit function's internal audit planning process ensures that limited resources are targeted to the areas of highest risk and significance.

An audit of Transport Canada's (TC) IM/IT Project Management Life Cycle Controls was included in the Department's 2010/11 Risk-Based Audit Plan, which covered a three-year period from 2010/11 to 2012/13. The purpose was to provide assurance to the Deputy Minister and the Transport Canada and Infrastructure Canada Audit Committee that the Department's management control framework for IM/IT projects is adequate and effective.

1.2 BACKGROUND

At the time of this audit, there were 43 IM/IT projects underway in the Department at a Total Estimated Cost (TEC) of $97M. The majority span multiple fiscal years with start dates as early as 2004.

Over 90% of the 43 projects underway are to develop software applications to help a branch to meet a particular need. Examples include inspection reporting databases to support regulatory programs and financial systems. The majority of these projects were initiated and are managed by groups outside of TIMSD (e.g., Safety and Security or Policy). Project sponsors, usually at the Director General (DG) level, have authority over, and are responsible for, these projects; TIMSD is responsible for providing functional direction. It is important to note that project sponsors and their Assistant Deputy Minister (ADM) are ultimately accountable for the successful completion of their IM/IT projects.

The remaining projects are infrastructure projects related to the delivery of IM/IT services to departmental staff. These projects were initiated by and are being managed by TIMSD.

A list of the IM/IT projects underway in TC is provided in Appendix A.

1.3 AUDIT OBJECTIVE & SCOPE

In assessing the Department's management control framework, the audit looked at the following:

the governance structure that senior management has put in place to direct, manage and monitor IM/IT projects;
the Department's risk management practices for IM/IT projects; and
the Department's controls to ensure that IM/IT projects deliver the expected benefits and are completed based on the approved budget, schedule and scope.

A judgmental sampling approach was used to identify 19 IM/IT projects from across the Department for a detailed review. Twelve of the 19 projects were still underway, five had been completed, and two had been cancelled. TEC of projects underway was $49M. TEC of completed and cancelled projects $19M Footnote 2.

A list of projects in the sample audited is provided in Appendix B. The audit team made requests for project information throughout the audit but in a number of cases the information requested was not available, for example, a listing of all projects underway with TEC including additional funds requested and original revised schedule. The audit team had to build on information obtained from the Project Oversight Secretariat (POS) in order to create the listing in Appendix B of sample projects with information about TEC and schedule. Of the 19 sample projects, five requested additional funding. On average the additional funds requested were $1.5M per project. Of the 19 sample projects, 12 had revised their estimated completion date. On average the additional time estimated to complete the projects was two years. (A list of timelines from projects reviewed is provided in Appendix C.)

1.4 AUDIT APPROACH

Our professional judgment is that this audit was conducted in accordance with the Internal Auditing Standards of the Government of Canada as prescribed by the Comptroller General of Canada.

Audit criteria were based on the following:

Control Objectives for Information and Related Technology (COBIT), an IT governance framework created by the Information Systems Audit and Control Association (ISACA) and the IT Governance Institute (ITGI), which provides generally accepted best practices for information technology project management;
Treasury Board Secretariat's (TBS's) Enhanced Framework for the Management of Information Technology Projects; and
Approximately 100 TC documents that in effect comprise the Department's IM/IT Project Control Framework (a list of documents is in Appendix D).

Satisfactory audit procedures were developed and sufficient relevant information was gathered to ensure the accuracy of opinions expressed in this report. Control areas were assessed for the adequacy of their design and their operational effectiveness.

The planning and conduct phases of the audit were completed between December 2010 and April 2011. The reporting phase, including fact verification, was completed between May 2011 and September 2011. The audit was conducted in-house using departmental audit staff.

During the planning phase, to understand the project management life cycle, the audit team interviewed the CIO and several TIMSD staff members. Detailed audit criteria were developed from TC's management control framework, Treasury Board (TB) policies and COBIT. A listing of audit criteria is attached in Appendix E.

During the conduct phase, to assess the effectiveness of project governance, risk management, and controls, the audit team interviewed the CIO, TIMSD staff members, chairs and members of IM/IT governance committees, and project sponsors and project managers (majority of which are outside TIMSD). As well, detailed reviews were performed of governance committee decision records and sample project documentation. The decision records and project documentation were assessed against the criteria identified in the planning phase. Factual evidence received from project managers was validated with project sponsors.

1.5 STRUCTURE OF REPORT

Audit findings are provided in four sections: Governance, Project Management, Project Monitoring/Reporting, and Post-Project Review.

Conclusions and recommendations to address weaknesses and gaps in the findings section are provided in Conclusions and Recommendations sections.

The Recommendations section includes a Management Response and Action Plan (MRAP) from the Department. The MRAP gives management's response to the audit recommendations, and commitments and timelines for addressing identified weaknesses or gaps.

2. FINDINGS: GOVERNANCE

A review of the Department's governance framework for IM/IT projects reveals major gaps/weaknesses.

IT governance is the combination of leadership, structures and processes implemented to ensure that the IT function sustains and enables the organization's strategies and objectives. According to the ITGI, IT governance is the responsibility of executive management. IT governance is not practiced in isolation nor is it the sole responsibility of IT management. Due to the complex and specialized nature of IT, executive management must set direction and ensure controls are in place while relying on the lower layers of the organization to provide information for decision-making and evaluation activities.

2.1 DEPARTMENTAL IM/IT GOVERNANCE COMMITTEES

Governance for IM/IT projects at TC is exercised by means of a number of committees, starting with the Transport Canada Executive Management Committee (TMX). TMX is responsible for setting strategic direction for the Department, including IM/IT, and for approving and monitoring IM/IT projects. Prior to 2009/10, TMX was supported by the TMX Capital and Asset Management Sub-Committee. This committee was chaired by the ADM, Programs and consisted of TMX level members. In 2009/10, this committee was discontinued, and in its place the Resource Management Council (RMC) was established with similar objectives. This council is co-chaired by the DG, Finance and Administration, and the Regional Director General of Quebec Region. The majority of the members are DG level.

TIMSD, headed by the Department's CIO, who reports to the ADM, Corporate Services, is responsible for providing IM/IT support services and functional expertise and direction to TC groups and regional offices.

A Business IM/IT Council, co-chaired by the CIO and a DG representative from the Department's Business Groups (i.e. Safety and Security, Policy, Programs, Corporate Services) is responsible for liaison between the Business Groups and TMX via the ADM, Corporate Services. The Business IM/IT Council is responsible for recommending IM/IT strategy, establishing IM/IT principles and standards, recommending IM/IT investments, monitoring the status of ongoing projects, and measuring performance. It is composed of a DG from each headquarters branch and a director from each region; these members are nominated by TMX.

The Business IM/IT Council has two sub-committees:

A Business IM/IT Investment Committee (Investment Committee), responsible for evaluating proposed major IM/IT investments against predetermined investment criteria; and
A Business IM/IT Architecture & Standards Committee, responsible for the overall governance of all IT architectures.

Since April 2009, TIMSD has had the POS function. One of the primary responsibilities of the POS is to assist the governance committees with monitoring of IM/IT projects. It is also responsible for providing support and guidance to project managers and ensuring that the Department's IM/IT policies are consistent with related Treasury Board (TB) policies.

IM/IT project governance bodies are not meeting their objectives.

The audit team expected to find that the Department's strategic IM/IT needs and priorities had been clearly defined and formally approved by TMX and that there would be evidence of this in TMX decision records. TMX decision records were reviewed for more than a two-year period, from January 2009 to April 2011. There was no reference to a substantial discussion of a departmental IM/IT strategy or approval of IM/IT strategic plans.

The audit team expected to find Terms of Reference (TORs) for the three IM/IT governance committees, that these TORs were reviewed and updated periodically, and that evidence that the committees were fulfilling their most important responsibilities would be found in committee decision records. The audit team examined the TORs and decision records for the three IM/IT governance committees for calendar years 2009 and 2010. Through this review, and through interviews with committee members and TIMSD staff, it was found that the three committees' TORs had not been updated in several years.

None of the three IM/IT governance committees had met as frequently as required by its TOR. The Business IM/IT Council's TOR, for example, states the Council will meet monthly or more frequently, if required, but records indicate the Council met five times in two years.

Additionally, based on the review of the decision records, it was noted that actions/decisions regarding some of the committees' key objectives did not take place such as recommending IM/IT strategic plans to TMX, monitoring IM/IT status reports and review of performance reports for completed projects.

2.2 PROJECT REQUESTS AND APPROVALS

The Department has established requirements for the preparation, submission and approval of all capital project requests, including IM/IT project requests. Sponsors of proposed IM/IT projects must complete three documents:

Project Proposal
Project Approval Document (PAD) and
Project Complexity and Risk Assessment (PCRA)

The Project Proposal is a document that must be completed to request consideration of a new IM/IT project. It should provide a project summary, estimated costs, and benefits information. The project proposal process for IM/IT projects is managed by TIMSD. The Investment Committee reviews and prioritizes proposals in terms of value/importance to the Department and makes recommendations to the Business IM/IT Council. The Business IM/IT Council reviews and accepts (or modifies) the Investment Committee's recommendations.

Following approval of the Project Proposal, a PAD is required. This process is managed by Corporate Financial Management. The PAD expands on information included in the Project Proposal, such as cost, schedule and governance. The PAD also includes an options analysis section to document the options (e.g., make vs. buy) considered.

Since April 2010, the PCRA has been a TBS requirement for projects over $1M. There are 64 complexity and risk-related questions across seven categories (i.e. project characteristics, strategic management risks, and procurement risks) to be completed. The purpose is to determine the risk and complexity rating (i.e., from 1 to 4) of a project and, in turn, determine the required project approval authority for the PAD. A PCRA rating of 1 requires ADM approval, 2 requires ADM and Deputy Minister (DM) approval, 3 requires ADM, DM and Minister approval and 4 requires ADM, DM, Minister and TB approval. In addition, Finance and TIMSD also review and approve all IM/IT PADs. The majority of PCRA ratings for TC are 2 Footnote 3.

A project funding request is also put forth to RMC and TMX for approval. Upon both RMC and TMX funding approval and PAD approval, a budget allocation is provided for the TEC of the project.

The oversight and challenge function with respect to IM/IT projects at the senior level of the Department is limited.

The audit team expected to find that TMX, as the senior decision-making body for capital projects, would receive information and supporting analysis about major IM/IT project proposals with the funding recommendations, in order to allow TMX to play a strong oversight/challenge role.

From the review of TMX and RMC decision records, it was observed that IM/IT project proposals were incorporated into aggregated capital project listing (e.g. Initial Budget Delegation) for approval. There was no indication of a TMX discussion of major IM/IT proposals, the level of IM/IT project activity, or the Department's capacity to manage that level of project activity.

There was also no indication that a TMX approved IM/IT strategic plan is used as a basis for IM/IT proposal assessments, to ensure alignment between projects and overall departmental needs/priorities.

The request and approval process for IM/IT projects is onerous and difficult to follow.

The audit team expected to find well articulated requirements for IM/IT project proposals; however, this was not the case. In order to map and understand all of the requirements related to the IM/IT project request and approval process, the audit team had to review approximately 20 documents, including guides, templates, policies, etc and it took several days to gain a thorough understanding of the process. These documents are in several different intranet locations and some have outdated information, links and references. Finally, the requirements are not scaled based on the size, nature, complexity and risks associated with the project.

Clarification of PAD procedures is required.

Requirements related to preparation, revision and approval of the PAD are documented in the TP117 Financial Policy and Procedures Manual. Although the process is clear when additional funding is required for a project, some other areas require clarification and enhancement. For example:

A revised PAD is required for a scope reduction, but scope reduction has not been defined, and it is not clear whether the project should continue while the revised PAD is being reviewed for approval.
A revised PAD is not required for an IM/IT project schedule change, although a significant extension could affect the project's viability, given the rapidly changing nature of technology. It could also affect departmental resources.

Although there was an approved PAD for all 19 projects in the sample, the options analysis was very limited for several of the projects.

The purpose of the options analysis is to show that there has been a thorough analysis of alternatives to meet a business need. The audit team observed that for a number of the projects, the options analysis covered only two options: the status quo and the proposed new application. There was limited evidence that other alternatives had been adequately explored. For example, only a small number of the PADs included a comparison of developing the application internally with the purchase of a Commercial Off-The-Shelf (COTS) product.

The Department maintains a listing of all its software applications, but the current process does not require project sponsors to review the listing to determine whether there is an existing application that could be altered or built upon to meet the proposed project requirements.

As well, the Department has access to Gartner, an information technology research and advisory company that can provide technology-related advisory services. Gartner offers a variety of research tools to help subscribers to search for COTS applications and compare features, but there was no indication that this is used on a regular basis and there is no mention of Gartner as an option for project sponsors in information materials.

Finally, given all departments have similar reporting requirements we would expect that part of the options analysis guidance would recommend project sponsors research the opportunities to adopt other departments' existing applications.

2.3 PROJECT MANAGEMENT CONTROL FRAMEWORK

A management control framework is a set of policies and procedures to ensure objectives are met and results achieved. TC's framework for the management of projects is described in multiple documents (see Appendix D) which are used to communicate roles and responsiblities for the effective management and oversight of departmental IM/IT projects.

Although some policy documents relating to IM/IT projects have been put in place, they are complex and there are some gaps and inconsistencies.

Clear policies and directives are imperative to the success of IM/T projects. TC has developed policy documents on the development of IM/IT projects, most notably the TC Business IM/IT Investment Operating Principles and the Application Management Framework.

Through the audit examination of IM/IT project policies, it was observed that policy/framework requirements are in several different places on the TC intranet site, making it a challenge for project sponsors to locate relevant documentation. Audit interviews determined that many project sponsors and managers were not aware of these documents. IM/IT policy and procedure documents had various titles, e.g., "Principles" and "Rules," that had not been defined and do not conform to the TBS's naming conventions for policy instruments.

Finally, the policies lack "scalability," that is they make no distinction between projects based on size, nature, risk or complexity, when it comes to project requirements.

The following provides some examples:

The TC Business IM/IT Investment Operating Principles provides an overview of the Department's IM/IT project management expectations. In the audit examination of this document and other related IM/IT project documentation, some discrepancies were identified and it was found that some of the requirements have not been fully implemented. For example, the TC Business IM/IT Investment Operating Principles requires that all IM/IT projects are to employ the Macroscope methodology in the management and development of IM/IT projects; however, other IM/IT project information states that the methodology is optional.

While Macroscope is an established methodology, it is extensive in terms of its information and documentation requirements and can be difficult to navigate for project managers unfamiliar with the tool. According to the TC Business IM/IT Investment Operating Principles, at the start of a project, project managers are required to review the listing of documentation within Macroscope and determine what is appropriate for their project. Through examination of the Macroscope tool, it was noted that this task would require the review of approximately 80 documents and there is no guidance as to how this methodology may be scaled appropriately for projects of differing complexity, size, nature and risk.

The Application Management Framework was created to guide the development of application related IM/IT projects. However, the Framework has not been formally integrated into the IM/IT project process. Audit interviews with project managers indicated that many were not aware of the Framework and how it fit within IM/IT project requirements. Audit examination of the Framework noted that it is general in nature and does not provide baseline prescriptive measures. In addition, key aspects of application development are not covered, e.g., security, segregation of duties, and user acceptance testing.

3. FINDINGS: PROJECT MANAGEMENT

While there were some positive management practices, there were also some major weaknesses, including weaknesses that affect all projects.

3.1 PROJECT OVERSIGHT

TB and TC policies both require the following roles be assigned for each IM/IT project:

Project Sponsor - has overall responsibility and accountability for the project. The project sponsor is typically a senior official (usually DG level) in the organization responsible for the business function that the project will support.
Project Manager - performs the day-to-day management of the project. Project managers should have demonstrated knowledge, skills and experience commensurate with the size, complexity and risks related to the project.

3.1.1 Project Steering Committees

Requirements regarding the establishment and responsibilities of IM/IT project specific oversight are not formally documented.

Based on audit interviews with members of TIMSD, project sponsors are required to establish steering committees to provide oversight for all mid to large size IM/IT projects. The audit team, therefore, expected to find documented steering committee requirements including scalability based on the size, nature and complexity of the project. However, the audit found that the Department has not formally documented the requirements around the establishment and responsibilities of the project steering committees. For purposes of the audit, steering committees for projects greater than $1M were reviewed.

The project sponsor is accountable for their IM/IT project and as such would be expected to establish appropriate oversight for a project and to ensure key decisions are adequately documented. There is currently no formal process within the management control framework to communicate the expectations of a project sponsor and to ensure that the requirements and responsibilities of the role are clearly understood.

Project steering committees are not fulfilling their oversight role.

The audit team expected to find steering committees or some other method of providing oversight for each of the sample projects. The audit found that the majority of sample projects did not have active steering committees in place with sufficient evidence (i.e. decision records) to demonstrate that the committees were providing effective oversight including management of project risks, issues, ensuring alignment with business objectives, etc.

Based on interviews with departmental staff involved in IM/IT projects it was noted that the large number of on-going projects often presents a challenge with respect to the organization and attendance of steering committee meetings.

3.1.2 Risk Management

Risk management is the overall process to identify, assess, manage, and control potential events or situations, and to provide reasonable assurance regarding the achievement of objectives. Without accurate information about IM/IT project related risks, senior management does not have the ability to provide effective oversight which could impact the success of a project. The audit examined the risk management practices in place at the project level.

A number of sample projects did not have a risk management plan.

The audit team expected to find a risk management plan for each sample project. The purpose of a risk management plan is to formally document how risks will be managed, monitored and escalated within a specific project. A risk management plan is essential to help ensure the effective management of risk and the successful completion of a project. A risk management plan template has recently been established by TIMSD to assist project managers with this task and is available on the TC intranet.

Each IM/IT project included in the audit sample was examined to determine if a risk management plan had been prepared. It was found that seven of the projects sampled did not establish a risk management plan.

The majority of projects did not maintain up to date risk logs with sufficiently detailed risk information.

The audit team expected to find that each project had established a risk log with adequately detailed risks and evidence that risks were addressed on a timely basis. Each sample project was examined to determine if a risk log with sufficient details was established and updated on an ongoing basis. The risk logs were also examined to determine if the risks identified in the PAD and PCRA were incorporated into the risk logs. It was found that 10 projects sampled did not maintain active risk logs with sufficiently detailed risks. In addition, it was observed some projects described very generic risks within the risk logs such as "scope creep" or "project schedule", limiting its utility to mange project risk.

The majority of project sponsors had not established a plan to address changes during the life of the projects.

The audit team expected that each project sponsor would have documented a process to manage project and system-related changes. However, auditors found that 11 of the 19 projects had not developed such a plan. These plans help to ensure that changes during the project life cycle are appropriately managed and the authority to make project changes is documented. Generally, authority is delegated to the project manager to make decisions regarding items that do not affect the project schedule, scope or budget. Authority regarding changes that could impact the project schedule, scope or budget is usually retained by the steering committee (note: at TC, scope changes and budget increases may require additional approvals). Without a plan in place to manage changes within a project there is increased risk of unauthorized or inappropriate changes to the project scope, resulting in schedule delays and/or cost overruns.

3.2 BUSINESS REQUIREMENTS

Business requirements for half the sample projects had not been documented.

Well documented business requirements are critical to the success of an IT project; in simple terms, they define what needs be delivered. The development of business requirements cannot be delegated to IT technical staff as it is the business application owner and users, not the technical staff, who have this knowledge. How a project will be delivered requires technical knowledge, but this question cannot be answered until business requirements are clearly documented.

A lack of clearly defined business requirements for an IM/IT application project increases the risk that required functionality will not be incorporated into the design of the application software with the result that the software will not meet the organization's needs. A lack of clearly defined business requirements also increases the risk that unnecessary functionality is built into the software which could result in schedule delays and/or cost overruns.

The audit team expected to find evidence of a formal process to assess, document and manage user requirements and expectations; however, this was not the case. Nine of the sample projects did not have adequately documented project business requirements. Guidance on how to define and document business requirements is not well developed in the Department's IM/IT project management control framework. During audit interviews with project managers, it was noted that the importance of business requirements was not well-understood and inexperienced project managers who were managing the initial stages of projects tended to have inadequately documented business requirements, which may have contributed to significant project delays.

3.3 PROJECT MANAGEMENT CONTROLS

3.3.1 Role of the Project Manager

There is no process to ensure IM/IT project managers have the necessary skills or experience to successfully manage projects.

The audit team expected the Department would have a process to ensure that project managers being considered for particular projects had the necessary competencies/experience. However, it was noted through audit interviews that inexperienced individuals had been given responsibility for managing some large and complex projects. It was also observed that some project managers were not fulfilling the responsibilities that TB policy has outlined for day-to-day managers of IM/IT projects. For example, some TC employees had been identified as project managers although no adjustment had been made to the responsibilities of their regular full-time position, with the result that project management responsibilities were being further delegated. This transfer of responsibilities had not been formally noted in the project documentation. The audit team was also advised by a number of project sponsors that it had often been difficult and time consuming finding qualified and experienced project managers.

3.3.2 Project Documentation

Project management documentation for a number of projects was inadequate.

Good project management practices and supporting documentation facilitate project oversight, help reduce risks, and help ensure projects are completed on time, within budget, and with expected benefits. The audit examined the adequacy of project charters, plans, and spending/budget plans.

Project Charter

The audit team expected to find a project charter for each sample project. The purpose of a project charter is to give a comprehensive project overview, ensuring stakeholders have agreed on such major aspects as objectives, roles and responsiblities, scope, deliverables, and required resources. Without a project charter, there is an increased risk that project team members and stakeholders will fail to coordinate and align their activities, increasing the risk of inappropriate decisions and costly delays.

The audit found that seven of the 19 sampled IM/IT projects did not have a project charter. Additionally, as the use of a standard project charter template was not required, the content and level of detail varied between the 12 charters that were in place.

Project Plan

Departmental policy requires every project to have a project plan approved by the project steering committee. The plan should include a project definition, objectives, responsibilities and baseline scope, cost and schedule information, expanding on information in the project charter. The audit team expected that each sample project would have an approved project plan.

However, from interviews with project managers, the auditor team found the majority of managers were unclear on project plan requirements and were not aware that project plans require steering committee approval. Only two of the 19 projects in the sample had approved project plans.

Spending Plans

The audit team expected to find a spending/budget plan for each of the sample projects and that it was updated periodically for the life of the project. The purpose of a spending plan is to ensure that costs are monitored and managed, reducing the likelihood of cost overruns. However, 11 of the 19 projects examined did not have a sufficiently detailed spending plan that was regularly updated. It was observed that spending plans varied considerably in content and detail, for example, some had only the total annual costs from the PAD, rather than monthly breakdowns. Also, the frequency of updating varied.

Project Documentation

To facilitate the exchange of project information there is a requirement that project documentation must conform to the Macroscope Management Suite naming standard. It was found from the documentation review that this requirement is not being followed.

The Department's Information Management Policy says all information of business value is to be stored in the Records, Document and Information Management System (RDIMS), but the audit found that this practice was not being followed for all projects.

3.4 PROJECT IMPLEMENTATION

Comprehensive training plans were not prepared for the completed projects.

The audit team expected to find that sponsors or managers of completed application projects would have developed comprehensive training plans (i.e., identified user groups and their training needs, developed training schedules and training methods, and identified associated costs). A training plan is important because benefits of an application are unlikely to be fully realized without adequate user training. In addition, a comprehensive training plan can reduce productivity losses or down time associated with the transition from one system to another.

It was observed that three of the four completed application projects had evidence of training materials and user guides, but none had a comprehensive training plan.

Communication plans had not been prepared for several projects.

The audit team expected to find communication plans in place for all the sample projects. Good communication during the project life cycle has two main benefits: it keeps stakeholders up-to-date on the project's status and it facilitates consensus and ownership of major project decisions and milestones. Poor communication can lead to misunderstanding and uncertainty which in turn can lead to delays. A communication plan helps define: roles and responsibilities, target audiences, and key messages. TIMSD has created a project communication template to help project sponsors/managers develop communication plans. Eight of the 19 sample projects had not created communication plans.

4. FINDINGS: PROJECT MONITORING

Overall, the Department's monitoring framework has gaps, is only partly implemented, and is not well understood.

The POS is composed of three staff. Prior to the establishment of the POS in April 2009 by TIMSD, very limited monitoring practices were in place. A primary role of the POS is to assist the IM/IT governance committees to monitor progress of IM/IT projects. The POS compiles and summarizes self-assessed status reports received from project sponsors. The POS also plays a challenge function with respect to the information provided in the status reports; however, given the resources of the group and the large number of on-going projects their ability to play a robust challenge role is often limited.

The audit team expected to find a monitoring and reporting regime that was aligned to the Transport Canada Monitoring and Measurement Framework For Departmental IM/IT Investments. However, based on a comparison of the framework with current practices it was found that not all aspects of the framework have been fully implemented. In audit interviews with finance staff, project sponsors, project managers and members of governance committees, it was found that many were unfamiliar with the framework and its requirements.

Project sponsors are required to provide the self-assessed project status reports three times per year. Self-assessments are to be based on six performance indicators: scope, schedule, budget, risk, operational requirements, and business fit. Reports are also to include references to important project documents, e.g., project charters.

When the POS has received all required status reports, a summary report is prepared. The summary is provided to the Chair of the Investment Committee. Based on the Framework, the Investment Committee reviews the summary report and gives its comments to the Business IM/IT Council. Following the review by the Business IM/IT Council, a further summarized version is to be provided to TMX.

The audit reviewed the IM/IT self-assessed status reports from project sponsors and the summary reports produced in 2010/11. It was found that all of the self-assessed project reports had been submitted and the summaries had been prepared by the POS as required. However, the following deficiencies were noted:

The status of some performance indicators was missing from status reports submitted to the POS, although they had been signed by project sponsors.
Performance indicators on some self-assessments were shown as green, although supporting documentation had not been completed. For example, some reports showed the Risk indicator as green although there was no risk management plan or risk log in place.
Scope and schedule changes were not documented. For example, if the schedule had changed, the assessment was based on the revised schedule, and there was no reference to the previous one.

A review of the decision records for the Business IM/IT Council and the Investment Committee found that the monitoring of projects underway (i.e. review of status and summary reports) was noted only once a year instead of quarterly as indicated in the Framework and there were no related decisions. A review of the RMC decision records found that the status of IM/IT projects was included a few times over the course of a year including some projects indicated "requiring attention". A review of TMX decision records found no record of TMX performing monitoring activities with respect to IM/IT projects. (See Appendix C –Sample Projects – Timelines).

As well, the Department has no requirement for independent third-party progress reviews, even for very large/complex projects, to assist senior management in their oversight role and provide independent confirmation that projects are on track with respect to scope, schedule and costs.

Once funding and PAD approval is obtained an IM/IT project is essentially funded in its entirety.

At TC, funding is provided for IM/IT projects each fiscal year based on the cost estimates in the PAD document through TMX Initial Budget Delegation funding approval. There is no gating process that ties funding to key milestones/deliverables, which would help ensure project costs, scope and schedule stay on track and projects stay aligned to the Department's needs.

TBS developed A Guide to Project Gating for IT-Enabled Projects to encourage departments to coordinate senior management scrutiny of large projects at key points and to possibly link these reviews to funding approvals. The TBS guide includes a full gating model, with seven gates, a streamlined five-gate model for medium-size projects, and a three-gate model for smaller, low-risk projects.

Once a project has been approved by TMX, the Department has no process to put a project that has not complied with departmental IM/IT policies on hold.

The audit team found that there is no documented process with clear criteria and escalation procedures which describes how TIMSD, IM/IT governance committees or even TMX would put a project on hold once its funding has been approved.

Financial controls are not adequate to reasonably ensure IM/IT project costs are accurately reflected within the financial statements of the Department.

As an IM/IT project progresses through its life cycle, associated costs are charged against a specific project code within the Work-in-Progress (WIP) asset account in the departmental financial accounting system. The WIP account is essentially a holding account while the project is under development (reported within the Asset section of the Balance Sheet). Upon completion of the project and when the asset is put in use, the costs are transferred from the WIP account to a Capital Asset account and depreciation expense is charged against the asset over the estimated useful life. During the life of a project the costs that have accumulated within the WIP account are required to be reviewed on a regular basis to ensure that the costs have been charged to the correct project codes and costs meet the definition of a capital cost. In addition, this review must ensure that completed and cancelled projects are removed from the WIP account in a timely manner. If project costs are not accurately recorded within the financial accounting system, material errors within the financial statements (most likely an overstatement of assets and an understatement of expenses) may occur.

Although the Department's period-end procedures include a requirement for an analysis of the WIP account, interviews with Finance staff indicated that this task is not performed consistently across the Department. Completed and/or cancelled projects are not always communicated to the Finance group on a timely basis, which impacts the accuracy of the analysis. As well, the status of some projects is unclear which further impacts the accuracy of the analysis. For example, one of the sampled projects that had been identified as "cancelled" was later classified as "deferred". The distinction between "cancelled" and "deferred" is very important from a financial reporting perspective. A deferred project could remain in the WIP account (for a reasonable amount of time); however, a cancelled project has no future value and therefore must be removed and expensed.

There may also be a risk of IM/IT capital costs being incorrectly allocated to IM/IT maintenance costs. A common financial control to mitigate this risk is the review of IM/IT maintenance expenses over a predetermined value. Based on audit interviews with Finance staff, there are no formal financial controls, such as the analysis of IM/IT maintenance costs, in place to ensure IM/IT capital costs are not expensed in error.

5. FINDINGS: POST-PROJECT REVIEW

Assessment and follow-up/reporting on completed projects is not in place.

The audit team expected to find robust post project review practices in place necessary to measure the value of IM/IT projects to the Department. According to the IT Governance Institute, the basic principles of IT value delivery are: on schedule, within budget and of appropriate quality (including original benefits promised). Formal measurement of these principles is imperative for ensuring value for IT investments.

The Transport Canada Monitoring and Measurement Framework For Departmental IM/IT Investments requires the completion of an IM/IT Results Measurement Report within 6-12 months of the end of a project. A template has been established for the report. The report requires the assessment of 4 criteria: cost savings, budget, client satisfaction and quality. There is currently no requirement in the template to assess performance against the project schedule. In addition, the quality measure does not include an assessment of the achievement of expected benefits.

Our audit program was to include an examination of the IM/IT Results Measurement Report for the completed sample projects; however, no reports had been prepared for the completed projects. There is currently no process in place to ensure the completion of the report and to utilize the information to measure success of IM/IT projects.

Through interviews with Corporate Finance staff it was noted that as of 2010, PADs require measureable benefits expected from the IM/IT project to be clearly defined. However, processes for documenting and assessing the achievement of the benefits upon the completion of a project and for providing relevant information to the governance committees including TMX have not been established. IM/IT projects started prior to 2010 are not required to retroactively define the expected benefits to be delivered. Without an effective process in place to assess the achievement of benefits the Department will not be in a position to assess value for money of IM/IT project investments.

6. CONCLUSIONS

The result is a very significant risk that the Department will not have its IM/IT needs met and will not receive good value for its IM/IT investments.

7. RECOMMENDATIONS AND MANAGEMENT ACTION PLAN

Based on the findings, we have developed the following six major recommendations. It is important to note that the successful implementation of the following recommendations will be dependent on the full collaboration and support of TMX. TMX members are responsible and accountable for the success of their IM/IT projects.

The Deputy Minister should ensure there is an approved IM/IT plan for the Department, that this plan is updated regularly, and that there are regular reviews of performance against the plan.
ADM, Corporate Services, should provide TMX annually a list of all recommended IM/IT project proposals with sufficient information (e.g. whether or not the proposed project can be obtained at a lower cost through adaptation of an existing application) to allow for informed decisions and oversight.
ADM, Corporate Services supported by the CIO and the DG, Finance and Administration, should review the control framework with the view to making it workable by streamlining it and addressing fundamental gaps. In particular, the project management framework should be scalable to take account of project size, nature, complexity and risk.
ADM, Corporate Services, supported by the CIO, should significantly strengthen monitoring of and reporting on IM/IT projects and, effective immediately, report bi-annually to TMX and, when significant risks are identified report on these more frequently. Progress reports should identify all scope, schedule and budget changes and significant issues/risks.
ADM, Corporate Services, supported by the CIO, should ensure that each IM/IT project proposal to TMX has clear measurable outcomes, that there is a post-project review of every IM/IT project, and that findings of post-project reviews are provided to TMX.
ADM, Corporate Services, should ensure controls are sufficient to provide reasonable assurance of the accuracy of the IM/IT project costs within its financial statements.

#	Recommendation	Detailed Management Action Plan	Completion Date (for each action)	OPI direct report for each specific action
1	The Deputy Minister should ensure there is an approved IM/IT plan for the Department, that this plan is updated regularly, and that there are regular reviews of performance against the plan.	Continue to engage in active business planning to support the DM and TMX with in strategic decision making through the following actions:		ADM, Corporate Services
		A. Present an annual Strategic and Investment Plan for TMX approval as part of each year’s Initial Budget Delegation and Integrated Business Planning Exercise.	Completed March 2012 for FY 2012/13 Ongoing	CIO/DG TIMSD
		B. Provide TMX with a semi-annual update of the IM/IT Strategic and Investment Plan to report performance against the plan.	November 2012	CIO/DG TIMSD
2.	The ADM, Corporate Services, should provide TMX annually a list of all recommended IM/IT project proposals with sufficient information (e.g. whether or not the proposed project can obtained at a lower cost through adaptation of an existing application) to allow for informed decisions and oversight.	Improve support of senior management decision making through the following actions:		ADM, Corporate Services
		A. As part of the Integrated Business Planning Exercise at Initial Budget Delegation, and within the Strategic and Investment Plan, present to TMX a list of all new recommended IM/IT projects with the appropriate investment justification.	Completed March 2012 for FY 2012/13 To be enhanced in the Jan – March 2013 timeframe	DG, Finance and Administration
		B. Update the IM/IT Project Proposal Template to align with the corporate ranking criteria used to support decision making across all capital strategies. The revised IM/IT Project Proposal will have a refocus on the business value of the project.	Completed for 2012/13 IBD. A Revised IM/IT Project Proposal Template to reflect lessons learned from the 2012-13 IBD process by: 2012-06-30	CIO/ DG TIMSD
		C. Develop a comprehensive IM/IT Project Approval Document Review Checklist as a tool for Project Sponsors and IM/IT Functional Reviewers to support the evidence found in the Project Approval Document ensuring all IM/IT and Project Management elements have been considered.	Completed: 2011-11-23	CIO/ DG TIMSD
3.	The ADM, Corporate Services supported by the CIO and the DG, Finance and Administration, should review the control framework with the view to making it workable by streamlining it and addressing fundamental gaps. In particular, the project management framework should be scalable to take account of project size, nature, complexity and risk.	Evolve the IM/IT Control Framework to streamline documentation and improve processes to provide clarity and clear direction to support improved project management with the following actions:		ADM, Corporate Services
		A. Conduct a review of the framework to ensure alignment between all levels of Control Documentation (TBS, TC Financial Policy, and Strategy Direction/Frameworks), avoid duplication, simplify and streamline while also enhancing documentation to address gaps identified in the audit.	2012-12-31	DG, Finance and Administration
		B. Develop and communicate simplified tools to support Project Sponsors and Project Managers throughout the investment lifecycle with the goal of increasing understanding and promoting an efficient IM/IT investment process.	Completed: 2011-11-02	CIO/ DG TIMSD
		C. Evolve the current IM/IT Project Control Framework documents into a comprehensive IM/IT Project Management Framework providing a single reference document to describe the activities and documentation required at each stage of managing an IM/IT project. This reference document will serve as a guide to ensure that the appropriate level of governance and management is applied and can be scalable depending on the size and complexity of the project.	Draft Completed 2012-03-31 To be finalized by: 2012-07-31	CIO/ DG TIMSD
		D. The Transport Canada Directive for System Development Life Cycle (SDLC) Methodology and Documentation was published with an effective date of April 1, 2012. This Directive was developed through consultation with departmental stakeholders in various forums. It will serve to strengthen the overall management of departmental IM/IT application development projects. Distribution to all IM/IT governance committees and communities was completed by May 17, 2012.	Completed: 2012-05-17	CIO/ DG TIMSD
		E. Clearly communicate the IM/IT Project Management Framework for all IM/IT projects in 2012-13 by developing an executive summary deck to provide Project Sponsors, Managers, Team Members and Governance Committee Members with information on shared accountabilities related to the management of projects and outcomes / results. This deck will replace the existing decks created and shared with Project Managers and Sponsors.	Draft Completed by 2012-05-31 Finalized by: 2012-07-15	CIO/ DG TIMSD
4	The ADM, Corporate Services, supported by the CIO, should significantly strengthen monitoring of and reporting on IM/IT projects and, effective immediately, report bi-annually to TMX and, when significant risks are identified report on these more frequently. Progress reports should identify all scope, schedule and budget changes and significant issues/risks.	Continue to strengthen the IM/IT project monitoring process and documentation to support the DM and TMX in strategic decision making through the following actions:		ADM, Corporate Services
		A. Present TMX with tri-annual dashboards on the status of all IM/IT projects: synchronized with the departmental financial reporting cycle. These reports will identify changes to budget, scope, schedule, risks and issues as well as mitigation actions.	May 2012 – Completed for P12 (March) November 2012 – to be completed for P6 (September) February 2013 – to be completed for P9 (December) Ongoing – cycle is repeated	CIO/ DG TIMSD
		B. Undertake a review of the departmental IM/IT Governance and Terms of References for all related committees. This will include a review of the roles, responsibilities and accountabilities for all departmental IM/IT governance stakeholders. This review needs to take into consideration the most recent organizational and Strategic Outcome structures and make adjustments where required to these structures. This will result in a renewed and refocused IM/IT Governance structure for Transport Canada and will expand the project awareness and responsibility across a broader, horizontal base of stakeholders.	2012-12-28	CIO/ DG TIMSD
5.	ADM, Corporate Services, supported by the CIO, should ensure that each IM/IT project proposal to TMX has clear measurable outcomes, that there is a post-project review of every IM/IT project, and that findings of post-project reviews are provided to TMX.	Increase the focus of performance measurement to the project management community; support Project Sponsors in the development of clear and realistic measureable outcomes; and provide evidence of post-project reviews for presentation to TMX :		ADM, Corporate Services
		A. Update the Project Approval Document’s Performance Outcome Measurement instructions to support Project Sponsors in developing measurable outcomes at the appropriate level of detail.	2012-12-31	DG, Finance and Administration
		B. Include a formal project close out process and incorporate the associated activities in a mandatory Closeout Report.	Completed: 2011-06-14	CIO/ DG TIMSD
		C. Include in the IM/IT Project Approval Document Review Checklist a section for Performance Measurement to assist Project Approval Document Reviewers to ensure outcomes are clear, realistic and measurable.	Completed: 2011-11-23	CIO/ DG TIMSD
		D. Develop an executive summary report of business performance measurement outcomes, for each completed project. Goal of executive summary is to demonstrate the value of the investment to TMX (through face-face discussion as required) on projects once they have formally been closed out.	2013-03-31	CIO/ DG TIMSD
6.	The ADM, Corporate Services, should ensure controls are sufficient to provide reasonable assurance of the accuracy of the IM/IT project costs within its financial statements.	Controls to provide reasonable assurance of the accuracy of IM/IT project costs will be provided through the following actions:		ADM, Corporate Services
		A. Request quarterly confirmation of Work-in-Progress (WIP) project status by FMAs/RCFAs to ensure timely monitoring, reconciling and clearing of WIP projects.	Complete (commenced September 2011)	DG, Finance and Administration
		B. Implement quarterly review of Project Monitoring Reports and review of Project Close Out Reports.	Complete (commenced October 2011)	DG, Finance and Administration
		C. Conduct detailed analysis, reconciliation and clearing of the WIP Opening Balances Clearing Account for the 2011-12 financial statements (one-time clean-up).	Complete April 20, 2012	DG, Finance and Administration
		D. Annual review of Repairs and Maintenance account as part of the year-end procedures, to identify any costs that should have been capitalized.	Complete May 15, 2012	DG, Finance and Administration

List of Appendices

APPENDICES HAVE BEEN REMOVED AND ARE AVAILABLE UPON REQUEST

Appendix A - IM/IT Projects Underway as of February, 2011

Appendix B - Sample IM/IT Projects

Appendix C - Sample Projects - Timelines

Appendix D - Control Framework Documentation

Appendix E - Listing of Audit Criteria

Footnote 1 Software projects may have infrastructure components managed by TIMSD on a cost recovery basis. (Return to footnote 1 source paragraph)

Footnote 2 ITravel, one of the completed sample projects was part of a larger project, Oracle Release 12/Fusion ERP Upgrade. Costs for ITravel were not tracked separately therefore the completed costs do not include the ITravel project costs. (Return to footnote 2 source paragraph)

Footnote 3 Page 57 of the Transport Canada Investment Plan 2010-11 to 2014-15 (RDIMS #5699338) (Return to footnote 3 source paragraph)

The following document is available for downloading or viewing:

Audit of IM/IT Project Life Cycle Controls ( PDF Version, 296 kb )

To access the Portable Document Format ( PDF ) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet:

Language selection

WxT Language switcher