| Print Version |
Table of Contents
Executive Summary
Introduction
Marine Safety's mandate is to protect life, health, property and the marine environment in the context of an efficient and sustainable marine transportation system worthy of public confidence. The directorate is currently undergoing significant change, with the merging of Marine Security in order to create the Marine Safety and Security directorate. The IT section under Marine Safety is also being moved to the Technology and Information Management Services Directorate (TIMSD) under the Corporate Services group.
The Marine Safety Core Architecture (MSCA) project was selected for this audit. Its objective is to integrate Marine Safety's key applications into one single infrastructure and consolidate data in order to improve its integrity, sustainability, consistency and accountability. This included integrating marine inspection reporting systems, time activity systems and certification systems under one infrastructure. However, as a result of a variety of factors including changing regulations and data security requirements, many of the deliverables under the MSCA project have undergone significant changes and have not been completed as scheduled.
Audit Objective and Scope
The overall objective of the audit was to assess the adequacy and effectiveness of controls in place to support the successful completion of the MSCA project. The audit was to also identify lessons learned and best practices related to developing IT applications to allow Marine Safety to make real-time adjustments whenever possible to other IT projects underway.
Conclusion
The audit found the control framework and related practices for the MSCA project need significant improvement to ensure its successful completion. Oversight of the project has been lacking, standard project management processes have not been consistently followed and there is no comprehensive plan to complete the project. It should be noted that the audit team was provided with evidence during the audit to demonstrate that work is underway by both the Department's Technology and Information Management Services (TIMSD) and Marine Safety area to the improve policies, procedures and practices around the governance and management of IT projects within the Department.
Statement of assurance/reliance
It is our professional judgment that the audit has been conducted in accordance with the Internal Auditing Standards of the Government of Canada. Satisfactory audit procedures have been conducted, and sufficient relevant evidence has been gathered to support the accuracy of the opinions provided in this report.
Signatures
Signed by
Dave Leach (CIA) Director, Audit and Advisory Services
September 28, 2012
Date
Signed by
Laura Ruzzier, Chief Audit Executive
September 28, 2012
Date
1. Introduction
1.1 Purpose
The System Under Development (SUD) Audit of the Marine Safety Core Architecture Project was included in the departmental 2011/12 audit plan due to the high degree of inherent risk associated with system development initiatives. In addition, the 2011 Audit of IM / IT Project Life Cycle Controls identified significant weaknesses in the Department's control framework for managing IM / IT projects.
The Marine Safety Core Architecture II (MSCA) project was selected based on a number of factors including: the total estimated cost of the project; the significance of the system to the Department, and the fact that it was being developed by a team composed of internal and contracted resources managed by Marine Safety (i.e. the system was not an off-the-shelf product).
1.2 Background
1.2.1 Marine Safety Directorate
Marine Safety, part of the Department's Safety and Security group, has a mandate to protect life, health, property and the marine environment. Marine Safety develops and administers a national program supported by approximately 620 employees. As part of its mandate, Marine Safety:
- develops, administers and enforces national and international laws and policies governing marine safety, and the protection of the marine environment;
- promotes safe practices and procedures;
- develops and maintains regulations, examinations, and training standards for the certification of seafarers - including issuing Certificates of Competency;
- responds to marine occupational safety and health issues;
- maintains a Canadian vessel registry;
- delivers an internal technical training program to the program's inspector community;
- delivers prevention-based programs to promote small vessel/recreational boating safety;
- conducts research in the marine transportation sector (e.g. safety equipment);
- administers the navigable waters protection program; and
- oversees pilotage matters.
A Marine Safety Application Management Unit has been established and tasked with the responsibility for the design, development, implementation and enhancements of all Marine Safety software applications, as well as application management (including maintenance and support) services. This unit currently has five full-time employees. The unit also employs external information technology (IT) developers and project managers through contracts. The Director General, Marine Safety is the Project Sponsor for all Marine Safety IT projects, including MSCA .
The Marine Safety Directorate is currently in the process of a major reorganization. The Directorate is merging with the Marine Security Directorate to form a single Marine Safety and Security Directorate Footnote 1. This reorganization is occurring at the same time as a number of significant legislative and regulatory changes are being introduced such as further delegation of ship inspection activities to third parties. As well, as part of the Department's recent decision to consolidate IM/ IT application development, the Marine Safety Application Management unit and its resources will be transferred to the Technology and Information Management Services Directorate (TIMSD) within Corporate Services. The full impact of these changes on the development of Marine related IT projects are unknown.
1.2.2 Marine Safety Core Architecture Project
The objective of the MSCA project is to integrate Marine Safety's key applications into a single infrastructure and consolidate data in order to improve data integrity, sustainability, consistency and accountability. MSCA includes national applications such as inspection reporting systems, time activity reporting systems and ship registry systems. This new infrastructure will also enable Marine Safety to meet the new Canada Shipping Act (CSA) 2001 regulatory requirements and to re-engineer the applications included in the scope of the project to meet departmental technical standards.
The proposed project plan, as described in the Project Approval Document (PAD) approved March 31st, 2009, was to integrate seven of Marine Safety's applications. The project was to start in March 2009 and be completed by October 2012, with a total estimated cost of $3.6M (not including taxes). Initial approval was given to spend $920K to complete the planning phase of the project.
In March of 2010, upon completion of the planning phase, an updated PAD was submitted for approval. The purpose of the second PAD was twofold: obtain approval to move to the development phase of the project and to expand the project's scope to include four additional applications (bringing the total applications under the scope of the MSCA project to 11). The updated total estimated cost of the MSCA project was $4.4M (not including taxes) and its new completion date was October 2013.
Based on the approved project plan, by May 31, 2012, the project should have spent approximately $3.2M and eight of the 11 applications should have been completed and put into production. The remaining three applications ( SVCP , NTARS , and SIRS ) were scheduled for completion by December 2012.
As of May 31, 2012, $3.6M was expended on the project, rather than the planned $3.2M, and only one application was completed and put into production. Three of the 11 applications were cancelled. Of the remaining seven applications, at the time of the audit, three are nearing completion; however, they are not scheduled to be put into production until March 31, 2013. Four applications are in various stages of development with no specific target dates for completion. The following table lists the 11 applications, their status and the amount spent. Appendix A provides a brief description of each.
| Application Name | Status | Estimated Cost | $ Spent (as of May 2012) Footnote 2 |
|---|---|---|---|
| * Denotes the four projects added under MSCA in May 2010. | |||
| Ship Registry Computer System (SRCS) | Complete and in Production | $340,000 | $397,673 |
| Small Commercial Vessel Registry System (SCVRS) | In progress | ||
| Port State Control System (CPSCS)* | In progress | $424,400 | $619,628 |
| Small Vessel Compliance Program (SVCP) | In progress | $370,000 | $15,000 |
| Boat Identification and Safety System (BIASS)* | In progress | $85,000 | $92,224 |
| Certificates of Competency Issuance System (CCIS) | In progress (estimated completion Mar/13) | $300,000 | $1,108,700 |
| Small Vessel Operating Proficiency (SVOP)* | In progress (estimated completion Mar/13) | $265,000 | |
| Fishing Seasons (FISH7)* | In progress (estimated completion Mar/13) | $225,000 | |
| National Time Activity Reporting System (NTARS) | Cancelled | $75,000 | $14,622 |
| Automated Certification & Examination System (ACES) | Cancelled | $300,000 | $0 |
| Ship Inspection Reporting System (SIRS) | Cancelled | $620,000 | $113,725 |
| MSCA Architecture | In progress | $952,900 | $1,200,603 |
| Contingency Funds | $464,600 | ||
| TOTAL | $4,421,900 | $3,562,175 | |
For the seven applications currently underway a number of factors have contributed to delays such as changes in regulations, security requirements, reductions in operating funding (required to support applications once they are completed), and lack of Marine Safety staff to develop and test applications.
Marine Safety has indicated that there are no further funds available for the MSCA project as the unspent funds (approximately $700K, reduced from $900K due to budget deficit reduction requirements) have been removed from the project. Marine Safety has reported that this was done in anticipation that deliverables and unspent funds from the MSCA project would be moved to other IM / IT projects in the Marine Safety Directorate. The other projects are CMOS/MSID and MPDIS. However, the transfer of MSCA deliverables to these projects has yet to be approved.
The CMOS/MSID project was started in 2009 and its original purpose was to map, record and track the process of the accreditation of third parties to inspect and certify vessels on behalf of the Minister of Transport. It is also intended to streamline reporting of Marine Safety data based on the consolidation of the applications under the MSCA project. Marine Safety is now in the process of requesting approval to significantly change the direction and scope of this project. The purpose of the CMOS/MSID project is now to support the Marine Safety's revenue generation strategy through the development of the Marine Safety Dispatch Tracking System application. The application will assist in the introduction of a new fee structure and provide efficiencies in revenue monitoring and collection. Three of the applications ( SVCP , NTARS , and SIRS ) that were cancelled under MSCA are planned to be completed under the CMOS/MSID project. One option currently under consideration for the completion of the CMOS/MSID project is the adoption of the US Coast Guard's Marine Information for Safety and Law Enforcement (MISLE) system. The MPDIS project started in June 2011. Its purpose is to consolidate the examination and certification issuance process into one system. Three of the applications under the MSCA project (FISH7, SVOP and CCIS) were moved to the MPDIS project due to data security requirements although this transfer of deliverables was not formally approved. Neither the CMOS/MSID nor MPDIS project has completed its original deliverables.
1.3 Objectives, Scope and Approach
The overall objective of the audit was to assess the adequacy and effectiveness of controls in place to support the successful completion of the MSCA project. The audit was to also identify lessons learned and best practices related to developing IT applications to allow Marine Safety to make real-time adjustments whenever possible to other IT projects underway.
The audit planning phase was used to gain an understanding of the MSCA project through preliminary interviews and documentation review, and to identify areas of greatest risk to target audit testing. Audit planning and scoping was very challenging for the audit team given the number of applications within the MSCA project and the changes that occurred to the applications during the life of the project; the majority of which were undocumented. The planning phase evaluated four risk areas generally associated with systems under development: governance risk, business risk, project risk, and technology/infrastructure risk.
To assess the effectiveness of the controls in place for the MSCA project, the audit team interviewed Marine Safety (including application developers and users) and TIMSD staff; reviewed available project documentation; and verified the status of each application.
1.4 Criteria
Based on the overall audit objective we expected to find the following:
- Governance - a management control framework is in place to ensure roles, responsibilities and authorities within which the project operates, and within which all major decisions concerning the scope and objectives, are well defined.
- Business – processes have been defined to successfully achieve the business solutions identified in the project's business case.
- Project - project management practices are aligned with industry best practices and applicable Treasury Board and departmental policies.
- Technology - the technology platforms chosen supports the business solution and the organization has defined plans to deal with the new technology.
A detailed list of criteria can be found in Appendix B.
1.5 Structure of Report
Audit findings are reported for each of the four risk areas: Governance Risk, Business Risk, Project Risk, and Technology/Infrastructure Risk. Conclusions and recommendations to address control weaknesses and gaps described in the findings section are provided in the Conclusions and Recommendations section. This section also includes a Management Response and Action Plan (MRAP) from the Department. The MRAP gives management's response to the audit recommendations, commitments and timelines for addressing identified weaknesses or gaps.
Appendix A provides a description of the MSCA applications and Appendix B describes detailed criteria for the audit.
2. Findings - Governance Risk
2.1 Senior Management Oversight
Senior management oversight of the MSCA project has been inadequate.
The 2009 approved project plan for the MSCA project stated that a MSCA steering committee would be established as part of the governance and oversight of the project; however, this committee was never established. In November 2011, the Marine Safety's Information Technology Steering Committee (ITSC) was established to provide oversight of all Marine Safety IT projects, including MSCA . Between May 2009 and November 2011 (the time period for which there was no steering committee), the project expended approximately $2.9M and significant changes were made to the MSCA project such as the cancellation of some of the applications as well as the movement of applications to other Marine Safety IT projects. Although the Transport Canada (TC) Business IM / IT Investment Operating Principles clearly state that the project steering committee must approve all changes to scope, schedule and cost, the changes that occurred within the MSCA were not documented or formally approved.
Although the ITSC has only recently been established and it is too early to make a definitive assessment on its effectiveness, there are indications that this committee, in its current form, may not be in a position to provide effective oversight to all Marine Safety's IT projects. Since its inception in November 2011, the ITSC has met three times and meetings were approximately one hour in length. Given that there are currently five relatively complex IT projects underway, as well as the organizational changes in the Marine Safety group, the frequency and length of the meetings does not appear to be adequate. In addition, the MSCA project sponsor, who is ultimately accountable for the success of the project, is not a member of the ITSC .
With the establishment of the ITSC , Marine Safety senior management is now receiving more information on the status of MSCA , including risks associated with the project; however, much of the information is communicated secretarially. In addition, some of the information reported to senior management does not accurately reflect the status of the project. In a review of the "Health" reports (i.e. status updates) sent to the ITSC , it was observed that the scope of the MSCA project was rated as "green" even though three of the original applications had been cancelled. In addition, the information contained in the reports makes it difficult for senior management to assess the progress of the project as the reports do not make reference to the progress of the project against the approved deliverables.
At the time of this audit TIMSD was in the process of strengthening TC's IM / IT Governance Model as a result of the recent internal audit of IM / IT Project Life Cycle Controls. The strengthened IM / IT Governance Model will describe the level of governance required for IT projects based on defined criteria, including the level of participation required for Steering Committees, Business Advisory Working Groups, and Technical Working Groups.
Marine Safety senior management did not have an accurate understanding of the status of the MSCA project.
In May 2012, a revised PAD was submitted to TIMSD for review. The purpose of the revised PAD was to obtain approval to close the MSCA project and transfer some of the outstanding deliverables and unspent funds to other MSCA projects. Although the PAD was reviewed and approved by the Project Sponsor prior to being submitted to TIMSD , it erroneously reported that eight of the applications had been completed when at the time none had actually been completed and put into production.
2.2 Project scope/change management
In some cases, the information provided in the PADs , as well as PAD instructions, is insufficient, making it difficult to assess project progress and results.
The audit team reviewed the two approved PADs for MSCA , as well as five other PADs (draft and approved) for MSCA , CMOS/MSID and MPDIS in order to gain an understanding of the projects. In a review of the approved MSCA PADs , some gaps and deficiencies were noted. For example:
- deliverables do not correspond to cost and schedule details making it very difficult, if not impossible, to monitor projects and assess the status of projects.
- expected benefits of the project were identified in various sections of the PAD , but the majority were not documented in a manner that would allow actual results to be assessed against expected benefits.
- the options analysis section of the MSCA PAD was limited. For example, Marine Safety reported that Commercial Off-The Shelf Products were not a viable option for the project, but no information was provided as to what research was conducted that led to this conclusion.
As part of the examination of the PADs , the audit team also reviewed the Guide to the Project Approval Document (PAD) Template and it was observed that there is currently no requirement to link deliverables or milestones to costs and schedules within a PAD . It was also observed that there is no specific requirement to document the research conducted to assess viable options for meeting the business requirements of a project. Expected benefits, documented in a manner that will allow for the assessment of the benefits upon the completion of the project became a PAD requirement in 2011, after the completion of the MSCA PADs .
Significant changes that have occurred within the MSCA project have not been appropriately approved through updated PADs .
As per TC 's Financial Policy and Procedures Manual, a change to a previously approved capital project requires a revised PAD to be drafted and submitted for approval. It is mandatory to resubmit a PAD when "the quality, capability, capacity or the scope of the project stages that had received departmental or Treasury Board approval is increased or reduced even though the original funding level may be unchanged. The significance of a change in project scope must be determined by the project sponsor and acted on accordingly."
The table below demonstrates the significant changes (including movement to other Marine Safety projects and cancelled applications) to approved deliverables of MSCA . In total, significant changes were made to seven of the 11 applications; however, none of these changes were appropriately approved through updated PADs. Marine Safety did start the process of revising the CMOS/MSID PAD at the beginning of 2012 to reflect the transfer of some of the deliverables from the MSCA project, but at the completion of this audit in August 2012, the document had not yet been approved.
| APPLICATION | MSCA | MPDIS | CMOS/MSID |
|---|---|---|---|
|
Small Commercial Vessel Registry System (SCVRS) |
|
|
|
|
Ship Registry Computer System (SRCS) |
|
|
|
|
Port State Control System (CPSCS) |
|
|
|
|
National Time Activity Reporting System (NTARS) |
|
|
|
|
Certificates of Competency Issuance System (CCIS) |
|
|
|
|
Automated Certification & Examination System (ACES ) |
|
|
|
|
Ship Inspection Reporting System (SIRS) |
|
|
|
|
Small Vessel Compliance Program (SVCP) |
|
|
|
|
Boat Identification and Safety System (BIASS ) |
|
|
|
|
Small Vessel Operating Proficiency (SVOP) |
|
|
|
|
FISH7 |
|
|
|
|
Legend |
|
|---|---|
|
|
Completed and in production |
|
|
Under development |
|
|
Cancelled under MSCA , planned for future development. |
|
|
Subject to approval, application is planned for development under another project. |
|
|
Application has been moved to a new project |
It should be noted that in November 2011, the Audit of IM / IT Life Cycle Controls found that the approval process for IT projects was difficult to follow. As a result, the Department was in the process of updating policies and procedures for the approval of IT projects during the same time period that Marine Safety was revising the CMOS/MSID PAD .
2.3 Achievement of project benefits
Since the MSCA project is not complete, an assessment of value for money and benefits achieved cannot be done; however, the audit did examine some of the processes in place to support an assessment of the benefits achieved and the success of the MSCA project.
There is no standard process for the assignment of project numbers which are used to monitor and manage IM / IT assets.
Project numbers are critical to monitor project progress, assess overall project results, and manage assets. The Department's Financial Management Advisors that support each of the Groups in the Department (i.e., advisors for Safety and Security, Policy, Programs, and Corporate Services Groups) are responsible for assigning project numbers to capital projects. Existing departmental policies do not provide clear guidance on how project numbers are to be assigned to projects and what constitutes a new project versus an expanded scope of an existing project. As a result, the use of project numbers is inconsistent.
The audit observed that for one of Marine Safety's projects, a project number was being proposed to be reused for a new project that had little connection to the original project. In addition, the original approved deliverables for the project were still outstanding. This practice makes it very difficult to track costs and to determine when a project is complete and in turn assess the overall benefits of the project.
The monitoring of IM / IT projects does not include verification that the project deliverables are complete and in production.
Although TIMSD has a well established review and approval process to migrate applications into production, they do not routinely use this information to monitor the status of an IT project described in PADs they are reviewing. When the audit team identified discrepancies between the PAD to close out the MSCA project and the information received regarding the status of the eight applications during interviews, they met with TIMSD to clarify the status of the applications. TIMSD stated that the submission of a PAD to "close out" a project is an unusual case. They further stated that since the PAD had been reviewed and approved by the project sponsor prior to being submitted to them they had not carried-out further review to verify the status of each application. Once the discrepancy was brought to their attention the status of the applications were verified by TIMSD 's unit responsible to migrate applications to TC 's production environment and it was confirmed that none of the applications had been completed when the PAD was submitted.
TIMSD 's Project Office Secretariat (POS) is responsible to ensure that the departmental IM / IT portfolio of investments is managed effectively throughout the entire investment lifecycle. This includes the collection and summarization of the monitoring reports completed for each individual IT project (currently 40+ projects). Project sponsors notify the POS when their project is complete; however, it is not standard practice to verify this information with the unit within TIMSD responsible for putting applications onto TC 's network.
3. Findings - Business Risk
3.1 Security Requirements
A project Threat Risk Assessment was not performed during the planning phase of the project.
Although the original project plan for MSCA references the need to complete a Threat Risk Assessment (TRA) during the planning phase of the project, a TRA was not completed until after the planning phase was completed.
The purpose of performing a TRA is to identify IT system security requirements/issues related to the project. If the results of a TRA indicate that additional work is required to satisfy security requirements, the scope, schedule and/or cost of the project could be impacted. Therefore, the TRA is generally completed during the planning phase of a project (after business requirements have been established).
The TRA for MSCA was finalized in March 2010, after the planning phase for MSCA had been completed. The TRA found that the entire database associated with the project would need to be classified as Protected B Footnote 3 due to the personal information tracked and maintained by four of the applications included in the MSCA project. The additional cost associated with this requirement was approximately $400K. As a result of the TRA , Marine Safety determined that it would be best to move these four applications to another Marine Safety project (MPDIS) which had already planned for a Protected B environment.
While the movement of the these four applications to the other project may have been an appropriate business decision, identifying this requirement after the PAD was prepared and submitted for approval led to delays with the MSCA project, and it is unclear if additional costs were incurred as a result.
Based on a review of the Department's IM / IT Project Management Framework, there is no direction or guidance on developing a TRA or when it should be completed.
3.2 Business Requirements
Business requirements for the MSCA project were managed informally.
Business requirements define what needs to be delivered and are the responsibility of the application owner, not technical staff. A lack of clearly defined business requirements increases the risk that required functionality will not be incorporated into the design of the application software, with the result that the application will not meet the organization's needs. A lack of clearly defined business requirements also increases the risk that unnecessary functionality is built into the software which could result in schedule delays and/or cost overruns.
The audit found that business requirements for the overall MSCA project were not documented and that the format and content of the business requirements for the individual applications varied significantly. A few of the applications did have detailed business requirements that were approved by the application owner; however, in other cases the only documented business requirements for the application were over 10 years ago and related back to the original development of the application. Some applications did not have any documented business requirements.
Deficiencies in the documentation of business requirements were also noted in the audit of IM / IT Project Life Cycle Controls. In response to the audit recommendations, TIMSD is in the process of making the formal documentation and approval of business requirements a mandatory requirement for all IT projects.
4. Findings - Project Risk
4.1 Development / Acquisition Process
Standard IT project management and development practices were not followed for the MSCA project.
Good project management practices and supporting documentation facilitate project oversight, help reduce risks, and help ensure projects are completed on time, within budget, and with expected benefits. For the MSCA project the audit found:
- reporting of actual expenses to budget estimates were not prepared on a regular basis;
- risk management practices were informal and risk logs were not updated on a regular basis;
- change management procedures to assess and approve scope changes for MSCA were not in place; and
- contracted resources were used for multiple IT projects within Marine Safety. In some cases contracts' Statement of Work did not clearly indicate the IT projects and/or deliverables the contracts were associated with, making it difficult to monitor the contract in relation to the IT projects.
It is important to note that the audit team was provided with evidence that improvements have recently been made to the project management practices within the MSCA project. For example, risk logs have been developed for some of the applications. A change management process has also been established and evidence was provided to support that changes to the MSCA project are following the process.
An important element of project management is determining key milestones for making funding decisions. Best practices as defined by COBIT and Treasury Board provide for a "gating" approach where management reviews progress, costs and schedule at key, pre-defined points in order to determine whether a project should continue. This best practice was not part of TC's IM / IT Project Management Framework at the time MSCA was started and was not in place for MSCA . TIMSD is now in the process of implementing a "gating" approach for all IM / IT projects.
A System Development Life Cycle (SDLC) is the process of developing a software application to meet specified business requirements. It covers many activities including why the application should be developed, the project feasibility, choosing the application design and architecture, implementing and testing it, up to delivering the system as a product to the user. The adoption of a standard SDLC methodology assists in the efficient use of resources, defines roles and responsibilities, facilitates information exchange and minimizes the risks with developing software applications.
Macroscope, a system development toolset that incorporates SDLC methodology, has been TC's standard for many years. Macroscope is large and complex and overly complicated for many small to medium-sized applications. TIMSD has acknowledged that the requirement to follow this standard was poorly communicated in the Department and adherence to the standard was not monitored. As part of the management action plan for the Audit of IM / IT Project Life Cycle Controls, TIMSD has recently developed a SDLC directive that states the requirement to utilize Macroscope and is in the process of scaling the Macroscope requirements to ensure it can be followed by projects of varying complexity.
5. Findings - Technology & Infrastructure Risk
5.1 Technology & Infrastructure Impact
There has been IT architecture issues with the MSCA project; however, the Department is in the process of enhancing related controls to prevent this issue from reoccurring in the future.
The development work on the underlying architecture for the MSCA project was started in 2009 and completed by August 2010, even though development work was not approved to be started until June of 2010 (after the updated PAD was approved). This architecture development work was completed by a contracted resource. Between August 2010 and January 2011, additional work was performed by Marine Safety IT staff to build on the architecture foundation created by the contractor. During this time, Marine Safety IT staff determined that the architecture that had been delivered by the contracted resource was excessively complex, highly customized and did not conform to departmental or industry recognized design methodologies. While the architecture was deemed suitable to support some of the applications identified under the MSCA project, it would not be a suitable long-term solution for the integration of all Marine Safety's key applications.
The audit team did inquire as to why the development work took place prior to when it was authorized and why the architecture was developed in a manner that did not support the overall objective of the project. However, there were no definitive answers to these questions since the project manager who was responsible for the MSCA project at the time, and oversaw the development work, retired from the Department in the summer of 2010. These decisions were also not sufficiently documented during the time of the original project manager.
Since 2010, the Department has implemented additional controls which will assist in preventing this type of situation from occurring in the future. For example, IT contracts must now undergo additional review and justification prior to be being put in place. In addition, TIMSD will soon release a revised IM / IT governance framework which will require the establishment of a technical working group, with TIMSD representation, as well as processes to enforce SDLC and project management requirements.
As the current architecture for the MSCA project has been deemed unsuitable for housing all identified key applications, Marine Safety is in the process of identifying alternative long-term solutions under the CMOS/MSID project. As previously noted, one option under consideration is the MISLE system from the US Coast Guard. In addition to a full and complete understanding of the business requirements of the Marine Safety group, a detailed assessment of the supporting MISLE technology is necessary to determine if this is the best option for Marine Safety. TIMSD has committed to working with Marine Safety on this assessment.
6. Conclusions
The audit found the control framework for the MSCA project needs significant improvement to ensure the successful completion of the project. Oversight of the project has been lacking, standard project management processes have not been consistently followed and there is no comprehensive plan to complete the project. It should be noted that the audit team was provided with evidence during the audit to demonstrate that work is underway by TIMSD and Marine Safety to improve policies, procedures and practices around the governance and management of IT projects within the Department.
7. Recommendations and Management Action Plan
| Recommendation | Detailed Management Action Plan | Completion Date (for each action) |
OPI direct report for each specific action |
|---|---|---|---|
|
Recommendation Rating - High ADM, Safety & Security should review all ongoing and planned IT projects within Marine Safety and Marine Security to assess alignment to business needs and priorities of the new Marine Safety and Security organization. The review should consider changes underway, changes to regulations, availability of resources (funds and staff), and the Department's application consolidation initiative. It should also conclude which projects should continue, which be put on hold or cancelled. The results of this review should be provided to the Deputy Minister for her approval. |
Marine Safety and Security will develop a Business Priority Mapping document to identify critical Marine Safety and Security IT projects and short term requirements that must be addressed in order to satisfy urgent business priorities. | March 2013 | DG, Marine Safety & Security and MSS executives |
| Once developed, the Business Priority Mapping document will be vetted by the Marine Safety and Security IT Steering Committee and Marine Safety and Security Executive. This document will take into account human and financial resource availability. Once vetted, it will be presented to ADM S&S and the Deputy Minister for approval. This document will serve to support Marine Safety and Security IM / IT investment requests for FYs 12/13 and 13/14. | DG, Marine Safety & Security and DG, Technology & Information Management Services and ADM, Safety & Security | ||
| ADM, Safety & Security will receive regular monthly status updates on all Marine Safety IT projects (including MSCA ) as a standing bilateral meeting item between ADM, Safety & Security and DG, Marine Safety & Security | ADM, Safety & Security and DG, Marine Safety & Security | ||
| ADM, Safety & Security will receive quarterly briefings on IT projects from DG, Marine Safety and Security and DG, Technology & Information Management Services. The briefing will cover: status update, financial situation, key deliverables (achieved and planned). | ADM S&S, DG, Marine Safety & Security and DG Technology & Information Management Services | ||
| Marine Safety and Security and TIMSD will establish an IM / IT Strategy / Investment Plan for Marine Safety and Security for the period FY 14/15 – FY 15/16. The Plan will integrate with the departmental IM / IT Strategic / Investment Plan. This document will ensure linkages to business plans and priorities. | ADM S&S, DG, Marine Safety & Security and DG, Technology & Information Management Services | ||
| ADM, Safety and Security and the CIO/DG, Technology and Information Management Services will present to the Deputy Minister for approval, on a bi-annual basis, an update on the Marine Safety and Security IT Strategy and its comprehensive IT Plans. This update will include status on IT projects and priorities, including those projects that should be continued, put on hold, or cancelled, resource requirements and key areas of risk. It will also include results realized / anticipated. | DM, Safety & Security and CIO/DG Technology & Information Management Services and Deputy Minister |
||
|
Recommendation Rating - Medium If it is determined that the MSCA should continue based on the review of all Marine Safety and Marine Security IT projects, the ADM, Safety and Security should develop a comprehensive plan to complete the MSCA project. This plan should include clear deliverables with associated costs, timelines, schedule, and responsibilities and be appropriately approved. A gating structure should be established so that formal reviews are conducted of the project progress, costs and schedule at key, pre-defined points, in order to determine whether a project should continue. |
Once the Business Mapping Document is tabled, if MSCA is confirmed as a priority, options will be developed for the completion of the remaining MSCA deliverables that are outstanding: CPSCS , Ballast Water, Small Vessel Manufacturers Monitoring System and the Small Commercial Vessel Registry based on business priorities. Options will include associated costs to complete and schedules for prioritization by Marine Safety and Security Executive and the MSS IT Steering Committee. This analysis will ensure linkages to business priorities. | January 2013 | DG, Marine Safety & Security and MSS Executive Committee |
| The ADM, Safety & Security will receive quarterly briefings on MSCA IT project update from DG Marine Safety and Security and DG Technology & Information Management Services. This briefing will cover: status update, financial situation, key deliverables (achieved and planned). (As mentioned in recommendation 1 above) |
ADM, Safety & Security DG, Marine Safety & Security and DG Technology & Information Management Services |
||
| The MSCA Project Approval Document will be updated with clear deliverables, identified gating structures and associated costs, timelines, schedule, and responsibilities for TC Business IM / IT Investment Committee and position as an investment for the FY 13/14- FY 14/15 cycle. | DG, Marine Safety & Security and DG Technology & Information Management Services and ADM, Safety & Security | ||
|
Recommendation Rating – Medium ADM, Safety and Security should strengthen the governance structure in place within Marine Safety and Security for IT projects to ensure there is effective oversight of the group's IT projects. This includes the consideration of the content and frequency of meetings, as well as membership of all governance bodies. As well the governance structure should align to TIMSD 's updated governance model for IT projects. |
Enhance the terms of reference of the MSS IT Steering Committee to include TIMSD and Finance senior management representation and strengthen the overall governance, project management and oversight elements of the MSS IT governance. | February 2013 | DG, Marine Safety & Security and DG Technology & Information Management Services and ADM, Safety & Security and Finance representative as required. |
| Establish the framework to support and create individual Project Steering Committees, technical working groups and Business Advisory working groups as required under the IM / IT Project Management Framework. | DG, Marine Safety & Security, MSS Executives and DG (or delegates) Technology & Information Management | ||
| Develop service level agreements to clearly define the roles, responsibilities, and service standards, and document all details of the transfer of IT Resources (FTE and OOC) from Marine Safety and Security to TIMSD . | DG, Marine Safety & Security, MSS Executives and DG Technology & Information Management | ||
|
Recommendation Rating – Medium The Department's IM / IT Project Management Framework is being updated and strengthened in response to recent internal audits on IM / IT Project Controls and IM / IT Procurement. The ADM, Corporate Services should ensure that weaknesses identified in this audit are also considered in the context of the revisions of the Framework currently underway, namely:
|
Update of the IM / IT Project Management Framework to include:
|
September, 2012 | CIO/DG, TIMSD |
|
Strengthen Project Approval Documentation (PAD) through the revision of the IM / IT Capital Project - Functional Review Checklist < RDIMS 7034057> to include improved IM / IT functional guidance and review in the following areas:
|
CIO/DG, TIMSD | ||
| Ensure a clear link between a completed capital project and the resulting asset being released into production through the revision of the IM / IT Project Closeout Report < RDIMS 6610467> to include a System Release Reference Table that will include Change Control Board references to all releases performed during the lifecycle of the project. | CIO/DG, TIMSD |
List of Appendices
NOTE. APPENDICES HAVE BEEN REMOVED AND ARE AVAILABLE UPON REQUEST
- Appendix A – Description of MSCA Applications
- Appendix B – Audit Detailed Criteria List
- Appendix C – Audit Reports Conclusion Scale
- Appendix D – Audit Reports Recommendation Scale
Footnotes
Footnote 1 The Marine Security Directorate is responsible for marine security policy, marine security regulatory affairs, and marine security operations, as well as functional authority for regional marine security operations. (Return to footnote 1 source paragraph)
Footnote 2 The breakdown of the funds expended on each application was provided by Marine Safety. The total spent was reconciled to the Departmental Financial System. (Return to footnote 2 source paragraph)
Footnote 3 Unauthorized disclosure could reasonably be expected to cause serious injury to an individual, organization or government. TC defines protected B information to include: a person's performance evaluation and character references, criminal records, solicitor-client privileges, medical records and departmental risk assessments, among others (Return to footnote 3 source paragraph)
The following document is available for downloading or viewing:
- System under Development Audit of the Marine Safety Core Architecture Project ( PDF Version, 473 kb )
To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: