Evaluation of Rail Safety Improvement Program

Evaluation report outlining results of the assessment of the Rail Safety Improvement Program.

On this page

• Executive summary
• Introduction
• Program profile
• Contributions
• Evaluation approach
• Evaluation findings
• Conclusion
• Annex: References

Executive summary

About the program

The Rail Safety Improvement Program (RSIP) is a grant and contribution program that consolidates the former Grade Crossing Improvement Program (GCIP), Grade Crossing Closure Program (GCCP), and Operation Lifesaver Program (OL). RSIP was created in 2016.

RSIP was provided with a budget of $64.7 million over three years (2016-17 to 2018-19) to support rail safety improvements. The objectives of the program are to support safety improvements to existing rail lines, closures of grade crossings, research to develop new technologies that enhance rail safety, and initiatives to raise awareness about rail safety issues across Canada.

About the evaluation

The evaluation of RSIP assessed the key issues of relevance, performance, and efficiency, with particular focus on the achievement of expected outcomes of the program. The evaluation employed multiple data collection methods: a document and literature review, key informant interviews, administrative and financial data review, and case studies. The evaluation covered the period from 2014-15 to 2018-19, and part of 2019-20.

What the evaluation found

There is a need for rail safety improvements in Canada and RSIP helps to address the risks related to grade crossings and supports the education and awareness of rail safety issues across Canada.

The evaluation found that RSIP has contributed to safety improvements through investments in infrastructure, technology and research projects, as well as education and awareness projects. The majority of RSIP funding has been allocated to high risks projects, as measured by the GradeX risk tool. However, there are opportunities to improve this risk assessment model. Overall, RSIP has contributed to risk reduction in the rail sector and, therefore, has increased rail safety for the public.

The evaluation also found that although RSIP has an expanded list of eligible recipients compared to the previous rail safety funding programs (i.e., GCIP, GCCP and OL), some of the new recipient types, particularly small municipalities and small shortlines, have encountered barriers in accessing RSIP funding and in certain cases, struggle with the cost-sharing ratio, when compared to Class 1 railways.

In terms of the design and delivery for the RSIP, the evaluation found that some phases of the RSIP application and approval processes took longer than anticipated. Specifically, the long and unpredictable waiting period between application review and approval is difficult for recipients in terms of project planning. However, in recent years, RSIP has introduced and implemented some Lean principles to improve process efficiencies.

Further, although the evaluation criteria used to evaluate RSIP project proposals have been established, there are opportunities to improve the consistent application of the criteria and the description of roles and responsibilities for all reviewers. Finally, there is evidence that the Program relies on human resources from within Transport Canada, but separate from RSIP, who help to deliver the Program and are not accounted for in the Program's costing model.

Recommendations

Recommendation 1: RSIP management should examine funding options, in collaboration with Rail Safety (who are working to address the recommendations from the 2018 Railway Safety Act Review) and Rail Policy, to identify if there is a need to allocate a portion of funding for all types of applicants such as municipalities, shortlines, and passenger rail during each funding cycle.

Recommendation 2: RSIP management should examine the extent to which the cost-sharing ratios defined in the RSIP Terms and Conditions remain relevant for all types of recipients, and to determine if new cost-sharing ratios should be considered given the financial capacities of the different categories of eligible applicants.

Recommendation 3: RSIP should develop a costing model that comprehensively accounts for all of the costs required to deliver the Program (including the human resources who are funded through A-base funding in other groups within Transport Canada) so that an accurate financial estimate can be determined for program renewal and planning purposes.

1.0 Introduction

This report presents the results of an evaluation of the Rail Safety Improvement Program (RSIP). The evaluation was conducted to address commitments made in the Program's {ATIP REMOVED} to meet the evaluation schedule listed in Transport Canada's annual five-year evaluation plan, and to address the coverage requirements outlined in the Policy on Results (2016) and section 42.1 of the Financial Administration Act.

1.1 Program Context

Canada has a large and well-developed rail network which impacts industry, communities, and the environment. Ongoing and emerging safety needs are tied to the changing environments of individual crossings, rail lines, and communities, as well as an evolving awareness of rail safety. Further, the operating environment of the rail sector varies across the country given land use around rail operations, climatic and environmental variation, economic needs and priorities, and technological innovation. Addressing these safety concerns is a shared-responsibility between railway companies, road authorities/municipalities, communities, and the government.

The government is committed to sustaining a safe and secure rail transportation system in Canada, which is important to Canada's economy and communities. In Budget 2016, the government announced a commitment of $143 million over three years to improve rail safety ($64.7 million) and transportation of dangerous goods.

In 2017, the government mandated a review of the Rail Safety Act (2018), resulting in sixteen recommendations to enhance rail safety. The report recommended that Transport Canada "improve grade crossing safety, including increased, and on-going funding for grade crossing improvements (i.e., Rail Safety Improvement Program)." The report also highlighted evolving safety challenges to Canadian rail safety, stating that grade crossing and trespassing accidents still cause the most deaths and serious injuries, and that the proportion of accidents caused by human factors is increasing in Canada.

Transport Canada has taken steps to implement recommendations from the 2018 Railway Safety Act Review report, including improving grade crossing safety to provide for a safer rail transportation system.

1.2 Program Description

The Rail Safety Improvement Program (RSIP) was allocated a budget of $64.7 million over three years (2016-17 to 2018-19) to support rail safety improvements. The Program's objectives are to support "safety improvements to existing rail lines, closures of grade crossings, research to develop new technologies that enhance rail safety, and initiatives to raise awareness about rail safety issues across Canada."^{Footnote 1} This aligns with departmental priorities.^{Footnote 2}
The program is comprised of two components:^{Footnote 3}

• The Infrastructure, Technology, and Research (ITR) component, which funds projects focused on public safety at and around federally and provincially regulated rail crossings and rail property, as well as projects that involve research and technological development that can contribute to rail safety; and
• The Public Education and Awareness (E&A) component, which funds public education projects to reduce collisions and trespassing incidents

RSIP is accessible to a variety of stakeholders. Eligible recipients such as VIA Rail and for-profit organizations are eligible to receive up to 50% funding. All other recipient types can receive up to 80%.^{Footnote 4}

Rail safety programming at Transport Canada has evolved over time and RSIP is the consolidation of the former Grade Crossing Improvement Program (CGIP), Grade Crossing Closure Program (GCCP) and Operation Lifesaver Program (OL). RSIP was launched in 2016.

2.0 Findings

2.1 Program performance

The performance of RSIP was assessed through an examination of its continued relevance, achievement of expected outcomes including the distribution of projects funded and targeting high-risk projects, and its contribution to overall risk reduction and improving the safety of the rail transportation system.

2.1.1 Performance – relevance

Because Canada is a country built on rail, railways "are a significant part of our infrastructure that go through many communities" (Interview Participant). As such, there are a wide variety of safety considerations and concerns. The Government of Canada has identified rail safety as a priority, to which Transport Canada is committed.^{Footnote 5} A variety of safety concerns can be addressed through a program like RSIP.

Evidence from document and literature reviews on rail safety highlighted several key safety issues. These include: train and road user interactions at crossings; pedestrians and other vulnerable road users^{Footnote 6}; trespassing^{Footnote 7} and suicides^{Footnote 8}; and derailments.^{Footnote 9} In addition, new considerations continue to emerge. Research into human behaviours has indicated that human factors are a major contributor to accidents.^{Footnote 10} These factors include both risky driver behaviour and decision-making^{Footnote 11}, as well as the actions of pedestrians and cyclists.^{Footnote 12} Findings from interviews aligned with what was found in literature. In particular, interviewees highlighted issues such as urban growth, pedestrian safety, and trespassing, in addition to the traditional focus on train-vehicle collisions at crossings. Increased traffic, greater exposure, insufficient and outdated systems, and general complacency among road users were noted as important considerations. A few participants also pointed out that there are a large number of crossings in Canada without any controls aside from lights, suggesting there is still work to be done. This is confirmed by TC's Grade Crossings Inventory,^{Footnote 13} which lists 22,820 crossings as of April 2019, with over two-thirds of these categorized as passive (i.e., without flashing lights, bells, or gates).

The ongoing demand for RSIP funding further helps to demonstrate the relevance of this program (Figure 1). This demand is expected to increase over the next several years, due to the new Grade Crossing Regulations coming into full effect in November 2021.^{Footnote 14} Given the dynamic conditions of the rail sector, there is a continued need for rail safety improvements.

Figure 1: Number of RSIP applications received for funding in fiscal years 2017-18 to 2020-21

Number of RSIP Applications Received, 2017-18 to 2020-21

Source: Program Data, December 2019. Note: RSIP-E&A was not accepting applications for 2020-21.

Description

This line graph shows the number of RSIP applications received from fiscal year 2017-18 to 2020-21. In 2017-18, RSIP received 7 E&A applications and 282 ITR applications for a total of 289 applications. In 2018-19, there were 9 E&A applications and 287 ITR applications for a total of 296. In 2019-18, there were 6 E&A applications and 242 applications for a total of 248. In 2020-21, RSIP received a total of 420 applications, all of which were for ITR projects. E&A applications were not open.

Clearly, there is an ongoing need for funding related to rail infrastructure and safety awareness. The majority of interview participants emphasized the need for and value of this program. While RSIP is not the only means of ensuring rail safety, it is "an important tool in the toolbox" (Interview participant). The consensus among interviewees was that if RSIP or a comparable program did not exist, stakeholders would have difficulty performing these upgrades.

2.1.2 Performance – RSIP projects

Since its inception in 2016, RSIP has funded 371 crossing, infrastructure, closure, and research/technology projects, as well as 19 education and awareness projects. To demonstrate the impacts of the Program, specific results from each of RSIP's components are outlined below, including specific case study examples. This is followed by a consideration of the current state of rail safety in Canada. Overall, evidence suggests that RSIP funding contributes to reducing the risk of rail occurrences in Canada, building upon the work of RSIP's predecessor programs and working in parallel with regulatory oversight and other safety initiatives.

1. Crossing, Infrastructure, and Closure Projects
   "Improvements might include features like level track, lights, bells, gates – you'd be surprised that these aren't very prevalent, especially in rural communities – these changes alone make a big difference with respect to what the program has set out to do." (Interview participant)

Overall, internal and external interview participants agreed that infrastructure and crossing projects do improve safety. Infrastructure and crossing improvements may include the addition or upgrade of gates, lights, bells, markings, and signage at an intersection; fencing along the rail line; road and/or sidewalk realignments; and sightline adjustments. These account for the majority of RSIP-ITR projects. Closures, which involve closing a crossing and road traffic travelling via an alternate route, comprise around 10% of RSIP-ITR projects between 2017-18 and 2019-20.

Of the more than 22,000 crossings documented in TC's Grade Crossings Inventory,^{Footnote 15} approximately 45% (10,293) are located in the Prairie and Northern Region (PNR) (Figure 2). The distribution of RSIP-ITR projects reflects the high number of crossings in this region: approximately 35% of funding between 2017-18 and 2019-20 was allocated to projects in PNR (Figure 3). Similarly, the region with the second highest number of crossings, the Ontario Region (ON), received approximately 25% of RSIP-ITR funding. Furthermore, this funding distribution also appears to reflect the proportion of high risk crossings in these regions (see Figure 2 and Figure 3).

Figure 2: Number of grade crossings in each region (Pacific, Prairie and Northern, Ontario, Quebec, and Atlantic), including the number of high risk crossings

Regional Distribution of Crossings, 2019

Source: TC Grade Crossings Inventory, May 2019.

Description

This figure is a map of Canada, overlaid with five pie charts. It illustrates the distribution of grade crossings across Transport Canada's five regions, as of 2019. Crossings are categorized as high risk (with a GradeX rank between 1 and 5500) and not high risk (with a GradeX rank over 5500). In the Pacific Region, 451 crossings were high risk and 2139 were not high risk. In the Prairie and Northern Region, 2000 were high risk and 8293 were not high risk. In the Ontario Region, 1871 were high risk and 2905 were not high risk. In the Quebec Region, 911 were high risk and 2539 were not high risk. In the Atlantic Region, 267 were high risk and 1444 were not high risk.

Figure 3: RSIP-ITR funding distribution by region

Approved ITR Funding Amounts by Region, 2017-18 to 2019-20

Source: Program Data, October 2019. Note: Amounts include all ITR projects (i.e., crossing, infrastructure, closures, research and technology)

Description

This pie chart illustrates the total approved funding amounts for ITR projects in each region between 2017-18 and 2019-20. Projects in the Pacific Region received $11,692,288. Projects in the Prairie and Northern Region received $23,545,142. Projects in the Ontario Region received $16,512,866. Projects in the Quebec Region received $7,577,457. Projects in the Atlantic Region received $5,829,416. Projects which spanned multiple regions received $488,408.

A closer look at individual projects demonstrates the specific results of RSIP. For instance, a case study with the City of London, Ontario, found that RSIP has indeed funded projects at problematic and high traffic crossings. Representatives indicated that a variety of specific improvements were made, including better signage, sightlines, and crossing surfaces, along with improved signals. Importantly, funding provided through RSIP enabled the City to accelerate the timeline for planned improvements, which ensured that risks were mitigated earlier.

Case studies also demonstrated that emerging safety concerns are being addressed through RSIP. For instance, pedestrian safety is a key issue that several RSIP-funded projects aimed to address. In Moncton, New Brunswick, crossing upgrades focused on pedestrian needs and accessibility issues. In White Rock, British Columbia, improvements were made to pathways and barriers to address regular trespassing which resulted from a rail line separating a popular beach from a parking lot. In these ways, RSIP is able to directly contribute to risk reduction across the country.

2. Research and Technology Projects

While research and technology projects account for a small percentage (approximately 5%) of the RSIP-ITR funding allocations, these projects move beyond traditional means of solving safety issues. RSIP has funded research and technology projects that address emerging safety issues and enable the development and application of new technologies.

There are a number of safety issues that may be explored and addressed through new technology. Researchers and companies are exploring and implementing innovative technologies to reduce risk along the rail line and at crossings beyond the application of traditional improvements (i.e., lights, bells, gates). For example, TRAINFO, which will receive funding through RSIP for the 2019-20 funding year, is a technology company specializing in mitigating rail crossing blockages. A blocked crossing can lead to traffic congestion and risky behaviour, as well as impeding the movement of emergency vehicles. TRAINFO sensors installed at and around crossings monitor blockage and traffic patterns, and algorithms analyze this data. Traffic signal timing can then be adjusted, messages can be shared with drivers to ensure awareness and allow for re-routing, and cities can better manage their traffic. This technology has been piloted and implemented in Vancouver, Winnipeg, London, and other locations across Canada. Results include a reduction in traffic delays, faster EMS response times, improved communication between the city and the public, better signal timing, and better data for planning transportation projects.

Another way safety issues can be understood is through risk modelling. The risk assessment tool GradeX (described in more detail below) can be used for risk ranking and inspection planning. It may also be used to analyze improvement needs at crossings and to explore risk-based funding across the regions. Recent work funded through RSIP involved modelling pedestrian usage and safety, which will be incorporated into GradeX to account for pedestrian risk factors. The same research team recently (2017-2019) worked to develop different applications, including one for inspectors and one for the general public.

3. Education and Awareness Projects

The Education and Awareness (E&A) component of RSIP is also small, approximately 5% of RSIP overall, but it has increased awareness of rail safety among Canadians. Interviews highlighted that education and awareness works in parallel to physical improvements. General awareness among drivers and pedestrians is important to avoid complacency, and importantly, some issues, such as suicide, cannot be addressed through technical solutions. Projects under RSIP have included education sessions, signage campaigns, and training development.

While it can be difficult to measure the influence of education and awareness activities, it can be approached through surveys, online traffic, and, at a high level, incident and accident rates, as demonstrated by the case study example of the Safety Train. As an additional example, the national education and awareness organization Operation Lifesaver (OL), which receives funding through RSIP, reported over 200,000 website visits and 125,000 campaign video views since 2017, as well as an increase in social media followers. In addition, OL surveyed participants and found that 93% of people said they benefited in terms of safety awareness after participating in an OL activity.

Contribution to overall risk reduction

Rail occurrences vary from year to year, but overall safety appears to be improving. An examination of recent Transportation Safety Board (TSB) data demonstrates that over time the number of crossing and trespassing fatalities has declined (Figure 4). Serious injuries at crossings is also declining (Figure 5). Although serious injuries due to trespassing appear to be an issue, it should be noted that the 34 trespasser fatalities in 2018 is below the 10-year average of 44. The TSB also stated that "in 2018, 14% of rail accidents involved vehicles or pedestrians at rail crossings, below the 17% average of the previous 10 years."^{Footnote 16}

Figure 4: The number of fatalities attributed to crossing and trespassing occurrences between 2000 and 2018.

Fatalities: Crossings vs Trespassing

Source: TSB Data.

Description

This is a clustered bar graph with linear trend lines, comparing the number of fatalities attributed to crossing occurrences to those attributed to trespassing occurrences between 2000 and 2018. In 2000, there were more than 30 crossing fatalities and more than 50 trespassing fatalities. In 2001, there were 40 crossing fatalities and approximately 55 trespassing fatalities. In 2002, there were approximately 45 crossing fatalities and 50 trespassing fatalities. In 2003, there were under 30 crossing fatalities and approximately 45 trespassing fatalities. In 2004, there were approximately 25 crossing fatalities and under 70 trespassing fatalities. In 2005, there were nearly 40 crossing fatalities and more than 60 trespassing fatalities. In 2006, there were fewer than 30 crossing fatalities and nearly 60 trespassing fatalities. In 2007, there were over 25 crossing fatalities and approximately 55 trespassing fatalities. In 2008, there were under 30 crossing fatalities and approximately 45 trespassing fatalities. In 2009, there were nearly 20 crossing fatalities and over 50 trespassing fatalities. In 2010, there were over 20 crossing fatalities and approximately 55 trespassing fatalities. In 2011, there were approximately 25 crossing fatalities and over 40 trespassing fatalities. In 2012, there were over 30 crossing fatalities and over 40 trespassing fatalities. In 2013, there were 30 crossing fatalities and over 40 trespassing fatalities. In 2014, there were over 20 crossing fatalities and over 30 trespassing fatalities. In 2015, there were approximately 15 crossing fatalities and 30 trespassing fatalities. In 2016, there were nearly 20 crossing fatalities and more than 40 trespassing fatalities. In 2017, there were nearly 20 crossing fatalities and over 50 trespassing fatalities. In 2018, there were more than 15 crossing fatalities and over 30 trespassing fatalities.

The trend lines on this graph indicate that over time, fatalities in both contexts have decreased, with trespassing fatalities consistently remaining higher than crossing fatalities.

Figure 5: The number of serious injuries attributed to crossing and trespassing occurrences between 2000 and 2018.

Serious Injuries: Crossings vs Trespassing

Source: TSB Data.

Description

This is a clustered bar graph with linear trend lines, comparing the number of serious injuries attributed to crossing occurrences to those attributed to trespassing occurrences between 2000 and 2018. In 2000, there were over 30 serious injuries at crossings and over 20 due to trespassing. In 2001, there were over 45 serious injuries at crossings and over 20 due to trespassing. In 2002, there were over 40 serious injuries at crossings and just over 20 due to trespassing. In 2003 there were over 50 serious injuries at crossings and nearly 20 due to trespassing. In 2004, there were 50 serious injuries at crossings and more than 30 due to trespassing. In 2005, there were approximately 55 serious injuries at crossings and more than 15 due to trespassing. In 2006, there were 30 serious injuries at crossings and over 25 due to trespassing. In 2007, there were more than 20 serious injuries at crossings, and more than 20 due to trespassing. In 2008, there were more than 35 serious injuries at crossings and 20 due to trespassing. In 2009, there were over 20 serious injuries at crossings and approximately 15 due to trespassing. In 2010, there were more than 25 serious injuries at crossings and fewer than 20 due to trespassing. In 2011, there were more than 20 serious injuries at crossings and just over 20 due to trespassing. In 2012, there were over 30 serious injuries at crossings and over 20 due to trespassing. In 2013, there were more than 25 serious injuries at crossings and 10 due to trespassing. In 2014, there were fewer than 30 serious injuries at crossings and just over 20 due to trespassing. In 2015, there were more than 15 serious injuries at crossings and just over 15 due to trespassing. In 2016, there were over 20 serious injuries at crossings and 20 due to trespassing. In 2017, crossing and trespassing occurrences resulted in the same number of serious injuries: over 20. In 2018, there were over 40 serious injuries at crossings and under 30 due to trespassing.

The trend lines on this graph indicate that over time the number of serious injuries attributable to crossing occurrences has declined: from approximately 45 to under 25. The number of serious injuries attributable to trespassing occurrences has also declined, but has remained generally stable around 20.

Similarly, analyses of the influence of the previous program, GCIP, showed a downward trend in collisions, injuries, and fatalities at crossings in Canada.^{Footnote 17} Importantly, through statistical modelling, these past evaluations concluded that GCIP succeeded in reducing the risk of accidents.^{Footnote 18} RSIP supports similar projects. In addition, research has found that education and awareness is also effective in addressing rail safety issues.^{Footnote 19}

Broadly speaking, the data indicates that rail safety in Canada is steadily improving. Evidence also suggests that the work supported by RSIP has helped to address safety risks across the country. Ultimately, this reduction in risk leads to a higher level of rail safety for Canadians.

2.1.3 Performance – targeting high risk projects

RSIP provides funding to support projects in improving rail safety (and does not provide funding for the maintenance of railways), and thus a key consideration in project selection is the safety risk that would be addressed through the proposed project. Risk is scored through inspector knowledge and inspection history, collision history, TSB information, and, where relevant, an internal risk assessment tool called GradeX. For proposed projects focused on a grade crossing, the GradeX risk ranking is one of the criteria that factors into the final score awarded to an application.

A review of Program data^{Footnote 20} indicates that RSIP has funded projects that are high risk based on GradeX (i.e., between 1 and 5500 in GradeX^{Footnote 21}). Between 2017-18 and 2019-20, just over half of all approved RSIP-ITR funding ($35.95M) was allocated to high risk projects. However, it is important to note that the RSIP-ITR component includes more than projects at crossings; there are also research and technology projects and infrastructure projects and these projects are not associated with a grade crossing and do not have a GradeX rank. In addition, some crossings may not appear in GradeX because they are currently federally unknown, typically because they are provincially regulated. When excluding the types of RSIP-ITR projects that do not have a GradeX rating for logical reasons, 257 crossing projects between 2017-18 and 2019-20 remained that had a GradeX ranking. Of these projects, 75% were statistically high risk (Figure 6).

Figure 6: Proportion of RSIP-ITR projects at high risk crossings according to GradeX rank.

RSIP-ITR 2017-18 to 2019-20: Projects at Crossings

Source: Program Data, October 2019. Note: Excludes projects without a GradeX rank (i.e., research and technology, infrastructure, crossings not currently included in GradeX).

Description

This horizontal 100 per cent stacked bar chart illustrates the proportion of high risk projects at crossings and not high risk projects at crossings from 2017-18 to 2019-20. In 2017-18, 79.55 per cent of RSIP-ITR projects were high risk (GradeX rank between 1 and 5500) and 20.45 per cent were not high risk (GradeX rank over 5500). In 2018-19, 71.76 per cent of RSIP-ITR projects were high risk and 28.24 per cent were not high risk. In 2019-20, 72.62 per cent of RSIP-ITR projects were high risk and 27.38 per cent were not high risk.

Several interview participants stated that GradeX plays a significant role in the evaluation of RSIP funding applications. Interviewees also cautioned that GradeX is not the determining factor for selection. The local knowledge of inspectors is vital, as they are often aware of additional factors or more up-to-date information that affect the risk associated with a given crossing, including sightlines, typical use, and environmental conditions.

While interviewees generally indicated that GradeX is a useful tool, several limitations were highlighted:

Several interview participants stated that GradeX plays a significant role in the evaluation of RSIP funding applications. Interviewees also cautioned that GradeX is not the determining factor for selection. The local knowledge of inspectors is vital, as they are often aware of additional factors or more up-to-date information that affect the risk associated with a given crossing, including sightlines, typical use, and environmental conditions.

While interviewees generally indicated that GradeX is a useful tool, several limitations were highlighted:

• Limited risk factors: GradeX does not account for factors such as site layout (e.g., sightlines, angle of approach, proximity to nearby intersections), human behaviour, and typical weather conditions. However, as stated, inspectors are generally aware of these factors and can account for them in their assessments of applications.
• Data quality: GradeX relies on data from other systems, the availability and accuracy of the data may be inconsistent. Data on a crossing may not be up to date, or, in the case of crossings that are provincially inspected and regulated, may not exist.
• Relative risk: Rankings in the inventory are relative, not a statement of absolute risk. As one interviewee explained, "it's not necessarily a high risk crossing, but a higher risk crossing."
• Crossings only: Because GradeX is specific to grade crossings, it cannot be used to consider infrastructure projects and other safety concerns.
• User biases: During interviews, it was learned that certain GradeX users view the tool with varying degrees of usefulness. Some users focus heavily on the limitations and tend to discount the utility of GradeX. Other users accept that it has limitations but feel that the risk information is still valuable, mentioning that it is another useful data source to consider when making funding decisions.

Participants also explained that although upgrades may be done at a crossing, the surrounding environment or usage could change, resulting in a higher risk ranking. Similarly, some crossings will remain 'high risk' even with all possible upgrades, due to factors that cannot easily be altered (e.g., site configuration, traffic patterns, and historic occurrence data). In addition, a crossing may be considered high risk, but the proposed project may not be the appropriate solution. For instance, a crossing risk may be better addressed through intersection realignment rather than increased signage.

In view of current limitations and emerging needs, there are ongoing improvements being made to GradeX. For instance, recent work has involved adding pedestrian risk factors to the statistical model.

In general, the consensus among interviewees was that the intent is for RSIP to target high risk crossings and that GradeX plays a quantitative and enabling role in this regard. Paired with the Program data, it appears that RSIP is successfully targeting high risk locations. However, in addition to the limitations involved in measuring risk, it is important to note that ultimately, RSIP is application-based; if projects at high risk locations are not applied for, then any desirable upgrades will not occur through the Program.

2.1.4. Performance – financial

In order to better understand RSIP's financial performance, the program's committed budget was compared to actual expenditures in order to determine what proportion of contribution funding committed each fiscal year was actually being used to deliver projects.

As demonstrated in Table 1, over a three-year period of funding, $56.2 million of the budget was committed to ITR and E&A projects. When examining the $53.7 million committed specifically to ITR projects, it was observed that $41.8 million of the budget was expended on projects (i.e., 77.8% of the committed budget). As can be seen in Table 1, each fiscal year RSIP has lapsed funds ranging from approximately 10% to 53% of the ITR committed budget. With the exception of 2017-18, the average proportion of lapsed funds was slightly over 10% annually. RSIP's ITR budget expended is consistently lower than the committed budget, which results in lapsing funds each year.

Table 1: RSIP budget committed and expended ($) by component, 2016-17 to 2018-19

Fiscal Year	Total budget committed	E&A budget committed	ITR budget committed	ITR budget expended	ITR budget variance
2016-17	10,637,305	300,000	10,337,305	9,390,294	10.1%
2017-18	24,308,144	1,071,933	23,236,211	15,183,327	53.0%
2018-19	21,292,642	1,154,594	20,138,048	17,252,808	16.7%
TOTAL	56,238,091	2,526,527	53,711,564	41,826,429	28.4%

Note 1: There is no column for E&A budget expended because there was no lapsed funding for these project types over the years examined.

Note 2: Budget committed is the budget approved or announced for funding RSIP projects each fiscal year.

Note 3: Budget expended is the budget payment, amount paid to recipients, for the project work that is completed.

Findings from document reviews and interviews highlighted a few key issues that contributed to the lapsing of funds:

• The planning and budgeting cycles of recipients (i.e., Class 1 railways, municipalities, etc.) typically do not align with TC's internal approval and announcement schedules, ultimately leading to a lack of alignment between key decision points leading to project delays and lapsed funds.
• The construction season is limited in most parts of Canada and due to the misalignment of planning and budgeting cycles, there is even less time for recipients to complete projects on schedule (especially for one-year projects).
• Funding for approved projects in Quebec has also been carried over because the province of Quebec has not yet given the authority to local municipalities to enter into agreements with the federal government. A contribution agreement template is being negotiated.

It is important to note that many of these factors are out of the control of RSIP program management.

While there is currently no threshold set for what is considered acceptable and unacceptable for lapsing RSIP funds, when reviewing documents from the Treasury Board of Canada Secretariat it was learned that if lapsed funds were greater than 5% of total voted authorities (i.e., at the aggregate, departmental level), that this situation may need to be examined in further detail. Some interviewees agreed that there is an opportunity for RSIP management to work towards decreasing the amount of lapsed funds moving forward.

2.2 Program efficiency

To assess the efficiency and economy of RSIP, the evaluation examined the extent to which: the RSIP funding is equally accessible to all types of eligible applicants, the design and delivery of the application process is efficient - including the due diligence process and the approval process - and the Program's management of its financial and human resources.

2.2.1 Access for eligible recipients

"RSIP is for everybody." (Interview participant)

As indicated earlier, RSIP is a merger of three former rail safety funding programs. Unlike the Grade Crossing Improvement Program (GCIP) and the Grade Crossing Closure Program (GCCP), RSIP allows for infrastructure work along the rail lines and projects on provincially regulated tracks and crossings in addition to federally regulated locations. RSIP also funds research and technological development and implementation projects, which under the previous programs were out of scope. In terms of rail safety awareness, RSIP's education and awareness component funds local and regional campaigns and is open to any organization to apply, while also continuing to fund the national campaign through Operation Lifesaver.

Several interviewees highlighted this as a positive change. For example, one said that "it's nice that RSIP is more inclusive to the smaller groups who need the money the most. RSIP allows us to distribute the funds a little wider and there's more flexibility in what we fund." Further, RSIP makes it possible for stakeholders to explore and implement innovative means of addressing safety issues. In another example, one interviewee highlighted that the Program is broader and more flexible than its counterpart in the United States, the Railway-Highway Crossings Program, commonly referred to as Section 130. As such, RSIP was viewed as a best practice: "there is this awareness now that Canada is taking leadership in addressing this issue [rail safety]."

Results of the evaluation indicate that most of the design and delivery features of RSIP are well implemented and appropriate for meeting its current objectives. However, some features relating to the diversity in eligible recipients may require further analysis to better understand barriers to access RSIP funding. Evidence from document and literature reviews, interviews and case studies indicates that eligible recipients other than Class 1 railways, have encountered a number of barriers to accessing RSIP funding. The main factors include:

• Capacity to complete RSIP applications. While they may be aware of RSIP, certain eligible recipients, like shortline railways and small municipalities, may experience barriers to funding due to limitations in capacity and resources. In other words, these entities may struggle to complete an RSIP application in a comprehensive and robust way, as compared to Class 1 railways who have dedicated resources for this purpose. For example, interview participants indicated that small municipalities/road authorities frequently lack the capacity in terms of time and knowledge to effectively complete applications. "If I were a road authority and wanted to apply, I wouldn't know how to start... you need almost a rail background to apply" (Interview participant). In addition, another factor that likely contributes to the small proportion of funding received by small municipalities (when compared to Class 1 railways), is due to the fact that RSIP staff can only evaluate a project proposal based on the quality of the application.
• Financial resources to complete RSIP projects. If successful in applying for funding, small municipalities have limited budgets and small tax bases and thus may struggle to fund the remaining portion of their subsidized projects. One interviewee mentioned that "smaller municipalities are reluctant to apply because their budgets are razor thin to begin with, so even though money is available through RSIP, they still need to commit money of their own and that may be a stretch". The funding story for a crossing in Macoun, Saskatchewan exemplifies the difficulties that small municipalities face (see Case Study 3).
• Awareness and uptake of RSIP. Some eligible recipients may be unaware of the availability of RSIP funding. Internal and external interview participants indicated that there are varying levels of awareness of RSIP. Railways, particularly the Class 1 companies, are well aware of the Program. Road authorities, particularly smaller municipalities, do not necessarily know about the Program and since RSIP is an application-based program, an application must be submitted in order for a project to be funded.
• Operational and environmental conditions. With the expansion of RSIP to include additional recipient types there have been some impacts that may not have been fully anticipated. Since one of the key evaluation criteria of the RSIP application process is targeted towards high risk crossing projects, some proposals from smaller eligible recipients (e.g., shortline railways, small municipalities/road authorities) are difficult to compare to proposals from Class 1 railways because the underlying operating circumstances of these recipient types are much different.

Findings from the evaluation suggests there might be a need for RSIP management to assess whether any special funding provisions should be implemented to enable recipients to compete for funding in a more structured manner (e.g., establishing portions of funding by recipient type). The expansion of RSIP's eligible recipients has been viewed as a positive change; however, there is outstanding work to be done to determine if other design modifications are necessary to ensure that these new recipient-types understand how project proposals are evaluated and have realistic expectations of the probability of their project proposals being funded.

When analyzing application approval data, it becomes clear that the proportion of applications approved by recipient-type varies considerably, with Class 1 having a high average application approval rate of 72% when compared to other recipients.

For example, program data shows that, in general, shortlines have a low application approval rate (between 2% and 5% in 2017/18 and 2018/19). In the most recent data available, 100% of shortline applications were approved, but this approval rate represents a total of only 5 applications that were submitted, which is a sizeable decrease from the previous years (40 and 45 applications in 2017/18 and 2018/19, respectively). In 2018-19, RSIP evaluated nine new projects and re-evaluated 36 projects that were previously not approved in 2017-18. Internal interviewees noted that this may, in part, be because shortlines now seem to have a better understanding of the Program and its eligibility requirements and evaluation criteria (e.g., applications for maintenance projects are not funded, incomplete and poor quality applications are not funded).

Internal interviewees noted that this may, in part, be because shortlines now seem to have a better understanding of the Program and its eligibility requirements and evaluation criteria (e.g., applications for maintenance projects are not funded, incomplete and poor quality applications are not funded).

In addition, municipalities have decreased the number of applications submitted over time, but have an upward trend in application approval rates, with nearly half of all applications being approved in 2019-20 at 47%. It should be noted that municipalities/road authorities have submitted the largest cumulative number of applications.

Findings from interviews and a review of literature found that shortline railways have raised concerns about receiving funding from Transport Canada and RSIP. For example, there was a specific recommendation made in the 2018 Railway Safety Act (RSA) Review where it was recommended that a portion of RSIP funding be allocated for grade crossing improvements involving railways other than Class 1 railways. Considering that there is ongoing work occurring within Transport Canada aimed at better understanding the funding needs of shortline railways, it will be important for RSIP to be involved and aligned with this work, to ensure that RSIP can be considered in any decisions made and that those actions can be integrated in RSIP's program design.

There is diversity among Canada's shortline rail companies (i.e., a railway with annual operating revenues of less than $250 million for two consecutive years), but typically these entities are small (e.g., fewer than 25 employees), freight-focused, operate in small towns and rural areas, and have high operating costs and low revenue. Despite this, many^{Footnote 22} are subject to the same federal regulations as Class 1 companies. When investments into infrastructure, locomotive stock, and safety management systems are needed, these companies can struggle: "Short rail have the same costs as the Class 1 but they don't have the purchasing power. They're stuck with the rules for the big guys, but they don't play in those leagues" (Interview participant). This message was heard throughout the evaluation and speaks to the need for RSIP management to better understand the specific needs of the non-Class 1 railway RSIP-eligible stakeholders (e.g., shortlines and small municipalities/road authorities).

In the United States, for example, shortlines are divided into distinct categories in recognition of variations in size and capacity. Importantly, U.S. shortline companies have access to loans, tax credits, and grants.

As discussed previously, small road authorities/municipalities do not have financial flexibility and can struggle to obtain funding for their portions of RSIP-related projects for which they will be reimbursed. This is assuming that they are successful in being approved for funding, but as discussed, they are typically at a disadvantage during the application phase because these crossings are typically in rural locations and are considered lower risk. Furthermore, small municipalities/road authorities, much like shortline railways, do not possess the same expertise, practice, or human resources to efficiently and effectively complete funding applications when compared to Class 1 railways.

Recommendation 1: RSIP management should examine funding options, in collaboration with Rail Safety (who are working to address the recommendations from the 2018 Railway Safety Act Review) and Rail Policy, to identify if there is a need to allocate a portion of funding for all types of applicants such as municipalities, shortlines, and passenger rail during each funding cycle.

Recommendation 2: RSIP management should examine the extent to which the cost-sharing ratios defined in the RSIP Terms and Conditions remain relevant for all types of recipients, and to determine if new cost-sharing ratios should be considered given the financial capacities of the different categories of eligible applicants.

2.2.2 Design and delivery

RSIP is managed and delivered by Transportation Infrastructure Programs within Transport Canada. In order to deliver RSIP, 3.25 full-time equivalents (FTEs) are employed. It should be noted that these resources do not work full-time on RSIP alone.

Under RSIP, the process for approving applications for project funding comprises the following three phases:

1. Application phase

   To apply for RSIP funding, applicants must submit an application package by using the new online application portal, or by completing alternate application forms (to be submitted by email or couriered to Transport Canada).

   There is an annual call for applications. Applications must be received by the submission deadline (August 1 each year).

   Transport Canada screens each application they receive to ensure that the applicant and project meet the eligibility requirements, and that the proposal contains adequate information so they can begin the due diligence review.

2. Due diligence phase

   All funding requests are subject to a full review and evaluation as part of the due diligence process. Project proposals will be assessed against each evaluation criteria to ensure they meet the mandatory selection criteria and the project-specific selection criteria.

   The selection criteria for RSIP funding includes: the relevance of the project, quality of the proposed plan, and value for money. Risk plays a significant role in the project evaluation process, with factors such as collision history, GradeX rank (where relevant), TC inspection history, and any TSB investigations all being considered. The knowledge of TC inspectors is also vital, in many cases, inspectors are familiar with the locations being applied for and are aware of additional factors to consider.

   Transport Canada staff from Transportation Infrastructure Programs, Rail Safety Operations, and the Innovation Centre are involved in the due diligence review. These groups do not have FTEs specifically dedicated to RSIP review and contributed on an as needed basis.

3. Approval phase

   Following due diligence, after all the RSIP applications have been evaluated and the Minister has approved the recommended list of projects, applicants are informed of whether they have been selected for funding or not.

   Following project approval and the waiting period, a Contribution Agreement is prepared for signatures by the recipient and the Crown, which outlines the terms and conditions of the contribution, and the obligations of all parties.

Following approval, recipients are responsible for carrying out planned projects according to the signed Contribution Agreement. For reimbursement of eligible expenditures, recipients must submit invoices so that RSIP staff can track and monitor the claims for completed projects. Note that the program's financial resources will be discussed in more detail below.

Figure 7: RSIP-ITR application and project cycle

RSIP-ITR Application and Project Cycle

Description

This image outlines the RSIP-ITR application and project funding cycle. The funding round opens on August 2 (e.g., August 2, 2018) for 12 months, until August 1 of the following year (e.g., August 1, 2019). This is followed by application review and evaluation for approximately six months, and an additional two months for approvals. This may be finished at the end of March (e.g., March 31, 2020). Following this, the next 12 months are dedicated to signing contribution agreements, completing work, and managing claims. Contribution agreements end the following March (e.g., March 31, 2021).

In examining the application and approval processes of RSIP, the evaluation found three elements of concern related to the timeliness of the application process, timeliness of the evaluation process, and lapsing of program funds since RSIP launched in 2016.

Application and approval Process

The timeliness of the application and approval process was consistently highlighted as an area for improvement by internal and external interview participants. The majority of interviewees noted that a long and unpredictable waiting period from the time of application to the time of awarded funding (i.e., contribution agreement) resulted in project delays, uncertain project timelines, or project cancellations. There are certain factors that are outside the control of the Program that can also delay projects; for example, interviewees noted factors such as poor weather conditions, a short construction season, and a shortage of materials and supplies.

Findings from interviews and case studies involving recipients pointed to the difficulties faced by some municipalities, particularly small municipalities, in committing financial resources without knowing if and when the program funding would be awarded. Moreover, the long timeframe for project approval was considered to be an impediment, particularly for municipalities, as it requires planning adjustments, budget allocation, and project execution in less than six months.

Although the approval timeline was considered too long (e.g., in some cases, more than 12 months after application), external interviewees found the Program flexible in terms of amending contribution agreements, with the support of knowledgeable and experienced program staff. Two case study examples with Moncton, NB, and White Rock, BC, indicated that RSIP allowed for project plan modifications and project extensions, as necessary. Program management recognized that the approval process could be streamlined to become more efficient and launched a Lean review. Enhancing the efficiency of the approval process could also help to mitigate the lapsing/reprofiling of contribution funds. These issues are discussed further in Section 2.2.3.

In recent years, following lessons learned from the previous funding rounds and two Lean workshops, evaluators found that the Program has made significant progress towards increased efficiency of the application process to better serve clients.

For example, after the first Lean workshop in March 2018, the Program reduced the length of time from approval tothe announcement of funding by three months. Specific steps, such as creating templates, batching agreements, and clear communication with stakeholders, were decided upon to improve efficiency from approval to signing of contribution agreements. Implementation of these activities allowed for contribution agreements to be issued to and signed by recipients sooner than the previous year.

More recently, in September 2019, the Program completed a second Lean workshop focused on a long-term action plan expected to reduce the project evaluation process from about 6 months to 4 months. This is expected to be accomplished through improving the application guide and webpage, and ensuring a standardized assessment and evaluation process and criteria. Interviews with Program staff confirmed the implementation of some of these Lean related activities such as the launch of the online application portal and the new welcome guide package for potential recipients.

Evaluation due diligence process

As part of the application process, project proposals are reviewed and evaluated against a pre-determined list of evaluation criteria identified by the Program. These include both mandatory selection criteria and project-specific selection criteria.

Through document reviews and internal stakeholder interviews it was observed that the review and evaluation of funding applications has two key areas for improvement, namely:

• Inconsistent application of the evaluation criteria. Evaluators found a general consensus among internal interviewees and participants from the Lean workshop that the evaluation criteria are not applied in a consistent manner by different review board members when assessing project proposals, and that there are various interpretations of these criteria. To ensure that projects meet the evaluation criteria, it will be important for the program to consider standardizing the evaluation process, and to ensure all reviewers have a shared understanding of the criteria.
• Clarification of roles and responsibilities of inspectors. While the roles and responsibilities have been defined to support the delivery of the Program, internal interviews revealed that there is less clarity on the role inspectors play under RSIP than there was in the previous version of the Program (namely GCIP), and that this transition has led to some confusion. Interviews with a few inspectors confirmed that they are solely involved in the evaluation phase. In the application phase, some interviewees noted that inspectors have varying levels of informal involvement and following the evaluation process, inspectors are not routinely informed of approval or completion of projects.

Given the knowledge and expertise of inspectors, some interviewees noted that the Program should consider utilizing these existing resources, which could help to improve the delivery of RSIP. As a result of the Lean exercise in 2019, RSIP staff have worked toward involving inspectors earlier in the process to optimize the information flow from RSIP to inspectors. For example, RSIP staff working towards implementing these process improvements by providing application information to inspectors on a regular basis and by identifying the level of complexity of project proposals in consultation with inspectors However, as indicated early, inspectors are not dedicated FTEs for RSIP.

Transport Canada staff from Transportation Infrastructure Programs, Rail Safety Operations, and the Innovation Centre are involved in the due diligence review. These groups do not have FTEs specifically dedicated to RSIP review.

2.2.3 Cost-efficiency

The evaluation team was limited in their ability to assess the efficiency of RSIP without having a detailed breakdown of all of the Program's costs. To demonstrate efficiency, specific information is required to assess the use of resources in relation to the production of outputs and progress towards achieving expected outcomes.

The total cost to deliver RSIP was not clear. Program costs are not fully collected and tracked in a manner that allowed Evaluation to conduct specific financial analyses. For example, internal interviewees noted that the budget expended (i.e., claim payments to recipients) is not separated by recipient-type in the financial system.

Considering these challenges, RSIP's efficiency was examined by comparing the human resources used to deliver the program to the number of projects delivered. Funding for 2.25 FTEs was allocated in the 2016 RSIP funding document. Given that, on average, RSIP program staff have managed 179 ongoing approved projects each year from 2017-18 to 2019-20, the RSIP program appears to be very efficient. However, these numbers slightly underestimate the number of FTEs because, as has been discussed, RSIP relies on other TC employees to help deliver the program. At the project proposal review stage, Rail Safety inspectors and employees from TC's Innovation Centre are involved and this task can be very time consuming, given the large (and increasing) number of applications that are evaluated each year. Since these additional human resources are not captured in RSIP financial reports, Evaluation & Advisory Services was not able to accurately account for these hidden costs.

For RSIP there are direct and indirect costs. The planned human resources (e.g., 2.25 FTEs) can be considered the direct costs while the other TC employees who are integral in helping to deliver the program (e.g., Rail Safety, Innovation Centre) would be considered indirect costs. The human resources from Rail Safety and Innovation Centre are not funded directly by RSIP, they simply contribute when required while still maintaining their normal duties and responsibilities. Stated differently, some of Transport Canada's ongoing funding (i.e., A-base funding) is helping to deliver its temporary funding (i.e., B-base funding, like RSIP). When RSIP is next up for renewal, it would be useful for the Program to account for all of these costs in order to improve program planning and ensure that the proper resources are accounted for to deliver the Program.

Recommendation 3: RSIP should develop a costing model that comprehensively accounts for all of the costs required to deliver the Program (including the human resources who are funded through A-base funding in other groups within Transport Canada) so that an accurate financial estimate can be determined for program renewal and planning purposes.

3.0 Conclusions and recommendations

3.1 Conclusions

With an expansive rail network, there are a range of ongoing and emerging safety concerns across Canada. The evaluation found that the Rail Safety Improvement Program (RSIP) plays an important role in addressing these safety needs. The majority of RSIP projects are related to rail crossings and infrastructure improvements, though the broad scope of the Program has also enabled the funding of technology and rail safety awareness projects. Taken as a whole, projects funded through RSIP have contributed to an overall reduction of risk and increased safety for Canadians.

With risk reduction as a central objective of the Program, risk level greatly influences project selection. RSIP consistently allocates funds to projects at high-risk grade crossings, as measured by the GradeX risk assessment tool. The limitations of this tool, some of which may be addressed over time (e.g., inclusion of additional risk factors), are offset in part by the knowledge of regionally inspectors, who also play a central role in ranking project proposals.

While the broad scope of the Program demonstrates a recognition of rail safety as a shared responsibility, eligible recipients other than Class 1 railways may struggle to access RSIP funding. Barriers for these stakeholders are typically related to limitations in capacity, financial resources, and awareness. In addition, because of the different operational contexts of Class 1 railways and smaller applicants, comparing proposed projects is not a straightforward process. With these barriers in mind, there may be changes required to ensure greater equity in the application process and funding structure while continuing to address key rail safety needs of communities.

With respect to efficiency, there are a number of areas for potential improvement. Challenges tied to the long wait time between application review and formal approvals were found to be a key concern among stakeholders. However, the Program has been working to increase efficiency and notable improvements have been made. Although the GradeX tool provides some consistency to this review, it was also found that overall the evaluation criteria were not consistently interpreted or applied by review board members. A shared understanding of the criteria, along with the roles and responsibilities of Rail Safety staff in partnership with the Program, should be established. It is important to note that the Program relies on FTEs from these other groups to effectively review project applications and administer the program. Further, findings from both administrative and financial data show that the Program continues to lapse funds and that their forecasting of the costs associated to administer the program could be refined and improved.

Overall, since its inception in 2016, RSIP has continued to contribute to a key government priority, a safe rail transportation system. Internal and external stakeholders unanimously agreed that funding through RSIP enables recipients to improve rail infrastructure, explore innovative solutions, and educate Canadians on rail safety.

3.2 Recommendations

Recommendation 1: RSIP management should examine funding options, in collaboration with Rail Safety (who are working to address the recommendations from the 2018 Railway Safety Act Review) and Rail Policy, to identify if there is a need to allocate a portion of funding for all types of applicants such as municipalities, shortlines, and passenger rail during each funding cycle.

Recommendation 2: RSIP management should examine the extent to which the cost-sharing ratios defined in the RSIP Terms and Conditions remain relevant for all types of recipients, and to determine if new cost-sharing ratios should be considered given the financial capacities of the different categories of eligible applicants.

Recommendation 3: RSIP should develop a costing model that comprehensively accounts for all of the costs required to deliver the Program (including the human resources who are funded through A-base funding in other groups within Transport Canada) so that an accurate financial estimate can be determined for program renewal and planning purposes.

3.3 Management response & Action Plan

	RECOMMENDATION	Management Response and Action Plan	Target Date	Responsible Lead
1	RSIP management should examine funding options, in collaboration with Rail Safety (who are working to address the recommendations from the 2018 Railway Safety Act Review) and Rail Policy, to identify if there is a need to allocate a portion of funding for all types of applicants such as municipalities, shortlines, and passenger rail during each funding cycle.	a) Analyze the RSIP to review how funds are currently allocated to various categories of applicants. (completed) b) Engage Rail Safety and Rail Policy to identify the safety or policy requirements to allocate funding under other potential scenarios. (completed) c) Identify options to allocate funding between categories of eligible applicants (e.g., municipalities, shortlines, passenger rail, etc.) during each funding cycle. (completed)	Q1 FY 2020-21	DG, Transportation Infrastructure Programs
		d) Engage TC Centre of Expertise on Transfer Payments to determine what changes could be done in the short-term within the current Terms and Conditions, and others that would require changes to the overall Program. (completed) e) Action the short-term changes for the evaluation of 2021-22 projects.	Q2 FY 2020-21	DG, Transportation Infrastructure Programs
2	RSIP management should examine the extent to which the cost-sharing ratios defined in the RSIP Terms and Conditions remains relevant for all types of recipients, and to determine if new cost-sharing ratios should be considered given the financial capacities of the different categories of eligible applicants.	a) Analyze the RSIP applications and the allocation of funding to various categories of eligible applicants. (completed) b) Engage Rail Safety and Rail Policy to identify the safety or policy requirements to allocate funding under other potential scenarios. (completed)	Q1 FY 2020-21	DG, Transportation Infrastructure Programs
		c) Engage TC Centre of Expertise on Transfer Payments and Corporate Finance on any required changes to the Program's Terms and Conditions. d) Engage the Treasury Board Secretariat on the recommended changes. e) Make recommendations to the Minister to amend the Program's Terms and Conditions. f) Apply Program's Terms and Conditions for 2022-23 projects.	Q3 FY 2020-21       Q1 FY 2021-22	DG, Transportation Infrastructure Programs
3	RSIP should develop a costing model that comprehensively accounts for all of the costs required to deliver the Program (including the human resources who are funded through A-base funding in other groups within Transport Canada) so that an accurate financial estimate can be determined for program renewal and planning purposes.	a) Analyze the RSIP workload, including a project category framework which includes the type of skills needed, the type and the number of hours of work needed, in addition to an analysis of overhead number of hours and skill set needed to run RSIP. (completed)	Q2 FY 2020-21	DG, Transportation Infrastructure Programs
		b) Develop an RSIP costing model in conjunction with TC Finance.	Q3 FY 2020-21	DG, Transportation Infrastructure Programs
		c) Complete and disseminate the report on the costing of RSIP for input to TC Programs, Corporate Planning, and Finance.	Q4 FY 2020-21	DG, Transportation Infrastructure Programs

Appendix A: Methodology

Evaluation scope and objectives

The objectives of the evaluation were to address the key issues of relevance and performance, and to provide program management with additional information for decision-making. The evaluation covered the five-year period from 2014-15 to 2018-19, and part of 2019-20.

Evaluation questions

The evaluation addressed the following questions:

Relevance

1. Is there a continuing need for RSIP's contributions to support eligible projects?
2. Is there a continuing demand for RSIP contributions and grants?
3. Are RSIP's objectives linked with (i) federal government priorities / responsibilities and (ii) departmental results framework (DRF)?

Performance

1. What has been the impact of RSIP to date?
2. To what extent were the desired outcomes produced between 2013 and 2018?
3. Is RSIP funding supporting the high risk areas?
4. Does RSIP's design meet the needs of all eligible recipients?
5. Are RSIP's services responsive to recipient needs?
6. To what extent were results achieved efficiently?
7. Are project selection procedures robust (e.g., evaluating applications)?
8. What are the barriers to complete projects according to plan?
9. What lessons have been learned over the merger of GCCP, GCIP, and OL?

Evaluation approach

This evaluation focused on the expected outcomes of the program as stated in the program's foundational documents. It used both qualitative and quantitative research methods.

Evaluation and Advisory Services (EAS) of Transport Canada managed and conducted the evaluation, developed the data collection instruments, undertook the data collection methods, analyzed the data and information, and wrote the evaluation report.

Data collection methods

Multiple lines of evidence were used to address the evaluation questions. The data collection methods included a document and literature review, key informant interviews, administrative and financial data review, and case studies.

Document and Literature Review

A document review was conducted to gain a thorough understanding of the programs and to provide insights into relevance and performance. The review included:

• Program foundational documents (e.g., Terms and Conditions, Contribution Agreements);
• Government priority-setting documents (e.g., Budgets, Mandate Letters, Speeches of the Throne); and
• Other key documents (e.g., industry reports, annual reports, previous evaluation reports, academic literature).

Key Informant Interviews

The objective of the interviews was to gather in-depth information related to the relevance, performance, and design and delivery questions. The interviews were semi-structured in nature to help collect qualitative information from a range of stakeholders. Interviews were conducted either in-person or by telephone, as required. Where appropriate, group interviews were conducted.

In total, the evaluation included 30 interviews from the following groups:

• Internal stakeholders (24);
• Recipients/industry experts (5); and
• Industry Associations (1).

Administrative and Financial Data Review

The administrative and financial data review addressed questions related to relevance, performance, and efficiency and economy. Examples of data reviewed included program performance data, and human resources and financial data.

Case Studies

The evaluation conducted six (6) regional case studies with recipients, covering eleven (11) projects to address relevance, performance, and design and delivery issues. Case studies were project-focused and selected across different recipient types, project types, and regions. Each case study involved a review of the relevant project files and a telephone interview.

Limitations

Performance Data

As indicated earlier, this is the first evaluation of RSIP and covers the five previous fiscal years from 2014-15 to 2018-19. Given the early stage of program maturity, RSIP-specific performance is only available from 2016-17 onwards, limiting availability, consistency and comparability of performance data (e.g., administrative data, financial data), Where possible, to mitigate the limitations for the evaluation, findings were triangulated and validated with other lines of evidence.