Table of Contents
- Executive Summary
- Program Profile
- About the Evaluation
- Evaluation Findings: Relevance
- Evaluation Findings: Performance
- Conclusions and Recommendation
- Management Action Plan
Executive Summary
The evaluation of Transport Canada’s support for the Canadian Transportation Research Forum (CTRF) Scholarship Program was conducted to assess the Program’s relevance and performance, as required by the Financial Administration Act and the Treasury Board Policy on Evaluation.
Transport Canada provides an annual contribution to the CTRF to support scholarships for transportation sector research. The objective is to encourage transportation research in Canada and support capacity building in the sector.
Between 2004 and 2014 Transport Canada supported 39 scholarships for graduate students at the Master’s and Doctorate level conducting research in transportation-related fields. The program was most recently renewed in 2012-13 for five years with an annual budget of $24,000, which funds four scholarships.
At Transport Canada, the contribution with the CTRF is administered by the Environmental and Transportation Initiatives Branch of the department’s Programs Group. Transportation and Economic Analysis Directorate within Policy Group provides strategic direction and reviews proposed scholarship awards to ensure they are aligned with departmental themes. The CTRF is responsible for assessing scholarship applications and distributing the scholarships.
Evaluation Scope and Approach
The evaluation of the CTRF Scholarship Program was conducted in the summer of 2015 and assessed the Program since its inception in 2004 to 2014, as the program had never been evaluated.
The evaluation was based on five interviews (with Transport Canada officials and CTRF representatives), a telephone survey of scholarship recipients (n=14), a review of program documents and an internet scan.
Major Findings, Conclusions and Recommendation
The evaluation found there is a high demand for CTRF’s scholarships, and there are few organizations providing scholarships to bolster Canada’s transportation research capacity. There may be an opportunity to refine the scholarship selection criteria to ensure a consistent alignment between the research it supports, Transport Canada’s mandate, and sectoral research needs/priorities.
The CTRF has achieved its intermediate outcome of contributing to increased transportation-related research capacity and knowledge. Recipients produced a significant number of academic papers and conference presentations, and many recipients said the scholarship had a direct influence on their academic output. Over the longer-term, scholarship recipients have gone on to produce a significant body of academic work, including more than 50 academic papers which have been cited more than 1,000 times. The vast majority of surveyed former recipients have continued on to jobs in a transportation-related field and most said the scholarship had at least some influence on their decision to pursue a career in the sector.
The Scholarship Program appears to be well-administered with clear application criteria and straightforward performance measures. Given that all funding flows to scholarship recipients and Transport Canada has minimal administrative costs, the program is highly economical. When asked how the program could be improved, scholarship recipients saw an opportunity for increased participation of recipients in CTRF’s annual conference. CTRF should consider creating greater synergies between the Transport Canada scholarships it awards and its annual academic conference, such as through waiving the conference fee for scholarship recipients or organizing a special panel for recipients to present their work.
The evaluation contains one recommendation:
- Transportation and Economic Analysis should review the scholarship selection criteria to ensure a consistent alignment between the research it supports, Transport Canada’s mandate, and sectoral research needs/priorities.
Program Profile
Background, Objectives and Expected Results
Transport Canada provides an annual contribution to the Canadian Transportation Research Forum (CTRF), a not-for-profit organization of Canadian transportation professionals, to support scholarships for transportation sector research. The objective of these contributions is to encourage transportation research in Canada and support capacity building in the transportation sector.
The scholarships are awarded to graduate students at the Master’s and Doctorate level studying in transportation-related fields at a Canadian university under the following themes:
- Economics, Efficiency and Competitiveness in Transportation
- Sustainable Transportation, and
- Transportation Safety and Security.Footnote 1
At the outset of the Program in 2004, it offered three scholarships per year valued at $4,000 each. Since the program’s most recent five-year renewal in 2012-2013 the program has offered four scholarships per year, each with a value of $6,000. The Program provided a total of 39 scholarships between 2004-2005 and 2014-2015.
Additional CTRF scholarships have also been supported by industry. For the past eight years, Canadian National Railway and Canadian Pacific Railway have each supported a CTRF scholarship each year.
The Scholarship Program is expected to achieve the following results:
- Immediate outcome: Awarding of four transportation-related scholarships, and
- Long-term outcome: Increased transportation research capacity and knowledge.
Students interested in applying for a scholarship must prepare a 300-word statement outlining how their program of research addresses one of the scholarship themes, submit an official transcript, and provide two letters of reference. The CTRF assesses the applications against established criteria with the assistance of a three-member committee comprised of academics and transportation experts from the private sector and government. The scholarship recipients are announced at the annual CTRF Conference.
To be eligible for the scholarships, students cannot be recipients in the year of their award of other major research awards from the Social Sciences and Humanities Research Council (SSHRC) or the Natural Sciences and Engineering Research Council (NSERC).
Program Management and Resources
At Transport Canada, the contribution for funding the Scholarship Program is administered by Environmental and Transportation Initiatives Branch of the department’s Programs Group. Transportation and Economic Analysis Directorate within Policy Group provides strategic direction and reviews proposed scholarship awards to ensure they are aligned with departmental themes. The CTRF is responsible for assessing scholarship applications and distributing the scholarships.
The program has an annual budget of $24,000 ($120,000 over five years), all of which goes to scholarship recipients. The CTRF covers the costs of administering the scholarships. Table 1 provides a breakdown of Transport Canada’s contributions since program inception.
| Year | # of Scholarships
Awarded |
Funding |
|---|---|---|
| 2004 | 3 | $12,000 |
| 2005 | 3 | $12,000 |
| 2006 | 5 | $20,000 |
| 2007 | 3 | $12,000 |
| 2008 | 3 | $12,000 |
| 2009 | 4 | $24,000 |
| 2010 | 3 | $12,000 |
| 2011 | 3 | $18,000 |
| 2012 | 4 | $24,000 |
| 2013 | 4 | $24,000 |
| 2014 | 4 | $24,000 |
| Total | 39 | $194,000 |
About the Evaluation
Evaluation Rationale and Scope
Transport Canada’s Evaluation and Advisory Services conducted the evaluation of the department’s support for the CTRF Scholarship Program between August and September 2015 to assess its relevance and performance. The evaluation was required under Section 42.1 of the Financial Administration Act and the Policy on Evaluation.
Since this is the first evaluation of the Scholarship Program since Transport Canada began funding the scholarships, it examined the program from its inception in 2004-2005 through 2014-2015.
Evaluation Methodology
The evaluation included the following lines of inquiry:
- Interviews were conducted with five key informants, including two CTRF representatives and three Transport Canada officials.
- A telephone survey was conducted with 14 scholarship recipients.
- A document review examined key foundational, background, contextual, and program operational documents.
- An internet scan was used to review similar scholarship programs available to support transportation sector research. Google Scholar was used to search for papers published by scholarship recipients and assess their impact through academic citations.
The following considerations and limitations influenced the conduct of the evaluation.
- Since a Performance Measurement Strategy was first developed for the program following its most recent renewal in 2012-2013, performance data were only available for the years 2012-2013 to 2014-2015.
- Fourteen of the 37 scholarship recipients agreed to participate in the telephone survey representing 38% of recipients.Footnote 2 While the response rate is considered good for a survey of this kind, it is not known to what extent the results reflect the views of all scholarship recipients.
Evaluation Findings: Relevance
To assess the relevance of the CTRF Scholarship Program, the evaluation examined the continuing need for the initiative, and alignment with federal government priorities, departmental strategic outcomes, and federal roles and responsibilities.
Continuing Need
There appears to be strong demand for CTRF scholarships as, on average, the CTRF received nearly five applications for every scholarship awarded since 2004. Most scholarship recipients surveyed said they learned about the scholarship through word of mouth from their faculty supervisors or colleagues.
Since transportation research can include many different disciplines—from civil engineering to psychology—statistics on the number of students pursuing transportation-related research were not available. Award recipients and CTRF representatives told evaluators that graduate students struggle to obtain funding for their transportation research. Evaluators found that the CTRF Scholarship Program is one of the few programs that provides funding targeted to the sector.
Another organization that provides significant support for transportation sector research is the industry-led Transportation Association of Canada (TAC). It established a foundation in 2003 to provide scholarships at the undergraduate and graduate level to address “shortages in the number of skilled people available for planning, design, construction and maintenance of Canada’s transportation infrastructure.”Footnote 3 In 2015, TAC provided 19 scholarships to graduate students valued between $4,000 and $5,000 each.
Other organizations, such as the Canadian Association of Transportation Engineers and the Canadian Institute of Planners, also provide smaller scholarships of one or two thousand dollars each. As well, a number of other transportation sector scholarships are linked to specific university programs, often in transportation engineering.
The CTRF scholarships are not targeted to specific research areas or questions. CTRF scholarship applications must only demonstrate how research falls under the general themes of: economics, efficiency and competitiveness in transportation; sustainable transportation; and transportation safety and security.
It is not clear that the CTRF scholarships are supporting the disciplines of highest need for transportation research funding. Both the CTRF Scholarship Program and TAC have recently provided the largest share of their scholarship funding to the same discipline, civil engineering. More than a third (36%) of CTRF Scholarship recipients, and two-thirds of TAC’s 2015 graduate scholarship recipients, were conducting research in civil engineering.Footnote 4 Moreover, a quarter of surveyed CTRF scholarship recipients had also received TAC scholarships, most for civil engineering research. Some recipients and Transport Canada officials expressed the view that there was generally more scholarship funding available for graduate students pursuing civil engineering than for other areas of transportation research such as business and economics.
| Discipline | # of Scholarships | % of Scholarships |
|---|---|---|
|
Data Source: Descriptions of Scholarship Awards on CTRF website
|
||
| Civil Engineering | 14 | 36% |
| Planning, Transportation Planning & Urban Planning | 9 | 23% |
| Business and Economics | 4 | 10% |
| Geography | 4 | 8% |
| Psychology | 3 | 8% |
| Mechanical Engineering | 2 | 5% |
| Information Technology | 1 | 2% |
| Science | 1 | 2% |
| Political Science | 1 | 2% |
| Total | 39 | 100% |
There is a perception among some Transport Canada officials interviewed that the program has not sufficiently addressed a shortage of research on transportation economics. Only 10% of CTRF awards went to researchers in the area of transportation economics and business and less than a quarter of scholarships were awarded under the theme of “Economics, Efficiency and Competitiveness”, as shown in Table 3.
| Research Theme | # of Scholarships | % of Scholarships |
|---|---|---|
| Transportation Safety and Security | 16 | 41% |
| Sustainable Transportation | 14 | 36% |
| Economics, Efficiency and CompetitivenessFootnote 5 | 9 | 23% |
| Total | 39 | 100% |
Further, about 15% of scholarships funded research examining urban issues such as urban design or cycling policy, which are more directly aligned to the mandates of municipal and provincial governments than that of Transport Canada.
Refining the scholarship selection criteria to encourage research in areas most relevant to sector needs and Transport Canada’s mandate would help to ensure that funded research is relevant. Targeting of scholarship awards to specific research priorities was suggested by some scholarship recipients interviewed for the evaluation as a potential means to improve the program. It was noted by Transport Canada staff that, although there was a desire when the program was last renewed to tailor the program to target specific research priorities, the program’s Terms and Conditions were not modified to permit this.
Alignment with Federal Roles and Responsibilities, Priorities and Departmental Strategic Outcomes
Transport Canada’s support for the CTRF Scholarship Program aligns with the federal roles outlined in the government’s 2014 science and technology strategy, Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation. One of the pillars of that Strategy is strengthening “the skills and capacity that keep Canada at the forefront of research and innovation.” Footnote 6 The federal government’s commitment to supporting research and innovation was also stated in federal budgets between 2011 and 2015.
The Scholarship Program’s objectives of supporting transportation sector research and capacity building align with Transport Canada’s Strategic Outcome of “an efficient transportation system”, through the sub-activity of transportation innovation. The three research themes supported by the scholarships align with Transport Canada’s three Strategic Outcomes of transportation efficiency, safety and security, and sustainability.
Evaluation Findings: Performance
Effectiveness – Achievement of Expected Outcomes
The effectiveness of the program was assessed by examining progress in achieving its immediate outcome of awarding scholarships and intermediate outcome of increasing transportation-related research capacity and knowledge. Given the program has been in operation for more than 11 years, the evaluation also assessed its longer-term impact by examining the academic output of former scholarship recipients.
Between 2004 and 2014 the Scholarship Program awarded 39 scholarships, 24 to Doctoral students and 15 to Master’s students. The research conducted by these students encompassed a wide range of subject areas from transportation supply chain management to driver aggression, and airport security to traffic management with intelligent transportation systems.
Evidence showed the program has achieved its expected outcome of contributing to increased transportation-related research capacity and knowledge through influencing students’ decision to pursue such research, increasing the amount of research, and contributing to other outcomes.
Influence on transportation-related research
The recipient survey showed that while most recipients had already chosen their research topic when they applied for the scholarship, 28% said the award influenced their decision to pursue transportation-related research. (Table 4)
| n=14 | ||
|---|---|---|
| To no extent | 10 | 71% |
| To some extent | 2 | 14% |
| To a great extent | 2 | 14% |
Increased transportation-related research
About 80% of recipients either published papers or made presentations in the year they received their scholarship.Footnote 7
In the three years since the last program renewal (2012-2013 to 2014-2015), scholarship recipients:
- had nine papers published or accepted for publication; six of those were in peer reviewed journals including the Journal of the Transportation Research Board, Journal of Transportation Engineering, Canadian Journal of Civil Engineering, Transportation and Environment, Research in Transportation and Business Management, and Urban Studies; and
- made 20 presentations at academic and industry conferences including the Canadian Association of Planning Students, World Conference on Transportation Research, Transportation Association of Canada Conference, Association Québécoise de Transport, Road Safety and Simulation International Conference, Veloworks Cycling Society and the CTRF conference.
Fifty percent of the scholarship recipients surveyed said that the Transport Canada-supported CTRF scholarship had a direct influence on the number of presentations or papers they produced, either by helping to pay for data, allowing time for additional research, or motivating them to continue their work through the ups and downs of the research process. According to one recipient, “It was a big motivation when I got the award. It helped me understand the importance of the problem I was considering.”
An examination of the academic output of CTRF scholarship recipients over the longer-term showed they have gone on to produce a notable body of academic work in the transportation sector. A Google Scholar search showed that researchers supported by CTRF have published more than 50 academic papers on transportation-related issues and these works have been cited more than 1,000 times.Footnote 8
Other outcomes
The scholarships have also contributed to other outcomes. For 29% of recipients surveyed, the CTRF scholarship acted as a springboard for obtaining further awards and funding. A couple of recipients told evaluators that the scholarship allowed them to spend extra time on their research or open up new research areas, which contributed to successful applications for major funding awards from SSHRC.
In two cases recipients used their research to produce reports for provincial Ministries of Transportation. One report was used by transportation decision-makers to select specific intersections and roadways for safety improvements. The other report provided guidelines for safe and sustainable pavement designs.
{ATIP REMOVED}
Almost all (90%) of scholarship recipients surveyed who had completed their studies were working or pursuing employment in a transportation-related field. Three-quarters (77%) of those currently working in transportation said the scholarship influenced their decision to pursue a career in the sector (see Table 5). Four former recipients were working at academic institutions directing major research projects or as assistant professors, while three others were working for provincial or municipal transportation authorities.
| n=9 | ||
|---|---|---|
| To no extent | 2 | 22% |
| To some extent | 5 | 55% |
| To a great extent | 2 | 22% |
Overall, all of recipients surveyed felt the CTRF Scholarship Program was effective or very effective in encouraging research and careers in transportation-related fields.
Scholarship recipients surveyed said they thought the program was well-administered by CTRF and interviewees raised no concerns about program administration. Recipients said they found the application process and selection criteria to be clear.
When asked how the scholarship could be improved, a quarter of recipients surveyed suggested that CTRF create a better link between scholarship recipients and its annual conference. While a number of the recipients surveyed had attended or made presentations to CTRF conferences during their career, only two made presentations at the conference in the year following their award. Recipients suggested that attending the conference or having the opportunity to present their work would support the capacity building objectives of the award. They said it would be helpful if CTRF could waive the conference fee for scholarship recipients or if a special panel could be arranged for award recipients to obtain feedback on their research.
Transport Canada first developed a Performance Measurement Strategy for the program following program renewal in 2012. Since then, CTRF has fulfilled the reporting obligations in its Contribution Agreement to provide information on the number of research papers published and number presentations made at conferences by scholarship recipients. This reporting is appropriate for a contribution program of the size and nature of the CTRF Scholarship Program.
Efficiency and Economy
The CTRF Scholarship Program has demonstrated a high degree of economy and efficiency given its negligible administrative costs. CTRF assumes the administrative cost of selecting the scholarship recipients, distributing the scholarships and reporting to Transport Canada so that all contribution funding goes directly to students. At Transport Canada the time required to manage the claims and monitor the Contribution Agreement is estimated to be 0.1 of a full-time equivalent.
Conclusions and Recommendation
The evaluation found there is a high demand for CTRF’s scholarships, and there are few organizations providing scholarships to bolster Canada’s transportation research capacity. There may be an opportunity to refine the scholarship selection criteria to ensure a consistent alignment between the research it supports, Transport Canada’s mandate, and sectoral research needs/priorities.
The CTRF has achieved its intermediate outcome of contributing to increased transportation-related research capacity and knowledge. Recipients produced a significant number of academic papers and conference presentations and many recipients said the scholarship had a direct influence on their academic output. Over the longer-term scholarship recipients have gone on to produce a significant body of academic work, including more than 50 academic papers which have been cited more than 1,000 times. The vast majority of former recipients have continued on to jobs in a transportation-related field and most said the scholarship had at least some influence on their decision to pursue a career in the sector.
The Scholarship Program appears to be well-administered with clear application criteria and straightforward performance measures. Given that all funding flows to scholarship recipients and Transport Canada has minimal administrative costs, the program is highly economical. When asked how the program could be improved, scholarship recipients saw an opportunity for increased participation of recipients in CTRF’s annual conference. CTRF should consider creating greater synergies between the Transport Canada scholarships it awards and its annual academic conference, such as through waiving the conference fee for scholarship recipients or organizing a special panel for recipients to present their work.
The evaluation contains one recommendation:
- Transportation and Economic Analysis should review the scholarship selection criteria to ensure a consistent alignment between the research it supports, Transport Canada’s mandate, and sectoral research needs/priorities.
Management Action Plan
| Recommendation(s) | Proposed Action | Forecast Completion
Date |
OPI |
|---|---|---|---|
|
Transportation and Economic Analysis should review the scholarship selection criteria to ensure a consistent alignment between the research it supports, Transport Canada’s mandate, and sectoral research needs/priorities. |
Accepted. Transportation and Economic Analysis (TEA) acknowledges the need for the review of the scholarship selection criteria. TEA has been engaging with internal and external partners, such as Canadian universities, on the development of a research action plan that will provide a long-term vision and identify strategic research priorities and outcomes in alignment with Government priorities and commitments as well as with transportation sector research needs. TEA will communicate to the CTRF its action plan priorities to inform the scholarship selection criteria, to foster relevant transportation-related research capacity and expertise in Canada. TEA will suggest to the CTRF that the revised research criteria for the scholarship program be applied to the next 5-year agreement (2017-2018/2021-2022) of the CTRF scholarship program. |
March 2017 |
Policy |