Evaluation of Clean Transportation Systems Portfolio (2007-08 to 2011-12)

Table of Contents

Acronyms

Executive Summary

Introduction:

This report presents the findings of an evaluation of NRCan’s Clean Transportation Systems (CTS) Portfolio of programs, part of the Clean Transportation Energy Sub-Activity (2.1.3.2 in the 2011-12 Program Activity Architecture). The Portfolio is administered by NRCan’s Office of Energy Research and Development (OERD), with various components led by NRCan’s Innovation and Energy Technology Sector (IETS), CANMET Materials Technology Lab (MTL), the National Research Council (NRC) and Environment Canada (EC). The evaluation assessed the relevance (need and NRCan role) and performance (effectiveness, efficiency and economy) of program activities from 2007-08 to 2011-12 for the five component programs totalling $63.7  million in NRCan-administered funds over five years. These include: 

• Hydrogen and Fuel Cells (H2FC; $29.1 million/46%): The goal of this program, created in 2003-04 and led by NRCan’s Innovation and Energy Technology Sector (IETS), is to expand the knowledge base and advance hydrogen and fuel cell technologies.
• Electric Mobility (EM; $9.2 million/14%): A new program beginning in 2007-08 and led by the National Research Council (NRC), the goal of the first five years of this program is to support the development of plug-in hybrid electric vehicles (PHEV).
• Advanced Fuels and Technologies for Emissions Reduction (AFTER; $11.1 million/17%): The goal of this program, created in 2000-2001 and led by NRC, is to support the design and use of conventional and new hydrocarbon fuels, alternative transportation fuels, and novel combustion technologies, and associated technologies able to reduce vehicle emissions and fuel consumption.
• Advanced Structural Materials for Next-Generation Vehicles (ASM-NGV; $4.8 million/8%) The goal of this program, created in 1999-2000 and led by NRCan’s CANMET MTL, is to develop conventional and new materials, structural components and vehicle subsystems that can be used on virtually all types of next-generation vehicles enabling them to reduce their weight, and improve their crashworthiness and overall fuel efficiency.
• Particles and Related Emissions (P&E; $9.5 million/15%): The goal of this program, created in 2002-03 and led by Environment Canada (EC), is to strengthen the scientific basis for policy and regulatory decisions affecting transportation-related emissions of particles and related atmospheric constituents.

This executive summary focuses mainly on the Portfolio level analysis, drawing on specific program examples to illustrate key findings. More details on the performance of individual programs are provided in the body of the report.

Evaluation Methodology:

All programs in this portfolio had been evaluated in 2007-08, with the exception of EM. The current evaluation used a calibrated risk-based approach that placed extra emphasis on H2FC and EM based on the risk-assessment performed as part of the plan for this study. H2FC was considered higher risk because of its high materiality (45% of NRCan’s CTS investment) and EM due to its newness (created in 2007-08) and therefore having never been evaluated before.

The evaluation study employed a theory-based approach known as contribution analysis, which examined and tested the theory of change behind the programs. The evaluation used multiple lines of evidence including document and administrative data review; 14 case studies; 71 interviews; and a bibliometric analysis.

The fieldwork for this evaluation study was conducted as Federal Budget 2012 was being delivered and implemented across funding partners, including NRCan, EC, Health Canada (HC) and NRC. This had potential to bias interviews as programs were experiencing the effects of reprioritizing within government. The evaluation methodology addresses this potential bias through taking a multiple lines of evidence approach and seeking confirmation of identified issues across lines of evidence rather than single sources where possible.

Key Evaluation Findings

Relevance:

Overall, evidence from the document review, interviews and case studies suggests that there is a clear and supported rationale for the federal government and NRCan to be involved in CTS programming.

Ongoing sector needs for CTS R&D in line with Federal government and NRCan strategic priorities: The CTS programs are generally consistent with federal government priorities and NRCan strategic objectives. The importance of the auto sector in Canada, the impact of the transportation sector on the environment, and harmonization of transportation regulations with the US make CTS an important, legitimate area for federal support.  The development of electric vehicles, fuel cells, alternative fuels and lightweight materials support Canada’s competitiveness and environmental objectives.

There is a legitimate role for the federal government to support basic and applied R&D (especially higher risk pre-commercialization R&D found in CTS) to improve cost-competitiveness of new technologies and processes, and to support development of codes, standards and related infrastructure. There is also a significant role to be played representing Canadian perspectives and industry in international fora and committees (e.g., within the International Energy Agency (IEA), Clean Energy Dialogue, US Department of Energy (DOE), and International Partnerships for Hydrogen and Fuel Cells in the Economy (IPHE)). 

OERD role in CTS is appropriate, but concerns raised over CTS priority setting within PERD and consistency of project selection processes: CTS programs are primarily funded through the Program of Energy Research and Development (PERD), an interdepartmental R&D program administered by NRCan’s OERD. The CTS role for NRCan in relation to other actors (e.g., other government departments) has been well co-ordinated and worked well in most areas.  However, with respect to OERD’s role as funding allocator, there is a widespread perception across CTS programs that the research mandates of other departments were not adequately considered.

According to OERD, as PERD funding was reduced during the 2007-08 to 2011-12 funding cycle, a narrower interpretation of PERD was taken to focus more on programs directly related to energy production, distribution and use, as opposed to health and the environment as was done in the past.

There was also a perceived lack of transparency in some project selection processes. While stakeholders held positive perceptions of the project selection processes in AFTER, ASM-NGV, and EM, they voiced concerns over project selection for H2FC in the beginning of the funding cycle. The lack of consistency in project selection processes and the use of a narrower interpretation of PERD eligibility created a perception that projects closer to NRCan’s mandate are favoured for funding over those of OGDs.

Performance (Effectiveness):

The evaluation found evidence across interviews, case studies, document review, and bibliometrics that the five Programs have, in general, supported their planned outcomes and made good progress on immediate and intermediate outcomes. Given the focus on applied research, it is still too early to show longer term commercial outcomes for most program investments. CTS planning documents suggest that even early ‘commercial’ outcomes were not expected in the Portfolio until 2015. Notwithstanding the early stages of expected development, progress towards longer-term outcomes, such as the adoption of new clean transportation technologies and greenhouse gas (GHG) reductions were observed in case studies.  Promising incremental changes and advances in technologies with commercial potential were also shown. The R&D conducted by these programs has also made important contributions to regulations, codes and standards which drive broader industry R&D.

Areas of significant progress include:

Increased collaboration and cooperation on international research goals, and strong engagement of key stakeholders in most cases:EM engaged a wide variety of stakeholders during its planning phase, involving technical experts in project selection. Projects entailed partnerships with several private sector firms involved in PHEV manufacture, electrical utilities and researchers from government departments. Similarly, the level of industry participation in AFTER as well as input from the AFTER Management and Technical Committee have helped to ensure that projects reflect market realities and address key technical issues. ASM-NGV is well connected to Canadian industry (with projects involving materials manufacturers, part suppliers and original equipment manufacturers), related research programs in the US, and international collaborative efforts (e.g., Magnesium Front End R&D project) involving major companies in Canada, the US, and China.

AFTER has engaged researchers and policy groups from federal departments involved in transportation-related emissions and fuel efficiency in program planning, project selection and carrying out projects. One of the most important outcomes of P&E has been the research network created among EC, HC and NRC on emissions research, giving HC and EC researchers opportunities to focus on transportation sector emissions and their impacts, leading to better data and predictive models that would not have otherwise been developed.

H2FC helped maintain research linkages among industry and government labs, and promoted international cooperation through participation in IEA and ISO Committees on Hydrogen and Fuel Cells. In the last four years of the program as funding declined, there was a lack of formal industry involvement in H2FC through an industry committee. However, industry views continued to be sought, albeit with less frequency, through H2FC participation in the annual Canadian Hydrogen Fuel Cell Conference, and IEA and IPHE meetings. 

Good R&D progress on clean transportation systems and technologies at applied research levels (including fuel cells, engines, and fuels), with some early commercial applications: H2FC contributed to fuel cell technology development within industry (e.g., membrane technologies), enhanced public sector capacity for testing (e.g., new fuel cell testing stations at NRC’s Industrial Materials Institute), and small-scale hydrogen fuel cell applications (e.g., stationary, back-up power, and material handling applications). The majority of EM projects involved technology development with at least one project that has led to commercial application, and technology developed in another is expected to be used in a next generation manufactured product. AFTER projects have also advanced the development of several engine technologies associated with emissions control, including several patents and in one case licensing and commercial production (i.e., prototype combustion instability sensor). Basic pre-competitive ASM-NGV R&D projects have addressed key knowledge gaps for select lightweight materials (e.g., magnesium, aluminium, and ultra high strength steel) in structural applications and under demanding automotive operating conditions (e.g., crash, vibration, corrosion, etc.). ASM-NGV has also helped to develop unique materials modelling capacity at Canmet-MTL which is necessary for new materials development but generally absent within industry. 

Improved understanding of the potential impacts of new fuels and systems on emissions (environmental and health impacts): Both AFTER and P&E have been particularly focused on this, often engaging in joint projects. AFTER projects have produced extensive emissions and fuel efficiency related data for a number of engine technologies and conventional and bio-fuel blends that have been provided to Canadian, US and international regulatory stakeholders. Similarly, the data and new knowledge generated by P&E projects have strengthened the basis for decision making regarding environmental, transportation and health policy, and regulations (e.g., new technology to determine toxicological effects from engine exhaust of biodiesel blends). 

Significant contributions to codes and standards and other regulatory applications (including input to harmonization discussions with the US): Industry interviewees noted that it is the regulatory, environmental and fuel efficiency standards that drive much of industry’s R&D and adoption of new technologies in the automotive sector. H2FC R&D and participation on international committees has impacted national and international codes and standards. For example, in 2007-08 the Program provided funding to the Bureau de normalisation du Québec’s Canadian Hydrogen Installation Code (CHIC) technical committee to support revisions and amendments to the code, and to implement the code for new hydrogen installations. The CHIC, originally created in January 2007 with H2FC support, provides Canadian industry and regulatory authorities with guidance for approving hydrogen as an energy carrier and facilitating approval of hydrogen installations across Canada.

Policy and regulatory applications of EM work were also evident. One project supported harmonization of regulations with the US by providing performance data on hybrid electric vehicles to the US Environmental Protection Agency (EPA). Another supported the information needs of electric power utilities and governments on the effect of PHEVs on the electrical power distribution system by producing reports and software that are being used to understand the implications of widespread introduction of PHEV`s. AFTER and P&E data are being used by HC, EC, Transport Canada and NRCan and has been provided to the US EPA and US DOE to inform development of policies and regulations related to emissions and emerging fuels. That said, a number of P&E program stakeholders believe that use of project results by Canadian policy groups could be improved through disseminating results more broadly and integrating them with other related CTS results. 

These achievements notwithstanding, there are several findings that present challenges for continued success of the CTS Portfolio of programs:

Dissemination of results has been generally good, although project technical results remain mostly confined to participants: EM and AFTER have demonstrated good dissemination of results beyond their project participants through conferences, publications that are highly cited and workshops with policy and regulatory groups. The bibliometrics analysis confirmed that a total of 839 CTS-funded publications were cited at least on par with and often above the world level in the respective five research fields (H2FC produced 428 publications, cited 88% above the world average; ASM-NGV produced 140 publications, cited 67% above the world average; AFTER and P&E combined produced 235 papers, cited 26% above the world average; and EM produced 75 publications, cited 5% above the world average). However, research publications notwithstanding, stakeholders within the ASM-NGV, H2FC, and P&E programs expressed concern that there was not a way for OERD to better disseminate results to industry beyond their project participants. ASM-NGV and H2FC results have primarily been applied by project participants often leading to further R&D projects. Several industry and public sector interviewees suggested that OERD could play a more significant role in communicating these results within the sector and across the CTS portfolio. Unless they are active participants in CTS sub-program management committees, industry interviewees noted that they do not learn about research findings and are not aware of any on-going OERD or CTS Portfolio efforts to summarize and disseminate research findings and directions from across the CTS portfolio.  

Many external economic factors have had significant impacts on the achievement of expected results over the last 5 years: CTS programs operate in a very complex and fluid environment. Industry health and economic conditions impacted industry support for R&D. This was especially so for the fuel cell sector where progress has been slow and industry ability to fund R&D has been declining. Similarly, a number of ASM-NGV partner companies were not able to meet their original commitments to project activities as a result of the economic downturn in 2009. The varying price of fossil fuels has also affected the demand for fuel-efficient vehicles.  The extent to which technologies developed through ASM-NGV, H2FC, and EM will contribute to a sustainable transportation sector, with net economic benefits to Canadians, will depend on commercial and market factors including new regulations, the price of oil, and competitiveness of the Canadian auto sector. 

Applied R&D results from H2FC and EM programs will require the support of broader regulatory and infrastructure projects to achieve widespread adoption:The H2FC and EM programs have pursued important incremental innovations but are, by themselves, unlikely to significantly influence growth of the use of hydrogen or PHEVs. A broader project to develop the technical infrastructure and regulations is needed to complement the research being carried out. For example, the challenges in implementing the H2FC Airports Demonstration project illustrated the importance of regulatory and infrastructure development (codes and standards) needed to complete an effective hydrogen fuel cell demonstration. As well, the EM Program is just one element of a comprehensive EM support program as outlined in the 2009 Electric Vehicle Technology Roadmap. Several interviewees were concerned about the lack of action on government regulations and support for infrastructure (e.g., charging stations). Without the other elements, the relatively small EM R&D program can make only limited progress.

Performance (Efficiency and Economy):

At the individual program level, the CTS programs have been managed efficiently and economically. In general, interviewees felt that the programs were well managed, and involved key partners in all aspects of program design and delivery, and were cost-effective. 

Key factors supporting the efficiency and economy of CTS programs include:

Effective use of partnerships and leveraging of funds: CTS Portfolio funding successfully leveraged financial and in-kind resources from others at a ratio of 1:1.5 (ranging from 1:1.2 for EM to 1:3.3 for ASM-NGV). The programs have effectively used federal research capacity in both expertise and facilities at NRCan, NRC, EC and HC. As well, the close alignment of CTS work with international R&D priorities has also created efficiencies by helping to focus research priorities and attract and leverage partnerships.

Maturity of CTS programs and delivery mechanisms: The maturity of most CTS programming has facilitated efficient and cost-effective implementation of projects. ASM-NGV, AFTER and P&E, all over 10 years old, have developed and leveraged good partnerships and relationships with external stakeholders, building on the achievements of previous work. In fact, the relatively new EM program has been able to take advantage of existing relationships developed by AFTER and P&E to quickly and efficiently build relationships with US and international partners. Furthermore, AFTER, P&E, and ASM-NGV have been consistent in their objectives over this time which helped to focus research efforts.  The use of PERD as the main funding vehicle for all CTS programs has also been highly effective because of its maturity, having developed efficient, effective procedures that are well known and understood by the federal R&D community.

There are still opportunities for improving economy and efficiency of individual CTS programs and at the Portfolio overall.

Missing federal strategy to guide CTS R&D funding decisions: Interviews, documents, and case studies all suggest that the long-established OERD-PERD process has worked comparatively well in most areas reviewed (not withstanding the complex programming situation). However, missing from the CTS portfolio level discussion is a relationship to a federal strategy or guide for transportation R&D that identifies the criteria or rationale for funding allocations to each component program/activity in the context of what other federal initiatives are pursuing. The number of programs in other departments with an active interest in CTS research areas may have introduced some confusion among stakeholders (e.g., industry, academia, and OGD researchers) with respect to federal R&D priorities and federal strategies for clean transportation systems, which creates a risk of inefficiencies in funding.

Evidence from interviews and case studies suggests that both the H2FC and EM programs could benefit from pursuing greater integration with other efforts in Canada. For example, while ASM-NGV has strong and ongoing linkages to other federal initiatives such as the Auto21 network, or Automotive Partnership Canada (APC)^{Footnote 1}, the H2FC Program did not have significant linkages with these initiatives and other federal initiatives.

Recommendations

In light of the above findings, the evaluation makes the following recommendations:

1. NRCan OERD should clarify the CTS Portfolio’s position within the overall federal transportation R&D context by consulting with other federal departments that have related transportation R&D mandates in the planning process and on a regular basis while projects are underway.  Note that policy, codes, standards and regulatory needs should continue to be considered alongside overall energy performance targets, technology development and commercial goals.

Context:

The CTS Portfolio programs are generally consistent with NRCan objectives of energy production, efficiency and use of renewable energy, long-term economic impacts, and more specifically to the Strategic Priority of Clean Transportation Energy. While there are clear linkages between the five programs and federal priorities and objectives, supported in part by the participation of other government departments in program planning and priority settings, there is a need to clarify how the CTS Portfolio can best be part of a federal strategy for clean transportation. The number of programs and activities in other departments with an active interest in CTS research areas may have introduced some confusion among stakeholders (i.e., industry, academia, and OGD researchers and policy makers) with respect to federal R&D priorities and federal strategies for clean transportation systems.  Stakeholders across programs noted that the level of integration and coordination of other government departments in CTS priority setting could be strengthened. 

Interviews and documents suggest that placing the five Programs under the ‘CTS Portfolio’ umbrella created expectations concerning coordination, priority setting and funding allocation at the Portfolio level that were not met. While the evaluation did identify a number of synergies between the program components, these tended to have occurred on an informal, rather than a formal basis. Clarifying the strategic links within the CTS portfolio and between the CTS portfolio and other federal initiatives with an interest in clean transportation R&D will help to better leverage results within the federal system. 

2. NRCan OERD should ensure consistency in project selection processes across all CTS programs so that they are transparent to all program partners.

Context:

Despite PERD being an interdepartmental program, there is a perception among several OGD and NRCan stakeholders that NRCan-led projects may be supported over projects from other departments. These concerns were raised with respect to a perceived lack of transparency in some project selection processes during the 2007-08 to 2011-12 period under review. While stakeholders held positive perceptions of the project selection processes in AFTER, ASM-NGV, P&E, and EM, they voiced concerns over project selection for H2FC in the beginning of the funding cycle. Under H2FC, a number of interviewees noted the limited external review of H2FC projects and compared the CTS process with that used by US DOE, which involves external reviewers and publishes project proposals and assessments online. While this requires resources that may be beyond what is available to OERD, ensuring consistent approaches to project selection processes within CTS programs will help to alleviate concerns among stakeholders over transparency in the use of PERD funds within CTS.

3. OERD should establish a strategy to disseminate the knowledge and transfer the technology progress gained on a project level more broadly across industry and policy makers within Canada.

Context:

Dissemination of results through research publications and conferences has been generally good, and, OERD provided pan-CTS results at the U.S.-Canada Clean Energy Dialogue Discussion Meeting in Washington, DC in February 2011, and at the CTS Portfolio Workshop and Poster Session in Ottawa in June 2009.  However, project technical results remain mostly confined to project participants. Stakeholders within the ASM-NGV, H2FC, and P&E programs expressed concern that there was not a way for OERD to better disseminate results to industry beyond their project participants. Several industry and public sector interviewees suggested that OERD could play a more significant role in communicating these results within the sector and across the CTS portfolio. Unless they are active participants in CTS component program management or technical committees, industry interviewees noted that they do not learn about research findings and are not aware of any on-going OERD or CTS Portfolio efforts to summarize and disseminate research findings and directions from across the CTS portfolio.  

Recommendations, Management Responses and Action Plans

Note: Since the evaluation was completed, OERD has reorganised its energy Portfolios, and the former CTS Portfolio is now called the Transportation Technology Area, which fits under a broader Energy End Use Portfolio. These responses use the new terminology.

Recommendations	Management Responses and Action Plans	Responsible (Target Date)
1. NRCan OERD should clarify the CTS Portfolio’s position within the overall federal transportation R&D context by consulting with other federal departments that have related transportation R&D mandates in the planning process and on a regular basis while projects are underway.  Note that policy, codes, standards and regulatory needs should continue to be considered alongside overall energy performance targets, technology development and commercial goals.	ACCEPTED – To be fully implemented for the next PERD cycle, for which planning will begin in 2014-15. OERD has adopted an improved strategic and operational planning process for new PERD cycles that involves, at the Technology Area level, early engagement of Directors General and equivalents from NRCan sectors and other departments and agencies, and a stakeholder workshop at which the views of both internal (federal) and external stakeholders are heard.  This early engagement provides valuable insight and advice that is broader than just a federal perspective. The Transportation Technology Area continues to fund R&D in support of policy, codes, standards and regulatory needs, as part of a mix of transportation energy R&D.	ADM IETS March 31, 2016
2. NRCan OERD should ensure consistency in project selection processes across all CTS programs so that they are transparent to all program partners.	ACCEPTED – As part of its commitment to continuous improvement of both program management and accountability, beginning in fiscal year 2007-08, OERD has taken measures to improve the consistency and transparency of the project selection process. For consistency, OERD uses a rigorous project proposal and selection process across all of its Portfolios and Technology Areas, including a standard proposal template, standard core assessment criteria (covering relevance, risk and impact), and strict rules on participation or otherwise by federal technical experts on project selection. Invitations to participate in Program planning exercises are issued to Directors General (or equivalents) of all potential participating departments and agencies. Calls for project proposals are managed by OERD and are issued to Directors General for distribution within their organisations. In order to provide a level playing field, a detailed guide to developing project proposals is made available to all project proponents, and project proposal templates provide overall guidance on what is required. Further, the project review and selection process is led by the Office of Energy R&D.	ADM IETS Completed
3. OERD should establish a strategy to disseminate the knowledge and transfer the technology progress gained on a project level more broadly across industry and policy makers within Canada.	ACCEPTED – OERD currently requires researchers to include a dissemination plan for each project proposal submitted for funding and requires that they report on these activities annually. Such activities include peer-reviewed papers published in professional journals and presentations at conferences, which are accessible to industry and policy-makers. OERD will explore, with NRCan corporate sectors, opportunities to broaden further the reach of the knowledge products arising from its funding programs, and will build these into its Portfolio action plans.	ADM IETS March 31, 2015

1.0 Introduction

This report summarizes evaluation findings for the Evaluation of the Clean Transportation Systems (CTS) portfolio managed by the Office of Energy Research and Development (OERD), Natural Resources Canada (NRCan).

The programs in the portfolio include:

• Hydrogen and Fuel Cells (H2FC)
• Electric Mobility (EM)
• Advanced Fuels and Technologies for Emissions Reduction (AFTER)
• Advanced Structural Materials for Next-Generation Vehicles (ASM-NGV)
• Particles and Related Emissions (P&E)

1.1 Evaluation Study Background, Objectives and Scope

The 2012-13 evaluation of the Clean Transportation Systems (CTS) portfolio is being done to inform decision-makers about the relevance and performance (effectiveness, efficiency and economy) of the five programs.  These programs’ last evaluation was in 2008-09 and covered activities between 2002-03 and 2006-07.  This evaluation covers activities between 2007-08 and 2011-12.

This evaluation is unique in that it is the first evaluation of a portfolio of programs that has already been evaluated once before in the Department’s new five-year evaluation cycle.  The 2009 Treasury Board Policy on Evaluation requires 100% coverage of all spending every five years, but also allows for calibration within that coverage.  As a result, this study was designed using a risk-based approach to focus a greater proportion of evaluation efforts on the higher risk CTS components. These were the H2FC component because of its high materiality (46% of NRCan investment in CTS) and EM component because it has never been evaluated before.   

The objectives of this evaluation are to:

• assess or confirm the relevance of Clean Transportation Systems programming in terms of addressing an actual need, in relation to federal government priorities, and in relation to the role of the federal government;
• assess the performance of the Clean Transportation Systems programming in terms of the degree to which it has achieved expected results, and whether there have been any unintended results; and
• assess the economy and efficiency of Clean Transportation Systems programming.

In terms of scope, this evaluation covered NRCan expenditures of approximately $64 million from 2007-08 to 2011-12 (see Table 1 in Section 2).

1.2 Organization of This Report

This report is organized as follows:

Section 2 outlines the methodology and limitations.  Further information on these can be found in the Evaluation Methodology and Workplan Report.

Section 3 describes the CTS Program Context.

Section 4 provides a CTS overview and programs profile.

Section 5 provides a summary and discussion of findings by evaluation issue, for each of the CTS programs.

Annex A lists the case studies used in the evaluation.

2.0 Study Methodology & Limitations

2.1 Study Methods

The evaluation of the Clean Transportation Portfolio of Programs was undertaken using a theory-based approach known as contribution analysis. This approach to evaluation starts with developing an early understanding of the theory of change behind the CTS Portfolio Programs as well as the important contributing factors to the success of these types of programs, and then tested and adjusted these as knowledge was accumulated over the course of the evaluation.^{Footnote 2}  The accumulation of knowledge in this fashion provided generative learning about the key factors contributing to success and what works for whom in what conditions and why.  These frameworks were used to address the core evaluation questions.

The methodology built on the ‘contribution analysis’ approach, as proposed in the recently released (December 2012) TBS guidance^{Footnote 3} and recent evaluation literature, and relied on multiple lines of evidence (interviews, case studies, document and file review, and bibliometric analysis). 

The following is a description of how each of these methods was applied in this evaluation.

Interviews:

A total of 71 individuals were interviewed from May to July 2012. The initial universe of interviewees was identified by OERD, and supplemented as the study team identified other knowledgeable informants. Interviews were selected to ensure a representation of stakeholder groups and project partners.  Of the list of 71 interviewees, 21 were able to provide project-level detail and participated in case study interviews.  This led to multiple consultations with select interviewees, and an overall total of 87 consultations / interviews were conducted for this project.

Select interviews were conducted early in the process to assist with identifying change theories and key literature to review. Most interviews took place after the initial contribution story was determined in order to validate it.

The chart below provides a breakdown of interviews by program and type.

Distribution of CTS Interviews by Type and Program

	CTS Portfolio	H2FC	EM	AFTER	P&E	ASM-NGV	Total1
NRCan	8	2	3	3		2	18
OGD	5	10	4	3	6		28
Industry		3	8	3	1	6	21
Academia				1			1
Other				1		2	3
Total1	13	15	15	11	7	10	71

Case Studies:

Fourteen case studies were completed for this evaluation. Case studies were selected across the five CTS program areas based on the calibration risk assessment (which identified H2FC and EM as priorities), materiality (i.e., size of NRCan investment), intended outcomes (i.e., projects work towards various outcomes), nature of the strategy / policy instruments ‘mechanisms’ involved, and areas of technology application (i.e., covering a sample of the technologies and stage of technology development involved in these programs).

Summary information on the case studies selected can be found in Annex A.

Document and File Review:

The document review included an examination of 245 documents provided by NRCan and others as identified during the interviews. This included five years of project reports, Program Annual Reports, CTS Portfolio reports and financial information for all five CTS programs.

Bibliometric Analysis:

As part of its performance measurement strategy, the five programs keep track of publications resulting from their CTS funded work. Using this information, the evaluation commissioned a separate bibliometric analysis in order to explore the impact of CTS funded researchers and their papers on their particular field worldwide. To facilitate this, a list of CTS publications was compiled based on project reports for each program.  Bibliometric data were produced using scientific publications indexed in the Scopus database (Elsevier). A dataset of scientific papers on CTS research was created using specialized journals and specific keywords related to five program subfields in order to situate CTS funded work with that of others worldwide.

2.2 Study Limitations and Mitigation Strategies

As noted in the Evaluation Assessment and workplan phases, this evaluation faced the following challenges:

The complex nature of R&D activities (timelines, attribution issues, etc.):  In order to address this challenge, the senior evaluation team  included two engineers and leaders in the evaluation field in Canada, each with experience in S&T evaluations, the technologies under review, and the contribution analysis approach to evaluation.  This team understood the complex nature of R&D projects as well as the complex environment in which federal S&T programs are delivered, an environment that includes technology as well as policy, standards and regulatory priorities.  The team had prior experience in working with all CTS partner departments (i.e., NRCan, NRC and EC), and assessing the relevant CTS funding envelopes.

The ‘intricate’ links between policy and regulations and R&D in the transportation sector: The intricate links in the CTS program portfolio were addressed by the approach adapted to this study.  A theory-based contribution analysis was used.  It provided a disciplined approach to assess the assumptions underlying the CTS program logic, and its key ‘mechanisms’ as described in the Evaluation Assessment.  The approach allowed for structured input from key stakeholders and provided a rigorous process for validating and refining understanding of the results logic and the factors effecting performance for both the implementation theory and the broader theory of change.  The assessment of key underlying assumptions (an element of the Evaluation Assessment) was critical to assessing why (or why not) program outputs and outcomes were achieved (e.g. Were there underlying issues with respect to the programs’ design / expectations or were there issues with the governance and delivery of the programs?).

Complicated delivery mechanisms presented challenges to determining attribution:  The CTS portfolio involved multiple departments, multiple funding envelopes for which complete and reconciled financial data were not available, co-funding arrangements, and a wide range of program participants, including federal researchers, universities, industry as well as international actors.  This presented a very difficult challenge for determining attribution. With respect to the attribution challenge, the contribution analysis approach, along with the complementary case study, Qualitative Content Analysis and other data synthesis activities, still did not fully resolve this problem.  In fact, it is unlikely that any approach could do so given the immense complexity of the CTS portfolio initiative.  Given these constraints, this study and report offers the most systematic, structured and disciplined approach available for assessing causality. 

The Bibliometric analysis should be weighed carefully in the context of the other evaluation lines of evidence:Results of the bibliometric analysis, while helpful for determining impacts of published work, should be interpreted with caution for two reasons: 1) there is a lag in publication between completed research and when a paper begins to be cited which means that only papers published between 2007-08 and 2010-11 were included in the analysis; 2) judging performance of a particular CTS program on world citation counts alone would be inappropriate. It is important to take into account the public-private context under which much of the work is done (i.e., the publication of papers is one indicator of achieving technical results, but publishing papers is not the core focus of most CTS funded work).

Timing of the evaluation fieldwork: This evaluation research was conducted as Federal Budget 2012 was being delivered and implemented across funding partners, including NRCan, EC, HC, and NRC. This may have introduced a bias into the interviews as programs were experiencing the effects of reprioritizing within government. The evaluation methodology addresses this potential bias through taking a multiple lines of evidence approach and seeking confirmation of identified issues across lines of evidence rather than single sources where possible.

3.0 CTS Context

3.1  The Policy Context

The combustion of fossil fuels for transportation in Canada is a major contributor to emissions of greenhouse gases (GHGs), criteria air contaminants (CACs) and other non-regulated emissions.  In 2009 the transportation sector accounted for 30% of Canada’s GHG emissions, continuing its trend as the single largest contributing sector; and road transportation accounted for 69% of 2009 transportation sector emissions.^{Footnote 4}

The 2007 national emissions inventory reported that transportation accounted for over half of carbon monoxide (CO) and nitrogen oxide (NOx) emissions, and a significant fraction of volatile organic compounds (VOCs) and fine particulate matter emissions.  These emissions have environmental and health effects, and have been identified as an important factor in air quality.

A primary driver of technology R&D and adoption in the automotive sector is emerging regulations and standards.  Given Canada’s policy of harmonization with US standards, a key driver behind industry investments in lightweighting, alternative energy sources, and fuel efficiency is the Corporate Average Fuel Economy (CAFE) Standard set by the US EPA.  CAFE sets fuel economy targets for 2016 and 2025 of 35 miles per US gallon (mpUSg) and greater than 54 mpUSg respectively.  The starting point in 2010 is 25 mpUSg and thus an overall 115% improvement in average fuel economy in 15 years.^{Footnote 5} 

3.2 CTS Portfolio and Program Context

A review of literature and past evaluation experience has shown there to be a number of external factors that influence R&D programming.  In April and May 2012, the evaluation team developed this results chain and specific factors and assumptions thought to influence the progress of CTS programs in consultation with NRCan OERD and the five program leads. These were specifically tested through this evaluation. The expected chain of results and assumption are summarized in Figure 1.

Text version

Figure 1: CTS Results Chain Factors and Assumptions

Figure 1 is a diagram of results expectations and assumptions about how the CTS programs will achieve their intended outcomes. The results expectations are numbered and listed on the left side of the diagram, while the assumptions (which are lettered) are listed on the right with arrows indicating which results they impact.
Result expectation 1. NRCan determines need and defines CTS and component program objectives      
Assumption A. Appropriate information, understanding and analysis of problems convert into appropriate program design, investment
Result expectation 2. NRCan and co-delivery agents invest in program(s)
Assumption B. Sufficient, appropriate and consistent funding and program assistance
Result expectation 3. The appropriate arrangements and (critical mass of) co-delivery agents engage with NRCan and other ‘partners/beneficaries’ to develop the program
Assumption C. Agendas remain consistent with key co-deliverers
Result expectation 4. Governance structures are formed and actively managed (Program Advisory Committees and OERD)
Assumption D. Support climate allows for clear governance
Result expectation 5. Program priorities are (clearly) set and projects are solicited (appropriately)
Assumption E.   Economic, management and political circumstances allow for appropriate public and private sector engagement in project proposals
Result expectation 6. Appropriate public and private sector participation / engagement in project proposals       
Assumption F. Key sector proponents have the capacity and commitment to participate in project proposals
Result expectation 7. Appropriately targeted and realistic proposals supported (i.e., they respond to public and private sector needs / market realities) 
Assumption G. Proponents have ‘will’ and ability to carry through on project commitments
Result expectation 8. Projects are conducted as anticipated (appropriately addressing needs). 
Assumption H. Target communities attracted to participate / engage in initiatives
Result expectation 9. Appropriate target groups (e.g., regulatory, industry, research community, etc.) are reached by CTS dissemination activities.
Assumption I.  Information / technology developments are ‘attractive’ and compelling to participants, used in making decisions
Result expectation 10. Groups reached by initiatives show positive reactions, capacity (knowledge, abilities, commitments):

• public sector - willingness and commitment to using scientific evidence in decisions, key influencers have info they need
• private sector – apply scientific knowledge and technology in development of new products and proceesses (vehicle fuels, systems and components)
• International partners (US and others) consider Canadian transportation sector regulations, products and processes to be environmentally responsible and preferred choic

Assumption J. Canadian transportation sector technologies (regulations, vehicles, fuels) are recognized as environmentally responsible, preferred choice (nationally and internationally)    
Result expectation 11. CTS objectives are met:

• Development and use of cleaner, sustainable transportation fuels and systems and regulations
• Adoption of cleaner sustainable transportation vehicle fuels and systems in domestic and international markets (sales of new technologies, fuels, transportation systems)

Assumption K. Canadian transportation fuels and systems are cost-competitive and meet international environmental standards
Result expectation 12. Reduced GHG and CAC emissions from transportation sector      
Assumption L.  Clean transportation technologies are a competitive advantage for the Canadian transportation sector
Result expectation 13. Sustainable transportation sector              
Assumption M. Net benefit to Canadian transportation sector companies leads to net benefits to Canada and Canadian communities     
Result expectation 14. Net benefit to Canada and Canadian communities            
*Key links for causal influence tests. 
Note that Assumptions A to E suggest that “Needs assessment, priority setting and governance factors appear to have strongly affected CTS programs”.
Note that Assumptions F to I suggest that “The engagement, reaction and supportive actions of key reach groups has varied considerably”. Assumptions F to I are also denoted as key links for causal influence tests
Note that assumptions J to M suggest “Longer term impacts and mission achievement”.

4.0 CTS Portfolio Overview and Program Profiles

4.1 CTS Portfolio Overview

The five CTS programs were designed to improve energy efficiency and reduce emissions in road transportation.  The CTS portfolio is part of the Energy Efficiency and Alternative Transportation Fuels sub-activity, which supports the Department’s overall Environmental Responsibility Strategic Outcome as described in NRCan’s 2011-12 Program Activity Architecture in Figure 2.

Text version

Figure 2: Clean Transportation Systems Portfolio in the Context of NRCan’s PAA

Figure 2 shows the Clean Transportation Portfolio within NRCan's 2011-12 Program Activity Architecture (PAA).

The programs are under NRCan Strategic Outcome two, Environmental Responsibility, for which the expected result is that Canada is a world leader on environmental responsibility in the development and use of natural resources.

Within this strategic outcome, the programs are part of the Clean Energy Program Activity, whose expected result is “increased energy efficiency, increased production of low-emission energy, and reduced environmental impacts associated with energy production and use.

Within this program activity is the Clean Energy Science and Technology Program Sub-Activity whose expected result is “Industry is engaged in the development and testing stages of new technologies”.

The sub-activity contains six groups of programs. These are Oil and gas (last evaluated in 2009-10), Clean transportation Energy (evaluated in 2012-13; this evaluation report), Clean Energy Systems for Buildings and Communities (last evaluated in 2012-13), Clean Energy Systems for Industry (last evaluated in 2011-12), Clean Electric Power Generation (last evaluated in 2009-10), and Sustainable Bioenergy (last evaluated in 2011-12).

The CTS programs are managed by NRCan, the National Research Council (NRC) and Environment Canada (EC) and receive funding from four different funding programs (the Program of Energy Research and Development (PERD), ecoENERGY Technology Initiative (ecoETI), the Climate Change Technology and Innovation Initiative (CC T&I) and the Clean Energy Fund (CEF)).

The CTS programs over the time period of examination were managed individually but governed cooperatively.  OERD and the Clean Transportation Systems Portfolio Committee, comprised of members from federal science-based departments with an interest in transportation and fuels, provide oversight.  The governance approach is a result of both the needs assessment conducted for the Strategic Plan and a function of the funding mechanisms.

Table 1 summarizes the five programs, including CTS and Partner contributions to program activities.

Table 1:  Overview of The Clean Transportation Systems Programs

Program (start date)	Program Goals	Lead Department	CTS Funding1 (2007-08 to 2011-12) $M NRCan2 $M Partners (incl. in-kind)	Program Approach & Participants
H2FC (2003-04)	Hydrogen and Fuel Cells (H2FC): The goal of this program, led by NRCan’s Innovation and Energy Technology Sector (IETS), is to expand the knowledge base and advance hydrogen and fuel cell technologies.	NRCan-IETS	NRCan = $29.1 Partners = $37.5 Leverage:  1:1.3	Federal researchers conduct R&D, and federal technology experts managing external R&D and demonstrations on hydrogen production, storage and use. Note:  NRCan funding includes $20.7 million in Grants and Contributions.
EM (2007-08)	Electric Mobility (EM): A new program beginning in 2007-08 and led by the National Research Council, the goal of the first five years of this program is to support the development of plug-in hybrid electric vehicles (PHEV).	NRC	NRCan = $9.2 Partners = $10.8 Leverage:  1:1.2	Federal researchers conduct R&D and support development of regulations for electric vehicles and vehicle components with support from industry, universities, and others. Note:   NRCan funding includes $1.6 million in Grants and Contributions.
AFTER (2000-01)	Advanced Fuels and Technologies for Emissions Reduction (AFTER): The goal of this program, led by National Research Council, is to support the design and use of conventional and new hydrocarbon fuels, alternative transportation fuels, and novel combustion technologies, and associated technologies able to reduce vehicle emissions and fuel consumption.	NRC	NRCan = $11.1 Partners = $18.7 Leverage:  1:1.7	Federal researchers conduct R&D on fuels and engine technologies with support from US DOE, and others. Note:   NRCan funding includes $668K in Grants and Contributions.
ASM-NGV (1999-2000)	Advanced Structural Materials for Next-Generation Vehicles (ASM-NGV): The goal of this program, led by NRCan’s CANMET MTL sector, is to develop conventional and new materials, structural components and vehicle subsystems that can be used on virtually all types of next-generation vehicles enabling them to reduce their weight, and to improve their crashworthiness and overall fuel efficiency.  (ASM-NGV was previously known as the Canadian Lightweight Materials Research Initiative (CLiMRI)).	NRCan – CANMET MTL	NRCan = $4.8 Partners = $16.0 Leverage:  1:3.3	Federal researchers conduct R&D on advanced materials, technologies, and transportation systems with support from industry and universities.
P&E (2002-03)	Particles and Related Emissions (P&E): The goal of this program, led by Environment Canada, is to strengthen the scientific basis for policy and regulatory decisions affecting transportation-related emissions of particles and related atmospheric constituents.	Environment Canada	NRCan = $9.5 Partners = $14.7 Leverage:  1:1.5	Federal researchers conduct S&T research and develop tools with some support from industry.
		TOTAL	NRCan = $63.7 Partners = $93.0 TOTAL Investment:  $156.7 Leverage:  1:1.5

¹ NRCan spending data are actual expenditures under the CTS funding mechanisms (i.e., PERD, T&I, ecoETI, and CEF), as reported by OERD.  Partner contributions include leveraged A-base (including NRCan A-base), industry, university, provincial, and international contributions and are based on Program Annual Reports (see Program Resource Level sub-sections below for more detail).  
² NRCan spending includes PERD, ecoETI, T&I and CEF.

The CTS Portfolio programs are funded from NRCan administered federal funding vehicles including the Program for Energy Research and Development (PERD), Climate Change Technology and Innovation fund (T&I), ecoENERGY Technology Initiative (ecoETI), and the Clean Energy Fund (CEF).  Figure 1 provides an overview of this funding within the Portfolio:

Text version

Overview of NRCan Administered Funding for Clean Transportation Systems Programs (2007-08 to 2011-12)

The CTS Portfolio programs are funded from NRCan administered federal funding vehicles including the Program for Energy Research and Development (PERD), Climate Change Technology and Innovation fund (T&I), ecoENERGY Technology Initiative (ecoETI), and the Clean Energy Fund (CEF). Figure 1 provides an overview of this funding within the Portfolio:

The bar chart is titled: Overview of NRCan Administered Funding for Clean Transportation Systems Program from 2007-08 to 2011-12.

The first row in the chart indicates that the Advanced Structural Materials for Next Generation Vehicles (ASM-NGV) program received $4,835,000 from PERD.

The second row in the chart indicates that the Particles and Related Emissions (P&E) program received $9,178,000 from PERD and $298,000 from ecoETI for a total of $9,476,000.

The third row in the chart indicates that the Electric Mobility (EM) program received $8,793,000 from PERD and $396,000 from ecoETI for a total of $9,189,000.

The fourth row in the chart indicates that the Advanced Fuels and Technologies for Emissions Reduction (AFTER) program received $10,956,000 from PERD and $124,000 from ecoETI for a total of $11,080,000.

The fifth row in the chart indicates that the Hydrogen and Fuel Cells (H2FC) program received $14,431,000 from PERD, $2,739,000 from CEF, $10,699,000 from ecoETI, and $1,230,000 from T&I for a total of $29,099,000.

In total, for the five programs, NRCan administered $63.7 million in funding from 2007-08 to 2011-12.

4.2 Hydrogen and Fuel Cells (H2FC) Program

Program Objectives and Expected Outcomes

The H2FC Program provides support to federal and private sector organizations for research, development and demonstration projects across the full spectrum of hydrogen and fuel cell technologies. ^{Footnote 6} The Program was established in 2003 as the Hydrogen Energy Economy program and renamed in 2007-08.  The Innovation and Energy Technology Sector (IETS) of NRCan manages the Program.

The Program was designed to balance short-term R&D needed for technology commercialization in early markets (e.g., back-up / stationary power, forklifts) with the long-term R&D needed to develop technology capable of resulting in a significant impact on air pollution, climate change and energy efficiency in the transportation sector.^{Footnote 7} 

The immediate and intermediate program outcomes are:

• Enhanced knowledge / understanding through the involvement and collaboration of the research community and key stakeholders domestically and internationally. 
• Prototype materials, components and systems to reduce the cost and/or improve the performance of hydrogen and fuel cell technologies.
• Codes and standards supportive of the deployment of hydrogen and fuel cell technologies.

The long-term program outcome is the deployment of hydrogen and fuel cell technologies that contribute to the well-being of Canadians by reducing emissions of air pollutants and greenhouse gases.

Program Activities

The H2FC Program funded 34 research, development and demonstration activities in four areas: Hydrogen Production, Hydrogen Storage, Hydrogen Utilization (i.e., fuel cells), and Safety, Codes and Standards (see Table 2). 

Table 2:  H2FC Sub-Programs, Objectives and Partners^{Footnote 8}

Sub-Program	Objective	Partners in Program Delivery
Sustainable Hydrogen Production	Develop technologies that improve the economics and efficiency of producing hydrogen from renewable energy including Canada’s abundant sources of clean electricity.	The main players in this area are industry and government laboratories.  Industry brings the perspective of technology commercialization and meeting end-user requirements.  Government laboratories have specially trained experts and sophisticated equipment.
Hydrogen Storage	Develop hydrogen storage materials and systems with reduced cost, lighter weight and greater efficiency for both nearer-term fuel cell markets and longer-term automotive applications.	Due to the fundamental nature of the R&D in hydrogen storage, the main performers in this area are universities and government laboratories that have specially trained experts and the equipment needed.  (Industry was identified as being an important partner to bring the perspective of technology commercialization and meeting end-user requirements.)
Utilization: Fuel Cells	Improve the cost, performance reliability and durability of PEM and solid oxide fuel cells and their supporting auxiliaries for stationary, portable and mobile applications.	Industry is a key performer of fuel cell R&D in Canada and much of the intellectual property rests here. In addition, H2FC funding leveraged R&D activity in universities and government laboratories (especially NRC).
Safety, Codes and Standards	Provide information to support the development of models and risk assessment tools leading to adoption of harmonized, science-based codes and standards.	The main players in this area are universities and government researchers; participation in international working groups at the International Partnership for Hydrogen and Fuel Cells in the Economy (IPHE) and IEA is important to ensuring Canada’s standards are consistent with, and able to influence, international standards.

Program Resources

The financial data provided to OERD in H2FC Program Annual Reports (for 2007-08 and 2008-09) and subsequent contributions to CTS Portfolio Reports (2009-10 through 2011-12) were used to build Table 3. 

The H2FC Program received support from four federal funding programs representing $30.1 million of the CTS funding envelope.  The contribution to H2FC activities and projects from all other sources (financial and in-kind) is approximately $37.5 million, for a total overall program of $66.5 million and a leverage ratio of 1:1.3.  Based on Program Annual Report data, Grants and Contributions (G&Cs) accounted for two-thirds of the total H2FC budget, with the balance spent in federal labs.

Table 3:  H2FC Program Funding Sources (CTS and Other Contributions), 2007-08 to 2011-12 ($000)

Funding Source	2007-08	2008-09	2009-10	2010-11	2011-12	Total
PERD	$4,286	$3,365	$2,390	$2,350	$2,040	$14,431
T&I	$1,230	$0	$0	$0	$0	$1,230
ecoETI	$0	$5,411	$3,497	$1,791	$0	$10,699
CEF	$0	$0	$0	$499	$2,240	$2,739
Total Program $ (PERD, T&I, ecoETI, CEF)	$5,516	$8,776	$5,887	$4,640	$4,280	$29,099
A-Base	$1,770	$1,107	$1,007	$484	$1,415	$5,783
Industry	$3,678	$5,526	$4,199	$4,136	$3,684	$21,223
University	$389	$793	$472	$468	$547	$2,669
NGO	$323	$185	$216	$216	$275	$1,215
P/T	$170	$698	$1,661	$684	$170	$3,383
Other (international)	$0	$0	$66	$66	$3,066	$3,198
Leveraged $	$6,330	$8,309	$7,621	$6,054	$9,157	$37,471
Total	$11,846	$17,085	$13,508	$10,694	$13,437	$66,570
Leverage Ratio	1:1.1	1.1:1	1:1.3	1:1.3	1:2.1	1:1.3

Source:  Program Annual Reports to OERD.  Note that data for 2009-10 excludes ecoETI support for G&Cs (data not included in Annual Report but is estimated to be approximately $100,000 based on other program reports).

4.3 Electric Mobility Program

Program Objectives and Expected Outcomes

The Electric Mobility (EM) Program began as a PERD initiative to create an R&D program focused on Plug-in Hybrid Electric Vehicles (PHEVs), after studies showed that the introduction of PHEVs to replace conventional fueled vehicles would improve energy efficiency and reduce GHGs and criteria air contaminants (CACs)^{Footnote 9}due to transportation.  In 2006-07, PERD provided a small amount of funding to the National Research Council (NRC) to identify priorities and develop a full program.  Feedback from a wide range of Canadian and US public and private sector stakeholders through an environmental scan and a workshop identified improving the performance of energy storage technology (batteries) and connection to the electrical distribution grid for charging were also identified as R&D priorities.^{Footnote 10 Footnote 11}

In 2007-08, a five-year PHEV R&D program known as Electric Mobility (EM) was created within the newly formed Clean Transportation Systems Portfolio. 

The overall objective of the Electric Mobility R&D Program is to conduct R&D on Plug-in Hybrid Electric Vehicles (PHEVs).^{Footnote 12}Expected Program outcomes were identified in the EM Program Logic Model, which describes the main elements of the program, from research focus to intended results and outcomes.^{Footnote 13}  The expected outcomes are as follows:

• Immediate: increased safety and capacity of PHEV batteries; reduced weight of PHEV drive components and improved range; improved capability to optimize powertrain designs; new understanding of the performance of PHEVs in Canada and of the impacts of PHEVs on Canadian power utilities.
• Intermediate: Policy and regulatory decision makers are informed and influenced by new knowledge regarding the impact of PHEVs on transportation emissions and energy efficiency.
• Final: Next generation vehicles contribute to improved energy efficiency, reduced emissions and increased economic benefits to Canada.

Program Activities

The environmental scan and workshop held in 2006-07 identified the main research activities for the new EM program, which became the four activity areas or sub-programs for 2007-08 to 2011-12.  There were a total of 16 projects funded under the EM program.  The Energy Storage Systems subprogram had eight projects, seven of which related to Li-ion batteries, carried out by both NRC researchers and industrial firms developing and manufacturing batteries. Other technology development projects focused on electric drive and power train improvements.  There were four projects under support for development of Regulations for Emissions and Fuel Efficiency.

Table 4:  EM Sub-Programs, Objectives and Partners

Sub-Program	Objective	Partners in Program Delivery
Energy Storage systems	Development of improved battery components and systems, and safer, more reliable batteries	These projects were carried out mainly by researchers at NRC’s institutes and industrial firms developing and manufacturing batteries. These projects have leveraged industry, university, and other public partnerships.
Electric drive components	Development of improved control systems and drive components	The main performers in this area are NRC and industry. These projects have leveraged partnerships with industry.
Power train optimization	Development of advanced powertrain simulation modeling	The main performers in this area are NRCan CANmet and university researchers.
Support for development of regulations for emissions and fuel efficiency	Development of standard test procedures for emissions and fuel economy for PHEVs, impacts of PHEVs on electricity generation, distribution and emissions, requirements for charging infrastructure.	The main players in this area are universities and government researchers including EC, NRC, DND, the centre for Energy Advancement through Technology Innovation. Partners include US EPA, universities, and power utilities. The Program leads Canada’s participation on the IEA Implementing Agreement on Hybrid and Electric Vehicles (IA-HEV)Footnote 14.

Program Resources

A summary of EM Program Funding Sources is provided in Table 5, based on financial data contained in EM Program Reports (2007-08 and 2008-09) and in CTS Portfolio Reports (2009-10 and 2011-12).  All EM Program funding ($9.2 million) during the last funding cycle came from PERD. Partners contributed $10.8 million in total. For all projects, the participating organizations provided financial and in-kind funding, in most cases, equal to or greater than EM Program funding.  The overall ratio of CTS funding to other sources was 1:1.2 for the funding cycle.

Table 5: EM Program Funding Sources (CTS and Other Contributions) 2007-08 to 2011-12 ($000)

Funding Source	2007-08	2008-09	2009-10	2010-11	2011-12	Total
PERD	$959	$1,955	$1,915	$2,000	$1,964	$8,793
ecoETI	$20	$140	$130	$106		$396
Total Program $	$979	$2,095	$2,045	$2,106	$1,964	$9,189
A-base	$340	$1,158	$1,244	$1,407	$1,247	$5,396
Industry	$558	$1,110	$768	$620	$604	$3,660
University	$10	$78	$73	$55	$60	$276
Other (international)	$127	$290	$320	$320	$370	$1,427
Leveraged $	$1,035	$2,636	$2,405	$2,402	$2,281	$10,759
Total	$2,014	$4,731	$4,450	$4,508	$4,245	$19,948
Leverage Ratio	1:1.1	1:1.3	1:1.2	1:1.1	1:1.2	1:1.2

Source:  Program Annual Reports to OERD. 

4.4 AFTER Program

Program Objectives and Expected Outcomes

The Advanced Fuels and Transportation Emissions Reduction Program (AFTER) was created in 2000 as a PERD Program, with funding for an initial five-year period ending in 2004-05.  The original objective of AFTER was to contribute to improved air quality through the development of technologies and fuels that would lead to reduced emissions from the Canadian transportation sector.  AFTER continued to receive funding during the next PERD cycle, and in 2009, became part of the Clean Transportation Systems Portfolio, with funding for a three-year period ending in 2011-12.

The AFTER objective for the 2009-10 to 2011-12 funding period was to develop knowledge and technology relating to transportation fuels, engine designs, aftertreatment technologies and human health effects in order to reduce emissions for GHGs and pollutants harmful to human health and the environment.^{Footnote 15} Expected outcomes are identified in the AFTER Program Logic Model, which describes the main elements of the program, including the program components, expected deliverables, intended beneficiaries  of the research and intended results or outcomes.^{Footnote 16} The intended outcomes are as follows:

• Immediate – new knowledge on emissions, energy efficiency and cost implications of oil sands derived fuels and biofuels used in conventional and advance technology engines; new technology options for reducing engine emissions; and new knowledge regarding human health and environmental effects and risks associated with biodiesels, ethanol blended gasoline and emissions from advanced fuels, new engine technologies and aftertreatments.
• Intermediate – Fuel specifications, engine designs and health assessments benefit from new knowledge, policies and regulations regarding transportation-related air pollutants.
• Final – health, environmental and economic impacts of transportation-related emissions are reduced. 

Program Activities

The AFTER research themes have remained relatively unchanged since the beginning, with minor modifications. During the period 2009-10 to 2011-12, AFTER carried out a total of 13 research projects. AFTER also participated in two IEA initiatives and the joint Canada / US Annual Transportation Technology and Fuels Forum (TTFF).  Funding was also provided for program management, external linkages and knowledge dissemination. The program focused on four research areas outlined in the table below.

Table 6:  AFTER Sub-Programs, Objectives and Partners

Sub-Program	Objective	Partners in Program Delivery
Fuel Composition and Performance	Understanding performance of conventional and oil sands derived fuels; various biodiesel fuel mixes; wheel to wheel analysis of the efficiency and emissions associated with several fuel and technology options	The main performers were government researchers at NRC, EC ERMS, and CanmetENERGY, with international partnerships (VTT Technical Research Centre of Finland).
Advanced Combustion Technologies	Understanding performance of oil sands derived fuels and blended oil sands derived and renewable fuels on efficiency and emissions in advanced technology engines	The main performers were government researchers at NRC, CanmetENERGY and EC ERMS with federal government (EC, TC) and international (DCL International) partnerships.
Engine Hardware and Exhaust Aftertreatment	Understanding effects of devices on fuel efficiency and emissions	The main performers were government researchers at NRC, researchers at RMC and Nexus Research Corp. Projects leveraged partnerships from private sector (Engine Control and Monitoring Company, and Armstrong Monitoring Corp.), universities (university of Waterloo), and EC.
Health and Environmental Effects	assessment of toxicity of biofuels and emissions on health	The main performers were government researchers at HC with partnerships at and EC, NRC.

Program Resources

A summary of AFTER Program Funding Sources is provided in Table 7, based on financial data contained in AFTER Program Reports (2007-08 and 2008-09) and in CTS Portfolio Reports (2009-10 to 2011-12).  Almost all AFTER funding came from PERD. 

The total CTS investment in the AFTER Program was $11.1 million over the period under review, with partners providing $18.7 million.  For all projects, the participating organizations provided financial and in-kind funding; in most cases, equal to or greater than AFTER Program funding.  The overall ratio of CTS funding to other sources was 1:1.7 for the 2007-2011 funding cycle.

Table 7: AFTER Program Funding Sources (CTA and Other Contributions) 2007-08 to 2011-12 ($000)

Funding Source	2007-08	2008-09	2009-10	2010-11	2011-12	Total
PERD	$2,130	$2,234	$2,162	$2,283	$2,147	$10,956
ecoETI			$124			$124
Total Program $	$2,130	$2,234	$2,286	$2,283	$2,147	$11,080
A-base	$2,365	$2,340	$2,524	$2,758	$2,684	$12,671
Industry	$680	$725	$656	$706	$505	$3,272
University	$45	$45	$20	$30	$30	$170
Other	$710	$281	$1,275	$125	$200	$2,591
Leveraged $	$3,800	$3,391	$4,475	$3,619	$3,419	$18,704
Total	$8,060	$7,859	$9,047	$8,185	$7,713	$40,864
Leverage Ratio	1:1.8	1:1.5	1:2	1:1.6	1:1.6	1:1.7

Source:  Program Annual Reports to OERD.
NOTE: Excludes $130k for IEA Combustion and Advanced Motor Fuels Implementing Agreements, and excludes funding from other PERD CTS Programs, specifically 2.1.1, to avoid double-counting.

4.5 Particles and Related Emissions Program

Program Objectives and Expected Outcomes

The P&E program, led by Environment Canada, is designed to strengthen the scientific basis for policy and regulatory decisions affecting transportation-related emissions of particles and related atmospheric constituents.^{Footnote 17}

P&E generates knowledge, tools and models related to the role of transportation in the production of particulate matter (PM) and the impact of PM on the environment and human heath, primarily in support of policy and regulation development.

P&E funding began in 2000 to meet needs for information on the environmental and human health impacts of transportation emissions to support policy and regulatory development, as well as contribute to technology planning.  Program R&D needs were defined in consultation with Environment Canada and Health Canada.  These two departments perform most of the R&D.

Program Activities

Program priorities were clearly set in the 2005-06 to 2008-09 Plan, and then again in the 2009-10 to 2011-12 Plan.  The three overarching research themes of the P&E R&D Program are:

• the role of transportation sources in the production of particulate matter (PM) in Canada (both direct emissions of PM and emissions of PM precursors);
• the evolution of PM in the atmosphere (transport, chemical modification, secondary aerosol formation, removal); and,
• the human health and environmental (air quality, visibility, haze, climate change) impacts of transportation-related PM.

The program outputs are largely knowledge, data and models and unlike the other program areas the clients are primarily policy makers rather than industry.  The P&E Management Committee includes policy representatives from key departments (e.g., Environment Canada, Transport Canada, Health Canada, NRCan).

P&E provided funding to 10 projects over the last five years.  Federal researchers at Environment Canada and Health Canada defined P&E projects, with some industry participation. 
These projects are grouped into four sub-programs outlined in the table below:

Table 8:  P&E Sub-Programs, Objectives and Partners

Sub-Program	Objective	Partners in Program Delivery
Emissions Characterization	The objective is to develop and apply technologies and methods to measure and describe the emissions of particles, precursors and tracer compounds associated with transportation-related sources	The main performers were government researchers at EC and NRC institutes.
Characterization of Ambient Particles and Related Atmospheric Constituents	The objective is to develop and apply technology and methods to measure and describe the atmospheric presence, transformation and fate of transportation-related particles, precursors and tracer compounds	The main performers in this area were government researchers at EC. The projects leveraged partnerships with HC, NRCan, and universities.
Modelling	The modeling activities of the Program are now incorporated into an integrated project under Theme 2 (the Advancing Local-scale Modelling through Inclusion of Transportation Emission Experiments (ALMITEE) project) which combined two research theme areas, ambient air monitoring and modelling. Nonetheless, this remains a significant activity within the Program ($2.2 million in PERD funding between 2008-09 and 2011-12)
Health and Environmental Effects	The objective is to develop and apply measurement and analysis techniques, systems and studies, to identify acute and chronic effects on humans and the environment (air quality) that occur as a result of particles and other pollutants associated with transportation-related sources	The main performers were government researchers at EC and HC.

The program manager estimates that approximately 20% of P&E projects directly link to ongoing AFTER projects (i.e., some joint funding of project activity through the two programs).

Program Resources

Program financial data provided in P&E Program Annual Reports (for 2007-08 and 2008-09) and contributions to CTS Portfolio Reports (2009-10 through 2011-12) were used to develop Table 9.  The P&E Program received support from PERD and ecoETI funding programs totalling $9.5 million and accounting for approximately 12% of the total CTS Portfolio funding envelope. 

The total investment in P&E activities from all other sources (financial and in-kind) was approximately $14.7 million, for a total overall investment in P&E research of $24.2 million and a leverage ratio of 1:1.5.

Table 9:  P&E Program Funding Sources (PERD and Other Contributions), 2007-08 to 2011-12 ($000)

Funding Source	2007-08	2008-09	2009-10	2010-11	2011-12	Total
PERD	$1,477	$1,474	$2,014	$2,215	$1,998	$9,178
ecoETI	0	111	104	83	0	$298
Total Program $	$1,477	$1,585	$2,118	$2,298	$1,998	$9,476
A-Base	$2,731	$2,144	$1,800	$2,783	$2,081	$11,539
Industry	$710	$665	$226	$265	$245	$2,111
University	$50	$50	$37	$80	$65	$282
Other (international)	$339	$96	$210	$100	$9	$754
Leveraged $	$3,830	$2,955	$2,273	$3,228	$2,400	$14,686
Total	$5,307	$4,429	$4,287	$5,443	$4,398	$24,162
Leverage Ratio	1:2.6	1:1.9	1:1.1	1:1.4	1:1.2	1:1.5

Source:  Program Annual Reports to OERD. 

4.6 Advanced Structural Materials – Next Generation Vehicles (ASM-NGV) Program

Program Objectives and Expected Outcomes

The Program on Advanced Structural Materials for Next Generation Vehicles (ASM-NGV) is driven by the need to improve transportation energy efficiency by reducing vehicle weight (i.e., ‘lightweighting’).  ASM-NGV was previously known as the Canadian Lightweight Materials Research Initiative (CLiMRI), which was established in 1999-2000, and is managed by the CANMET Materials Technology Laboratory (MTL).  Co-delivery partners include Auto21, Automotive Partnerships Canada (APC), MagNet, NSERC, NRC-Industrial Materials Institute (IMI), and Transport Canada.

The challenges associated with developing new materials for the automotive sector are complex and highly inter-dependent on factors including vehicle class, material supply and fundamental properties, processing and application.  The objective is to find new methods in processing and materials design to achieve improvements in the weight specific properties of engineering materials. 

The Program is part of a worldwide effort to develop lightweight (i.e., low density and/or high strength) materials and components for fuel-efficient vehicles, such as future diesels, fuel cells and hybrid vehicles.  The Program involves participation with industry, universities and other government departments.

The Program objective^{Footnote 18} is to develop and implement lightweight materials in structural components and vehicles for the purposes of meeting the following outcomes:

• Reduced greenhouse gas and related emissions through weight reduction and improved vehicle efficiency; and
• Improved competitive performance of the Canadian primary metals, automotive, truck, bus, and rail car manufacturing industries and their associated parts suppliers.

These materials and components may be used on all types of next-generation vehicles enabling them to reduce their weight, improve their crashworthiness and increase overall fuel efficiency.

Program Activities

ASM-NGV comprises two sub-programs with a total of 11 projects.  Sub-program objectives and partners are identified in Table 10.

Table 10:  ASM-NGV Sub-Programs, Objectives and Partners^{Footnote 19}

Sub-Program	Objective	R&D Partners and Collaborators
Sub-Program 1: Body Structure Technologies 3 Projects	Focuses on the development of new materials, processes and integration technologies for automotive and bus body frames and components.  Lightweighting materials include magnesium, aluminium, polymer composites and advanced high-strength steel.  The sub-program also addresses the need for enabling technologies and data related to forming, joining and corrosion of advanced materials.	ASM-NGV’s research network is comprised of 17 companies, 6 universities, and 4 government departments and funding agencies. This sub-program contains the largest ASM-NGV project (Magnesium Front-End R&D- MFERD), which involves extensive collaboration with international researchers (China and US DOE, US Automotive Materials Partnership) and industry (Ford, GM), and is co-funded by Auto21.  Canadian companies involved in the project (Centre Line, Promatek (Magna-Cosma), Husky and Meridian Technologies) represent different steps in the value chain. Other partners in this sub-program include Dofasco and Stelco (re. ultra-high strength steel), and three universities (McGill, McMaster and Ecole Polytechnique).
Sub-Program 2: Powertrain and Chassis Technologies 8 Projects	Focuses on fundamental research into materials and processing technologies that are particularly promising for next generation vehicle powertrains.  Vehicle-based energy efficiencies may be gained by use of advanced combustion strategies and powertrain designs, lighter-weight materials for internal combustion engines (ICE) and hybrid electric powertrains.  The activity also addresses potential corrosion issues associated with the use of alternate fuels.	ASM-NGV’s research network is comprised of 17 companies, 6 universities, and 4 government departments and funding agencies. Industry partners include: Dana Corporation, Novelis, GM, Ford, Magna, and university researchers at McGill, Waterloo and McMaster universities.

Program Resources

The financial data provided in ASM-NGV Program Annual Reports (for 2007-08 and 2008-09) and subsequent contributions to CTS Portfolio Reports (2009-10 through 2011-12) were used to build Table 11. 

PERD is the only funding program that provided support to ASM-NGV (almost $5 million over five years), and the Program represents 7.7% of the CTS funding envelope.  Each project involved industry members who provided in-kind and / or financial contributions.  The largest contributions were provided by the US and China to an international magnesium research project (more detail is Section 5).  The contribution to ASM-NGV projects from non-PERD sources was $16 million, for a leverage ratio of 1:3.3.

Table 11:  ASM-NGV Program Funding Sources (CTS and Other Contributions), 2007-08 to 2011-12 ($000)

Funding Source	2007-08	2008-09	2009-10	2010-11	2011-12	Total
PERD	$900	$995	$995	$950	$995	$4,835
Total Program $ (PERD)	$900	$995	$995	$950	$995	$4,835
A-Base	$690	$950	$1,070	$1,110	$956	$4,776
Industry	$460	$500	$340	$420	$476	$2,196
University	$754	$400	$270	$180	$230	$1,834
Other (international research programs)	$25	$1,200	$2,000	$2,000	$2,000	$7,225
Leveraged $	$1,929	$3,050	$3,680	$3,710	$3,662	$16,031
Total	$2,829	$4,045	$4,675	$4,660	$4,657	$20,866
Leverage Ratio	1:2.1	1:3.1	1:3.7	1:3.9	1:3.7	1:3.3

Source:  Program Annual Reports to OERD.

5.0 Evaluation Findings

5.1   Relevance

5.1.1 On-going Need

Evaluation Issues	Lines of evidence	Assessment
Is there an ongoing need for the programs and activities?  How are target groups served by the program?	Document  Interviews	Clear ongoing need for R&D to support reduction of transportation emissions.  Debate as to whether the H2FC program design can meet current fuel cell industry needs.

Discussion and Analysis

The combustion of fossil fuels for on-road transportation in Canada is known to be a major contributor to GHGs, CACs, VOCs and non-regulated emissions. In 2009 road transportation accounted for 21% of Canada’s GHG emissions, continuing to be the single largest contributing sector. The 2007 national emissions inventory reported that transportation accounted for over half of CO and NOx emissions, and a significant fraction of volatile organic compounds (VOCs) and fine particulate matter emissions.  These emissions have environmental and health effects, and have been identified as an important factor in air quality.

The five CTS Programs respond to the need to reduce these emissions, each one focusing on a different aspect, as described below. 

H2FC responds to the need to advance the development of hydrogen and fuel cell technologies for use in applications as a clean alternative to internal combustion engine (ICE) powered vehicles using fossil fuels. There continues to be significant research, technology development and infrastructure needs to be addressed before these technologies will be commercially viable in the on-road vehicle market.  As a result of recent changes in the fuel cell industry, including reduced investments in the development of fuel cells for on-road vehicles, there is a debate as to whether the present H2FC program meets current industry needs.

The EM program supports the increased use of clean electrical energy to power PHEVs as an alternative to conventional ICEs powered by fossil fuels, by conducting R&D to improve PHEV performance. Stakeholders identified improved battery capacity, safety and reliability as their highest R&D priorities and these issues are the major focus of EM R&D.  EM also responds to government needs for technical capacity to support the development of policies and regulations to support the use of PHEVs.  The research program takes into account factors specific to Canada, such as the cold weather performance of batteries and the availability of charging capacity.

AFTER R&D addresses the government’s need for scientific information to support the development of appropriate transportation policies and regulations for emissions from ICEs equipped with various engine technologies, burning fossil and blended fuels.  The program also responds to industry’s need for support in the development of novel combustion and aftertreatment technologies which reduce vehicle emissions and fuel consumption.

P&E responds to the needs of government transportation policy and regulatory groups for information on the environmental and human health impacts of transportation related emissions of GHGs, CACs, NOx, particulates and other non-regulated emissions resulting from the combustion of a variety of fuels and fuel blends in conventional and advanced ICE systems.  

ASM-NGV supports industry’s needs for developing lightweight metals and polymers for use in vehicle structural components and vehicle subsystems as a strategy to reduce vehicle weight, improve fuel efficiency and address emerging regulatory targets. The Program has focused on emerging high strength steel, aluminium and magnesium applications identified by industry as priority areas.

5.1.2   Government Priorities and NRCan Strategic Objectives

Evaluation Issue	Lines of evidence	Assessment
Are the programs and activities consistent with government priorities and NRCan strategic objectives?	Document review File review Interviews	The CTS portfolio is consistent with departmental and government priorities.

Discussion and Analysis

In response to global warming, and concerns about the environment and air quality, Canada has established the Clean Air Agenda, the Canadian Environmental Protection Act and related policies, in order to reduce GHG, CAC and other non-regulated emissions in Canada. Interviews and the document review show clear linkages between the five Programs and federal priorities and objectives, supported by the participation of OGDs in program planning and priority setting.  The Portfolio supports NRCan’s objectives related to energy production, efficiency and use of renewable energy, long-term economic impacts, and more specifically to the Strategic Priority of Clean Transportation Energy.  It also supports the regulatory needs of federal departments. Environment Canada is responsible for regulations related to GHGs emissions and Health Canada is responsible for regulating health related emissions including critical air contaminants, as well as toxicity of fuels. 

H2FC

Hydrogen and fuel cell technologies are consistent with federal government and NRCan environmental objectives related to climate change, air quality and sustainable development.  Fuel cell technology is one possible means of achieving low and zero-emissions for the transportation sector.  Hydrogen can be produced from a wide-range of energy resources (e.g., nuclear, biomass, renewable and hydrocarbon sources) and when combined with oxygen in an energy conversion device, like a fuel cell, the resulting by-products are electricity, water vapour and heat. 

The H2FC Program also supports economic and industry development.  In the early 2000s Canada was widely accepted by industry and public sector stakeholders (as evidenced in past sector strategies and confirmed by interviewees) to be a leading developer of fuel cell technology and hydrogen infrastructure.  The sector was, and remains, characterized by innovative, small companies with significant R&D interests and needs, the largest cluster of which was in Vancouver.  H2FC provided support to Canadian industry directly through grants and contributions for R&D to improve performance and reduce the cost of fuel cell technology, and indirectly by developing the infrastructure (codes, standards, policies) required to support sector development.  Over the past five years, the sector has seen the sale of some Canadian H2FC companies or divisions to international firms and reduced levels of federal H2FC R&D investment (as noted in 5.1.1, and seen in the dPoint Humidifier and Airports Demonstration case studies). 

EM

The Clean Air Agenda established in 2006 remains the primary policy supporting efforts to reduce GHGs and air pollutant emissions.  Analysis has shown that replacement of conventional ICE vehicles by PHEVs could be an effective mechanism to achieve these objectives.  The EM Program is designed to carry out R&D to support PHEV market development in Canada. 

The EM Program, led by NRC, includes participation by NRCan, Environment Canada, Transport Canada and DND.  A number of EM projects involve the private sector and contribute to the development of improved batteries and drive train components.

AFTER

The Clean Air Agenda established in 2006 remains the primary policy supporting efforts to reduce GHGs and air pollutant emissions.  NRC is the lead department, working with NRCan, Health Canada and Environment Canada.  In transportation, Canada is developing new regulations for emissions in harmony with the US.  To support policy and regulatory decision making, it is important to be able to evaluate the effect of advanced fuels and engine and aftertreatment technologies on both transportation efficiency and emissions. An example project is the health impacts of biodiesel use in Canada (Health and Environmental Effects of Advanced Biofules project) which was funded by AFTER and carried out by Health Canada.

AFTER contributes to NRCan strategic objectives as described in the current PAA (see Section 4.1) by conducting R&D on advanced fuels and technologies that could result in reduced emissions from the transportation sector.^{Footnote 20}

Interviewees noted that AFTER R&D supports the information needs of Environment Canada, Health Canada and NRCan related to emissions from a range of fossil and renewable fuels for current and advanced internal combustion engines.

P&E

The Program supports NRCan’s Strategic Priority and meets the objectives of the Clean Transportation Systems Portfolio.  The objective of P&E is to conduct RD&D on the fate and impacts of transportation-related emissions of particles and related atmospheric constituents.

The Clean Air Agenda established in 2006 remains the primary policy supporting efforts to reduce GHGs and air pollutant emissions.  Environment Canada is the lead department, working with NRCan, Health Canada and NRC.  Canada continues to develop regulations related to transportation energy efficiency that are harmonized with those in the US.  To support these policies and regulations, tools and data to evaluate the effect of advanced fuels and engines, and related technologies, on transportation efficiency and emissions are needed. Interviewees noted that P&E activities and outputs support the information needs of Environment Canada, Health Canada and NRCan. 

ASM-NGV

The goal of the ASM-NGV Program is that clean air solutions (based on high-performance, new or improved, lightweight materials, ultra high-strength steels, and other material innovations) are developed for the automotive sector in partnership with universities, industry and federal scientists.  By supporting R&D on advanced technologies and materials that increase energy efficiency the program is aligned with NRCan and federal objectives.

Canada has a policy of aligning regulations with the US. The US EPA regulations call for a reduction in fuel consumption of 35% between 2011 and 2016, and a fuel economy of 56.2 mpg across the fleet by 2025 (the average is currently 25 to 30 mpg).  The need to greatly improve the fuel efficiency of the automobile fleet has led to efforts to increase the use of lightweight material, introduce novel manufacturing technologies (e.g., materials forming, machining), and develop radically different engines and powertrain technologies.

Federal interest in ASM-NGV research is reflected in its participation in the Clean Energy Dialogue with the US.  In February 2009, the federal government signed the Canada US Clean Energy Dialogue (CED) with a goal of promoting cross-border energy R&D on future generation biofuels, clean engines / vehicles and energy efficient homes and buildings.^{Footnote 21}  The Clean Energy R&D Working Group supports a range of R&D and demonstration projects and is developing a Clean Energy R&D Framework and Roadmap to explore near-term options for Canada and the US to meet their GHG reduction targets by 2050.  The CTS program that will be most affected by the CED is ASM-NGV.  The CED was renewed for a further two years in 2012.

5.1.3   Legitimate and Appropriate Role

Evaluation Issue	Lines of evidence	Assessment
Is there a legitimate, appropriate and necessary role for the federal government in these programs and activities?  Is NRCan’s role appropriate in the context of the role of others?	Document review Interviews Case studies Bibliometrics	Government plays a legitimate and appropriate role.

Discussion and Analysis

Documents, literature and interviews suggest that the importance of the auto sector in Canada and the impact of the transportation sector on the environment and harmonization of transportation regulations with the US make CTS an important, legitimate area for federal support.  The development of electric vehicles, fuel cells, alternative fuels and lightweight materials support Canada’s competitiveness and environmental objectives. There is a legitimate  role for the federal government to support basic and applied R&D to improve cost-competitiveness of new technologies / processes, support for the development of codes, standards and related infrastructure. Early stage R&D projects are usually viewed as high- risk, and are unlikely to be conducted at significant levels in Canada without federal government involvement and funding (however, case examples show how fundamental R&D findings can be applied to refine existing products or processes).  There is also a significant role to be played representing Canadian perspectives and industry in international fora and committees (e.g., within the IEA, Clean Energy Dialogue, US DOE and IPHE). 

The bibliometric analysis shows that NRCan is well integrated into the CTS research framework in Canada. Through the CTS Portfolio programs it has funded leaders in clean transportation research who are well connected to the other main researchers in the field. NRCan’s strongest collaboration links have been with University of Waterloo, McGill University, and NRC – the main hub for CTS related research in Canada.

CTS programs are primarily funded through PERD, an interdepartmental R&D program administered by NCRan OERD. Interviewees across CTS programs held the perception that PERD investment decisions during the period under review (2007-08 to 2011-12) did not necessarily take into consideration the supporting research mandates and needs within OGDs, and that the extent to which other departments’ related priorities influence CTS priorities and planning could be strengthened. According to OERD, as PERD funding was reduced during the 2007-08 to 2011-12 funding cycle, a more focussed  interpretation of PERD was taken to focus more strictly on programs related to energy production, distribution and use.

H2FC

The challenges facing full commercialization of hydrogen and fuel cell technologies must be addressed through policy mechanisms and technology improvements – both of which require collaboration between industry and government, and internationally to increase the incorporation of these technologies in the global energy portfolio.^{Footnote 22}  Some of the technology needs require basic and applied research (e.g., materials) and public sector support for this pre-competitive R&D is appropriate. 

The federal government has a unique ‘honest broker’ role to play in the development of codes, standards, testing, and certification (i.e., the infratechnologies needed to support H2FC market development) in Canada and internationally.  With support from H2FC, NRCan represented Canada at the IEA Hydrogen Implementing Committee (NRCan was the past Chair of its Executive Committee), which includes representatives from 17 countries including the US, France, Germany, Japan, and the UK.  Since the 1990’s, NRCan participated in three sub-tasks: hydrogen safety, biohydrogen and hydrogen storage materials.  Canada also participated in the IEA’s Fuel Cell Implementing Agreement (signed in 1990 by 15 countries including the US, France Germany and Japan).  According to interviewees, H2FC funding was critical to this participation, supporting the costs associated with committee membership and providing research / capacity to participate in the technical sub-committees. 

Evidence from the document review and interviews indicate that NRCan’s primary H2FC role is to support R&D taking place in industry.  Other federal departments have a role to play in the H2FC Program: NSERC is the main supporter of R&D taking place in universities, NRC leads government’s in house R&D efforts, and DND is a key research performer and early adopter. These organizations participated in the Program through the H2FC Program Executive Committee and the Hydrogen and Fuel Cell Coordinating Committee. Concerns were expressed from some interviewees regarding the distribution of PERD funding and in some cases a lack of feedback on why certain projects were selected over others. 

EM

Research is needed to support the development of the policies and regulations necessary to ensure the safety and performance of batteries and other components of electric vehicles being sold in Canada.  It is important for government to ensure that the scientific data that informs policy and regulatory decision-making is credible, accurate and free from real or perceived conflict of interest.  As explained by Tassey^{Footnote 23}, a key role for government R&D is the development of technical infrastructure, facilities, and standardized test procedures used to make measurements and generate the data needed by policy makers.

The 2006 NRC Environmental Scan notes that PHEVs represent a realizable, near-term non-disruptive clean transportation solution to achieving the policy goals of GHG reduction and fuel economy.^{Footnote 24}  However, it also reports that a consensus of key stakeholders in Canada and the US consider that “government policy is needed to overcome industry inertia.”  These observations supported the decision to introduce a program to provide R&D support to improve the competitiveness of PHEVs compared to conventional single ICE powered vehicles.

The need for government support for electric vehicle R&D is described in the ‘Electric Vehicle Technology Roadmap for Canada’ published in 2009.^{Footnote 25}  The Roadmap reports that the effective development of an electric vehicle industry in Canada requires government and industry to work together to develop “advanced batteries, a charging infrastructure, electricity storage devices, codes and standards and policies, as well as public education and consumer acceptance.”  Large-scale commercial introduction of PHEVs requires both technical innovation and a supportive regulatory and policy framework.  EM research is designed to contribute to both.

As noted earlier, PERD (managed by OERD) provided 95% of total EM Program funding during the 2007-08 to 2011-12 period, and thus OERD has a necessary and appropriate role in Program oversight.  In addition to NRCan, NRC plays a significant role in the Program.  A review of EM projects shows that the majority are associated with improvements to battery performance and NRC leads many of these projects.  PERD funding requires participating departments to support projects through financial and /or in-kind contributions.  The majority of EM research projects are mid-stage, requiring several more years of technical development before potential application; the high level of risk associated with these projects means that they are unlikely to be fully funded by industry.

AFTER

It is important for government to ensure that the scientific data that informs policy and regulatory decision-making is credible, accurate and free from real or perceived conflict of interest.  As pointed out by Tassey,^{Footnote 26} one key role for government R&D is the development of technical infrastructure, facilities, and standardized test procedures used to make measurements and obtain the data used by policy makers.  Document review and interviews both show that government policy and regulatory groups at NRCan, Environment Canada, and Health Canada are major beneficiaries of the AFTER Program.  As such, it is appropriate that research groups from these departments led those research projects aligned with their needs and interests. 

AFTER is primarily funded by PERD.  As PERD is managed by NRCan OERD, the role of OERD in the high-level oversight of AFTER is both necessary and appropriate. PERD requires departments to support projects through financial and /or in-kind contributions. The majority of research projects involve measurement of engine performance or mid-stage technology development (i.e., laboratory validation and prototypes), with several additional years of technical development required before potential application.  The research projects with potential industrial application are mid-stage, with high risk and are unlikely to be fully funded by industry.

P&E

P&E program stakeholder groups are federal researchers, regulators and policy groups.  Under CEPA, Environment Canada has the responsibility for establishing emissions regulations for vehicles and fuels and the mandate to research fuel emissions.  The vision for the P&E Program remained essentially unchanged from the previous funding cycle.

Document review and interview findings show that the primary output of the Program is improved information for decision-making on Canadian (and in some cases US) transportation policy and regulations.  As Canada’s regulations in this area are harmonized with those in the US, there has been a shared workplan for fuels and emissions research in place between the US and Canada (US EPA and EC) over the last funding cycle. 

A number of interviewees perceived the distribution of PERD funding to favour NRCan-led projects and technology-based projects over those within P&E that typically related to transportation-related emissions on health and the environment. 

ASM-NGV

Participation by the federal government has helped to mitigate the R&D risk to Canadian industry in some fundamental areas, while building private and public sector capacity.  The ASM-NGV Program Plan (2007) also notes that this program is consistent with the approach used in the US, China, Germany, Australia and Britain.

There are a number of federally funded R&D programs in place that complement ASM-NGV (e.g., Auto21, Automotive Partnerships Canada and MagNet).  ASM-NGV coordinates with these initiatives by including representatives of these organizations on the Advisory Committee, participating in their program committees, and co-funding projects where appropriate. 

According to interviewees (public and private sector) NRCan-MTL has the unique facilities and capacity (e.g., complex modelling and simulation) needed to support ASM-NGV projects, and long-established ties with industry, other research organizations, including NRC, NSERC and related R&D programs.  With this network, NRCan‘s role in managing and implementing the Program is appropriate.

5.2   Performance (Summary followed by details by Program)

Evaluation Issue	Lines of evidence	Assessment
To what extent have intended outcomes been achieved as a result of the program?	Document review File review Interviews Case studies Bibliometrics	Generally short-term outcomes have been achieved and good progress has been made towards intermediate and long-term outcomes.

Evaluation Issue	Lines of evidence	Assessment
Have there been any unintended (positive or negative) outcomes?	File review Interviews Case studies	Positive unintended outcomes were identified in the H2FC and AFTER  programs.

Evaluation Issue	Lines of evidence	Assessment
Are the programs and activities the most economic and efficient means of making progress towards intended outcomes?	Document review File review Interviews Case studies	Individual programs are managed efficiently and economically.  There are opportunities for coordination and integration of federal activities related to transportation R&D within and beyond the CTS portfolio.

A contribution analysis was completed for each CTS Program to examine how each program unfolded against expectations, from needs identification through to long-term outcomes, and the factors that affected performance.  The following pages present the results of this analysis against evaluation issues for each of the five CTS programs.

5.2.1   Hydrogen and Fuel Cells Program

5.2.1.1 Objectives Achievement

To what extent have intended outcomes been achieved as a result of the programs? 

Summary Findings

The Program funded 34 R&D and demonstration projects over the period under review and early results are evident in the areas of: R&D networks and international cooperation; new technologies and knowledge; and, codes and standards. 

• The Program helped maintain research linkages between and among industry and government labs, and promoted international cooperation through participation in IEA Committees on Hydrogen and Fuel Cells, International Partnerships for the Hydrogen and Fuel Cells in the Economy (IPHE) and ISO.  However, the Program did not have significant linkages with other federal initiatives (e.g., Auto21, Automotive Partnership Canada (APC)^{Footnote 27}). 

• H2FC contributed to fuel cell technology development in industry (e.g., membrane technologies, as demonstrated in the dPoint Humidifier and Ballard Proton Exchange Membrane (PEM)^{Footnote 28} System for Hydrogen Generation case studies) and enhanced public sector capacity for testing (e.g., new fuel cell testing stations at NRC-IFCI as described in the Prototyping PEM Assemblies for Automotive Fuel Cells case study).  The intermediate to long-term impact of technology developments on industry partners is uncertain.  For example, dPoint has recently refocused its R&D efforts on building systems and in 2009 Ballard sold its automotive fuel cell division (AAFC) to Daimler / Ford (the research facility is still located in Vancouver).  

• In January 2007, the Canadian Hydrogen installation Code (CHIC) was published by the Bureau de Normalization du Québec (BNQ) as a National Standard of Canada.^{Footnote 29}  It provides Canadian industry and regulatory authorities with guidance for approving hydrogen as an energy carrier and facilitating the approval of hydrogen installations across the country.  In 2007-08, the H2FC Program provided funding to the BNQ CHIC technical committee for revisions and amendments to the code, and to implement the code for new hydrogen installations.  

• Ten H2FC demonstration projects were funded by ecoETI.  The Airports Demonstration case study highlighted a number of potential issues with achieving outcomes through demonstration projects given the stage of technology development, and the absence of suitable regulations to facilitate the safe deployment of those technologies.

Discussion and Analysis

Participation in program planning and degree of outreach to federal and industry stakeholders was initially strong, but has declined in recent years as federal funding decreased:  H2FC program planning and project selection involved key federal departments (NRC, Transport Canada, Industry Canada, DND and NSERC) through the H2FC Program Executive Committee, comprised of senior federal advisors.  Three interviewees expressed concern with Program Executive Committee membership (i.e., possible conflict of interest as some members had submitted proposals to the program), the limited external input to the review process, and the extent to which each project received a comprehensive technical review.  The Program also convened a Project Committee for federal researchers and project managers.  Both the Program and Project committees meet once a year to review progress and consider the need for any redirection. 

Information is also shared through the large (20 plus federal representatives) interdepartmental Hydrogen and Fuel Cell Coordinating Committee (H2FCCC), which is primarily a forum for information exchange.  H2FCCC met up to six times a year between 2001 and 2008, during which time there were several similar federal funding programs in place to fund hydrogen and fuel cell research, development and demonstration.  According to the program manager, the frequency of H2FCCC meetings declined in the past four years when available discretionary federal funding (through CTS and other programs) decreased and there was little to discuss. 

The Hydrogen Technical Advisory Committee (HyTAG) was comprised of six to eight industry and academic experts and until 2009 met annually to provide high-level strategic advice regarding the nature and direction of hydrogen and fuel cell research.  The most recent meeting was in June 2009 at which time, due to several vacancies on the committee, it was decided to reconsider its structure and purpose.  According to the program manager, that process did not take place as it became apparent that the federal interest in the sector was declining. Several interviewees attributed the overall decline in interest in all committees to the year-over-year decline in federal investment in hydrogen and fuel cell R&D, which was matched by a reduction in industry R&D spending.  The program manger notes that consultations with industry took place at other fora including the annual Canadian Hydrogen Fuel Cell Conference, the biennial World Hydrogen Energy Conference, the annual Canada-US Transportation Technologies and Fuels Forum (TTFF), and IEA and IPHE meetings. 

Partners in program delivery include the primary federal departments with an interest in hydrogen and fuel cell research as well as other key stakeholder groups:  Government partners in program delivery include:

• NRC – The NRC Fuel Cell Program brings together expertise from six NRC research institutes, including the Institute for Fuel Cell Innovation.  (Due to the recent (2012) reorganization at NRC, NRC-IFCI no longer exists as a separate entity.)
• Department of National Defence – DND is involved in developing technology for micro power applications and in the International Energy Agency Annex VIII (PEM Fuel Cells) through the Royal Military College.
• NSERC – Through its strategic grants and partnerships programs, NSERC supports basic and applied R&D at Canadian universities.  It has supported a number of H2FC related networks, including the Solid Oxide Fuel Cell (SOFC) network and the Hydrogen Network.
• Industry Canada – Industry Canada plays a role in policy development and sector analysis.  It co-chairs the Hydrogen and Fuel Cell Committee (along with NRCan).

Other Program stakeholders include industry, universities, codes and standards organizations (national and international), and regulatory groups.

H2FC projects built on previous R&D activities funded by PERD, T&I, ecoETI, and the Canadian Transportation Fuel Cell Alliance (CTFCA); the majority of interviewees reported that the 34 H2FC projects reflected sector needs: H2FC projects were developed and managed through cost-shared agreements and partnerships with key target groups including industry, academia, provincial governments and other federal organizations, which helped to ensure that projects were aligned with sector needs.  Proposals were reviewed by the H2FC Program Executive Committee and recommended to the CTS Portfolio Committee for approval.  PERD funding was allocated to 17 projects, ecoETI funded 10 demonstration projects, and CEF funded 7 projects.  ecoETI funding was directed primarily to the private sector to complete nine demonstration projects were initially funded by the Canadian Transportation and Fuel Cell Alliance (CTFCA), a seven-year program that developed and demonstrated hydrogen-fuelled vehicles and fuel stations across Canada.  The Airports Demonstration project (which was the subject of a case study for this evaluation) was the only project that was not a continuation of an earlier project.

H2FC funding facilitated international outreach / cooperation:  H2FC Program funding supported Canada’s (NRCan’s) participation in international fora including the International Partnership for Hydrogen and Fuel Cells in the Economy (IPHE), IEA (Hydrogen and Fuel Cell Implementing Agreements) and ISO standards setting bodies.  The IPHE has coordinated members’ activities since 2003 to accelerate the adoption of hydrogen and fuel cell technologies into the global economy.  The four priority focus areas of the IPHE are: 1) accelerating the market penetration and early adoption of hydrogen and fuel cell technologies and their supporting infrastructure; 2) policy and regulatory actions to support widespread deployment; 3) raising the profile with policy-makers and the public; and 4) monitoring hydrogen, fuel cell and complementary technology developments.  IPHE has 18 member governments.

H2FC led to new R&D partnerships at the project level, but limited program-level interaction with related initiatives:  Individual H2FC projects led to some new R&D partnerships and collaboration by R&D performers.  The extent to which linkages with other Canadian funding initiatives (Auto21^{Footnote 30}, NSERC, APC) were developed / maintained does not appear to be significant. 

The H2FC Program contributed to new breakthrough fuel cell and hydrogen technology development:  Findings from interviews, file review and three case studies (Ballard PEM System for Hydrogen Generation, dPoint Humidifier, and Prototyping Proton Exchange Membranes (PEM) Assemblies for Automotive Fuel Cells) show that R&D projects targeted research priorities as identified by government labs in consultation with industry proponents, and built on R&D results from the Hydrogen Energy Economy Program and CTFCA.  Several interviewees noted that funding from other sources might have had a greater impact on technology development (i.e., Ballard’s project with DOE) due to the relative size of the programs and the (often) longer-term and more specific objectives.

The objective of most H2FC R&D projects was to reduce the cost and/or improve the performance of hydrogen and fuel cell technologies; others were directed at safety issues and related codes and standards.  Noteworthy program highlights include:

• In the dPoint Humidifier case study, Program funding assisted the company to develop a membrane architecture that is more efficient, smaller and over 15 times less expensive than previous designs used in fuel cell humidifiers.  The availability of low-cost fuel cell humidifiers is important because it could accelerate the adoption of PEM fuel cell technologies in the automotive and other markets.  PERD funding was viewed by senior company managers as essential to moving the company from early stage R&D to commercialization, and its growth from 8 employees in 2007 to 30 in 2012, particularly in the context of the current economic climate and the scarcity of venture capital.  However, the case study also showed that fuel cells now account for less than 10% of dPoint’s revenues.  The majority of the company’s revenues are from the sale of Energy Recovery Ventilators (ERVs), which use the membrane architecture developed with PERD funding (an unintended impact of the PERD funding).  (ERVs are added to building heating, ventilation and cooling (HVAC) systems to improve the energy efficiency of building operations.) The company will continue to monitor the fuel cell market. However, company representatives note that without sustained federal R&D support the level of continued company investment in fuel cell R&D and applications is uncertain.

• The Prototyping PEM Assemblies for Automotive Fuel Cells case study illustrated the benefits of involving vertically-integrated, private sector partners.  The project was delivered by NRC-IMI in collaboration with three private sector partners (GM, Ballard and Greenlight), which represent different stages of the supply chain.  Results included new processes, testing equipment and PEM materials with a significant increase in durability.

• Hydrogenics Corporation made improvements in hydrogen production from PEM water electrolysis technology.  These improvements included the continuous operation of the PEM water electrolyzer for 22,000 hours with no failure and no performance loss.  (Previous materials had failed after approximately 8,000 hours.)  The electrolyzer also exceeded the US DOE energy efficiency target for 2017.

• Defence Research and Development Canada is researching the integration of fuel cell auxiliary power units for mobile military applications.  In 2010-11, the project made advancements in four areas: hydrogen cylinder safety, fuel cell performance and material costs, system design, and demonstrations of fuel cell use on-board vehicles. 

H2FC R&D supported the development of codes and standards and participation in international codes and standards committees:  In January 2007, the Canadian Hydrogen installation Code (CHIC) was published by the Bureau de Normalization du Québec as a National Standard of Canada.  The CHIC provides Canadian industry and regulatory authorities with guidance for approving hydrogen as an energy carrier and facilitating the approval of hydrogen installations across the country.  The CHIC was instrumental in developing the basis for a standardized regime for regulatory approval of hydrogen installations and equipment across Canada and ensures that Canadian producers of electrolysers automatically meet international requirements and therefore can have their products certified in international markets based on the ISO standard.  According to a senior level NRCan interviewee, since being published several Canadian companies have directly benefited from the CHIC, and Ontario and BC have adopted the CHIC into their provincial regulations.  Quebec, which has not formally adopted the CHIC, used the Code requirements in their approval of the refuelling station for the Airport Demonstration Project.

The H2FC Safety, Codes and Standards activity, including R&D projects and financial support for travel, supported Canada’s participation in key international standards setting organizations.  For example: (i) the R&D project for simulation work on hydrogen wall jets and hydrogen venting (by the Université du Québec à Trois Rivières) fed into Task 19 (Hydrogen Safety) of the IEA Hydrogen Implementing Agreement; and (ii) the Bureau de normalisation du Québec had stewardship for ISO TC 197 on Hydrogen Technologies, including the development of hydrogen fuel product specifications (supported by H2FC funding).^{Footnote 31}

H2FC projects led to the development of new capacity, tools and equipment for fuel cell R&D: The Prototyping PEM Assemblies for Automotive Fuel Cells case study illustrates the range of H2FC impacts.  The project involved NRC researchers, and research staff at GM, Ballard and Greenlight – companies representing different links in the fuel cell supply chain.  Greenlight developed customized testing equipment for NRC and the other companies helped ensure that the equipment and subsequent fuel cell prototypes met industry needs (for the automotive and stationary markets).  As stated in the final project status report, the new state-of-the-art testing stations give industry and public sector researchers unique access to equipment to assess trade-offs between improved performance, durability and cost in a more timely and cost-effective manner.

H2FC Demonstrations faced some unique challenges associated with codes, standards and safety issues: The Airports Demonstration case study (which realized limited success in meeting its original targets for the development, implementation and operation of hydrogen and fuel cell technologies in an airport environment) illustrated a number of challenges associated with demonstration projects that contributed to poor project performance:

• The complexity of the project was underestimated at the project design stage;
• The codes, standards and safety measures needed to support new technology deployment (especially in a sensitive, risk-averse commercial environment) were not in place;
• The project did not involve the required partnerships and collaboration across the supply chain; and
• A lack of buy-in from the targeted end-users (i.e., airport staff).

As noted above, this was the only ecoETI project that was not a direct follow-on from a CTFCA funded project and, based on interview findings and project file reviews, the others achieved better progress towards their objectives and were on schedule (according to the 2010-11 Program Report).  However, project reports and interviews also showed that each of the demonstrations faced issues related to end-user acceptance, lack of widely understood codes and standards, and limited fuelling infrastructure that were to some extent underestimated at the projects’ planning phase. These issues affected the extent to which results could be achieved.

H2FC funded researchers are well cited internationally; however broad dissemination of results within the sector has been a challenge:  Project results are disseminated by project team members through publications, presentations, and conferences. In terms of publications, H2FC has produced the highest quantity of funded papers over the evaluation period (430 followed by AFTER with 170). As well, these publications are cited 88% more often than the world average for papers in this field. Having said that, there is no formal way of broadly promoting H2FC results to Canadian industry and unless companies were participants in the projects they are unaware of the results.  Several industry and public sector interviewees indicated that OERD could play a more significant role in communicating H2FC results (and CTS findings more broadly) within the sector to further leverage research results.  That said, R&D results have informed discussions and input to working groups under the IEA Hydrogen and Fuel Cell Implementing Agreements.

The long-term H2FC program outcome is the ‘deployment of hydrogen and fuel cell technologies that contribute to the well-being of Canadians by reducing emissions of air pollutants and greenhouse gases.’  At present, the deployment of such technologies is limited mainly to small-scale applications (e.g., stationary, back-up power, and material handling applications) rather than commercial on-road transportation.  For example, a fleet of 20 buses is in operation in Vancouver, and a Wal-Mart distribution center in Alberta invested in a fleet of 75 forklifts.  The company estimates that the forklifts will result in an estimated reduction in GHG emissions of 530 tons/year and save approximately $2 million in operating costs over seven years.^{Footnote 32}

As the technologies become cost-competitive their adoption by automotive manufacturers will contribute to cleaner transportation systems.  However, according to interviewees, the timeframe for these impacts is in question.  Interview and document review findings show that the high levels of interest and spending (by industry and government) in the late 1990s and early to mid-2000s on fuel cell and hydrogen R&D did not lead to the expected results (e.g., the projected market for fuel cell powered automobiles by the mid 2000s did not materialize).  Manufacturers are now projecting commercial fuel cell automobile sales by 2015 (mainly in California, Japan and Europe, particularly Germany where re-fuelling infrastructure is expected to be deployed).

5.2.1.2 Unintended Outcomes

Have there been any unintended (positive or negative) outcomes?

Summary Findings

Just one significant unintended outcome emerged during the evaluation.  Some H2FC project results led to unexpected spin-off benefits in other sectors and/or additional research in complementary research fields.

Discussion and Analysis

Spin-off benefits have accrued in sectors other than road transportation:  As noted above, dPoint began as a fuel cell company in 2005; however, fuel cells now account for less than 10% of the company’s revenues, and Energy Recovery Ventilators (ERVs), which improve the energy efficiency of building operations, account for 90%.  As a result of the slow development of the fuel cell sector the company is now focused on applying its membrane technology (developed with PERD support) in ERVs.  dPoint was recently awarded an ecoEII grant under the Buildings Portfolio to pursue the development of this market. 

5.2.1.3 Internal and External Factors

What are the factors (both internal and external) that have facilitated or hindered the achievement of expected results?

Summary Findings

A number of internal and external factors related to public management, program context and stakeholder engagement negatively affected Program success.  Annual Program funding declined by approximately 50% over the past two years of the Program, which affected the extent to which the Program’s original priorities and objectives could be pursued.  A number of committees / organizational structures were either disbanded or became less active during the course of the Program, giving stakeholders fewer formal means of providing input to, or hearing about results of, project activity. 

Several external factors also hindered outcomes: the slower than expected rate of fuel cell technology / market development (in Canada but also internationally), limited Canadian industry capacity, international R&D priorities funding, and the declining federal support for all hydrogen and fuel cell R&D.

Discussion and Analysis

Internal factors

The lack of a federal-level strategy creates uncertainty vis-à-vis research priorities and approach:  At the time of this evaluation, there is no integrated, strategic planning document for fuel cell and hydrogen research at a federal level within Canada.  Given the number of stakeholders with an interest in H2FC R&D (NRCan, NRC, NSERC networks, individual universities), and the number of funding initiatives, interviewees felt that a federal level strategic plan is needed to identify federal R&D roles and technology gaps in the context of industry needs. That said, it is important to note that there is no single entity tasked with developing such a strategy (not OERD nor any other organization).

Year-over-year reductions in H2FC funding created concerns for all stakeholders regarding the level of federal commitment to this technology:  Annual PERD funding for the H2FC program declined by $2 million over the five years under review (from $4.35 million in 2007-08 to $2.35 million in 2011-12), increasing the relative contribution and reliance on C-base funding (e.g., ecoETI and CEF).  Prior to 2007-08, H2FC was part of the Hydrogen Energy Economy program, which included some funding for electric vehicles, though the focus was on hydrogen. Creating the EM program to focus on electric mobility has led to some of that funding now going to electric vehicle projects rather than hydrogen. According to OERD presentations to, and endorsed by, the ADM and DG level PERD committees in 2007-08 and 2008-09, H2FC was asked to refocus on longer term R&D, while plug in electric vehicles were seen as more likely to have near and medium term impacts on clean transportation as well as potential to be a stepping stone for fuel cell vehicles. This reduced funding for fuel cell projects created uncertainty among project proponents as to the federal government’s long-term commitment to the development of hydrogen fuel cell technology and sector. 

External factors

H2FC technology and market development has been slower than expected:  Hydrogen and fuel cell technologies face a number of challenges to commercialization and widespread adoption (e.g., cost, performance, codes and standards, infrastructure) and the commercialization of fuel cell cars did not occur by 2005 as was projected in the 1990s.  Interview and document review findings show that Original Equipment Manufacturers in North America, Europe and Asia continue to develop fuel cell vehicles and other types of electric vehicles.  In recent statements posted on company websites, Toyota, GM, Daimler Benz and Honda announced their intentions to market fuel cell cars by 2015 (primarily in California, Japan or select European markets).  Public and private sector investment in related infrastructure (e.g., hydrogen fuelling stations) in Canada significantly lags that of Europe, especially Germany.  This is a market reality that affects firms’ ability to commercialize fuel cell technology for the transportation sector in Canada.

Participation by NRCan (and OGDs) in international committees supported Canada’s regulatory agenda and helped align Canadian standards with international standards: Canada’s past active participation on international committees responsible for developing the regulatory environment for hydrogen and fuel cells (codes, standards) helped ensure that Canadian perspectives were well represented, and promoted alignment between developing Canadian and international codes, standards and regulations.  With H2FC financial support, Canada participated on the IEA Hydrogen and Fuel Cell Implementing Committees (leading specific sub-tasks related to hydrogen safety), and several ISO and US committees. 

Canadian fuel cell and hydrogen industry capacity has declined in the last five years: In the early to mid-2000’s Canadian companies, and related R&D capacity, was considered to be ahead of much of the international competition.  According to program managers and industry interviewees, many (previously key) industry players are no longer active in hydrogen and fuel cell development.  The recent sale of several Canadian companies (e.g., AFCC, the automotive fuel cell division of Ballard^{Footnote 33}, to Daimler^{Footnote 34} for 50 million Euro, and the sale of both General Hydrogen and Cellex to a US firm), company bankruptcies, a re-focusing of some companies’ research efforts away from fuel cell technology and an overall decline in R&D investment, led a number of interviewees to question the extent to which Canada had maintained (or would be able to maintain) this leadership position.

5.2.1.4 Economic and Efficient

Are the programs and activities the most economic and efficient means of making progress towards intended outcomes?

Summary Findings

Overall, the H2FC program has been managed efficiently, although there are several areas in which the efficiency of the program could be improved. H2FC leveraging of partner investments over five years is estimated at $37.5 million for a leverage ratio of 1:1.3 and is on par with the total leverage ratio for the CTS portfolio (i.e., 1:1.5). The discussion of factors affecting Program outcomes (above) included a number of items that reflect administrative gaps and consequently may affect the cost-effectiveness of the Program.  For example, the lack of federal strategy, and level and approach used to involve industry may have affected Program effectiveness (the Program planning and review processes may not have fully engaged industry, thus affecting the overall effectiveness of the Program design).  The processes used by the US DOE for their Hydrogen Program (with similar goals and funding approaches as H2FC) to solicit and evaluate project proposals, were identified by a number of interviewees (with experience with the DOE Program) as more transparent than those for the H2FC Program. While the US DOE review process is considered very high quality, substantial resources are committed to this which are not available to PERD.

Discussion and Analysis

Over the period 2007-08 to 2011-12, the H2FC Program received approximately $30 million in PERD, T&I, ecoETI and CEF funding.  The total investment in H2FC activities from all partners is estimated at $37.5 million for a leverage ratio of 1:1.3.

CEF, ecoETI and the T&I Program used the existing federal science and technology platforms (e.g., PERD Committees, CTS Portfolio Committees) to support program delivery.  According to project leaders and NRCan managers this approach helped develop project opportunities in a timely manner.  As noted earlier, CEF support was for two-years and most projects were completed within 18 months following a six-month project selection process, which would have taken longer in the absence of existing PERD / CTS platforms.

OERD reduced H2FC funding by $1 million a year in 2008-09 and a further $1 million in the following year.  In 2007-08, the Program was also renamed (from the previous Hydrogen Energy Economy Program) and a new vision was developed that involved eliminating some R&D activities and re-focusing others.  According to the Hydrogen and Fuel Cell Program Plan^{Footnote 35} OERD provided direction to move away from R&D for hydrogen production from fossil fuels and gaseous hydrogen cylinder storage technologies.  In areas of continuing R&D, the H2FC Program Executive Committee had to determine more narrow outcomes to reflect the decline in funding.  Evaluation interviews and document review did not reveal a clear process for arriving at this funding decision. 

The impact of federal spending reductions on hydrogen and fuel cells is illustrated by one of the case study projects.  The Prototyping PEM Assemblies for Automotive Fuel Cells NRC-IMI project, that developed fuel cell prototyping techniques and equipment, noted in its final Project Status Report that the extent to which this project will continue to support CTS objectives depends in part on the future of H2FC public and private sector funding.  The reduction in PERD funding for fuel cell and hydrogen research, and the reduction in industry R&D funding makes the future use of the knowledge, protocols and materials technologies as applied to fuel cells (stationary or automotive applications) less certain.

The lack of an inter-departmental strategy and declining levels of industry and federal involvement in the program over the past five years had a negative effect on the efficiency of program planning (i.e., no federal wide strategy to help inform future investment decisions).  These factors also affected a number of key outcomes, as there were fewer opportunities to broadly disseminate R&D findings to industry and public sector stakeholders, which negatively affects the efficiency of technology uptake and network development outcomes.

There is evidence that some projects (e.g., the Airports Demonstration case study) did not reflect the stage of technology and infrastructure (codes and standards) development needed to complete an effective hydrogen fuel cell demonstration project.  This raised questions as to how consistently technical reviews were conducted on H2FC project proposals, and how these findings informed project selection.  The limited external input to H2FC project selection and concern about committee membership were identified as issues for the 2007-08 funding cycle.   Some interviewees noted the potential for conflict of interest in the project approval committee, which often included one or more project proponents.  Interviewees noted the limited external review of H2FC projects and compared the process with that used by US DOE.  Several interviewees had experience with US DOE Programs and indicated that the H2FC approach to project selection, monitoring and reporting practices did not compare favourably to those used by DOE.  At a strategic level, the US DOE prepared an Interagency Action Plan for Hydrogen and Fuel Cells (December 2011) that described an integrated plan for coordinating US federal agency efforts to research, develop, demonstrate and deploy hydrogen and fuel cells.  At the project-level, annual project evaluations take place and a team of reviewers formally reviews each project.  While the US DOE review process is considered very high quality, substantial resources are committed to this which are not available to PERD.

The separate categorization of H2FC fuel cell technologies and Electric Mobility technologies in CTS Portfolio planning documents draws a distinction between fuel cell and battery powered vehicles that some stakeholders (industry, federal researchers) noted does not exist from a technical standpoint.  Furthermore, the categories are not considered exclusive by key studies on the subject, including the report by McKinsey ‘A Portfolio of Power-Trains for Europe: A Fact-Based Analysis’^{Footnote 36} that compares battery electric vehicles, plug in hybrid vehicles and fuel cell electric vehicles technology benefits and possible contributions to Europe’s carbon dioxide reduction target for 2050. According to OERD, to respond to evolving PERD priorities, brought about in part through reductions in PERD funding overall, that called for a focus on electric vehicles, the new EM program was created as a distinct program using some of the funding that was originally within the H2FC program for fuel cells.  This refocusing was endorsed by the interdepartmental ADM Panel and DG Committee that are part of the PERD governance structure.

5.2.2   Electric Mobility Program

5.2.2.1 Objectives Achievement

To what extent have intended outcomes been achieved as a result of the programs?

Summary Findings

The EM Program, a new CTS program, funded 16 research projects in this cycle (2007-08 to 2011-12).  Immediate and intermediate outcomes fall under two categories: reach and engagement, and project results and early outcomes. The evidence suggests that the Program has made good progress towards intended outcomes.

In designing the program, EM engaged a wide variety of stakeholders during the planning phase through interviews and a workshop.  Members of the EM Management Committee and technical experts were involved in project selection.  Several private sector firms involved in PHEV manufacture, electrical utilities and researchers from government departments with interest in PHEVs were involved in projects as performers and partners.  In addition to supporting Canadian electric vehicle policy and regulatory needs, EM developed valuable links with the US EPA and international organizations (e.g., through IEA programs). 

The majority of EM projects involved technology development. Improving batteries was the main focus.  Projects led by NRC involved mid-stage research to improve the performance of battery components.  Projects led by industry focused on close-to-market applications and testing. At least one project led to commercial application, and technology developed in another is expected to be used in a next generation manufactured product.  Policy and regulatory applications of EM work were also evident. One project supported harmonization of regulations with the US by providing performance data on hybrid electric vehicles (HEVs) to the US EPA to support the development of HEV regulations.  EM also supported the needs of electric power utilities and governments for information on the effect of PHEVs on the electrical power distribution system by producing reports and software that are being used to understand the implications of the widespread introduction of PHEVs and other HEVs to the transportation fleet.  Technical results were also disseminated beyond program participants through conferences, workshops, and publications (cited on par with the world average for that field).

Discussion and Analysis

There was significant stakeholder engagement in program planning and implementation:  Considerable effort was spent to engage key public and private sector stakeholders in the identification of R&D priorities as part of the program planning process.  In 2006-07, before the EM Program began, PERD provided a small amount of funding to NRC to identify priorities and develop a full research program.  During that first year, an environmental scan was completed, involving interviews with over 30 key North American stakeholders.^{Footnote 37}  NRC also led a workshop with the hybrid vehicle R&D community and over 55 representatives of Canadian and American public and private sector organizations and universities attended.  Based on the priorities identified through these activities, the EM Program was created in 2007-08.  Program planning and oversight is supported by the Program Management Committee, which includes representatives of government department policy groups and OERD.^{Footnote 38}  A number of public and private sector stakeholders participated in research projects.  These included government researchers^{Footnote 39}, firms involved in battery and PHEV component manufacture, the US EPA and Argonne National Laboratory.  Interviewees reported that Canadian electric power utilities did participate but not to the extent expected due to concerns about sharing of utility data. 

Most EM research projects (12 of 16) involve technology development, three involve performance measurement and testing and one analysis of the impact of PHEV charging on the electrical grid. The following is a general description of the types of projects, their application and outcomes. 

Key technology development areas were targeted, and several prototypes developed:  Eight projects involve improvements to the capacity, safety and reliability of Li-ion batteries, identified as the highest priority by stakeholders. Four were carried out by NRC researchers and one by a university.  In two of the three projects carried out by private sector firms, the companies involved have developed prototypes. This included batteries developed with improved cold weather performance that were evaluated in a demonstration involving a car manufacturer. EM also carried out research associated with other PHEV components.  As described in one case study^{Footnote 40}, a project involving the development of improved electric drive components was only partially successful technically. However, the private sector partner has continued development in-house and is incorporating the technology in a new product.

Project results contributed to the development of policies and guidelines:  Several projects involving performance measurement and testing of batteries supported the development of policies and guidelines by government and industry. One project led by Environment Canada ERMS researchers developed standardized test procedures for measuring emissions and fuel economy in PHEVs and medium and heavy duty Hybrid Electric Vehicles (HEVs), and made measurements for use by the EPA and others.^{Footnote 41}  Through this project, Canada participated in the development of North American harmonized transportation-related policies and regulations. Another project found wide variation in the performance of different types of Li-ion batteries under different temperatures and operating conditions. These results emphasized the need to test batteries under standardized conditions to properly inform consumers.

EM research also produced several PHEV policy background studies as well as a report entitled “The National PEV Charging Infrastructure Deployment Guidelines and Policies for Development and Deployment of Hybrid Vehicles.”^{Footnote 42} This report is being used by electric power utilities to support the development of strategies for responding to the widespread deployment of PHEVs in the transportation vehicle mix.  The report is also being shared with municipalities and others to inform the development of infrastructure to support the charging requirements associated with increased use of PHEVs in the Canadian on-road fleet.

Results were disseminated through annual workshops, industry conferences and publications:  The results of EM projects are disseminated beyond project participants in a number of ways.  The scientific results are disseminated widely to other researchers through publications and conference presentations. The bibliometric study results showed that while EM produced the lowed number of NRCan-funded publications during the evaluation period, they were well-cited in the field (just as often as the world average). Also, there has been encouraging growth in publications observed over the Program’s short lifespan adding 3.3% to Canada’s share of research in this field in just 4 years.  EM also invites policy and regulatory groups and other stakeholders to semi-annual workshops where project results are presented and discussed.  EM also shares project results through the annual TTFF and Canada-US Clean Air Dialogue attended by Canadian and US transportation related policy, regulatory and research groups. These are important mechanisms that contribute to the harmonization of Canadian transportation regulations with those of the US.

During the first funding cycle (2007-08 to 2011-12), most EM projects carried out mid-stage research and were successful in advancing several technologies closer towards commercialization, particularly battery components.  In at least one project, commercial application has already taken place, and in another, it is expected that technology will be used soon in a next generation manufactured product.  The Program has also generated battery performance data used by Canadian and US transportation policy regulatory groups for use in developing upcoming regulations related to on-road electric vehicles.

However, EM alone is unlikely to influence growth in the use of PHEVs significantly.  As discussed in the PERD funded 2009 industry-led Electric Vehicle Technology Roadmap, Canada would need to develop a broad range of technical infrastructure and supportive policies, regulations and programs to complement the research carried out by EM.

5.2.2.2 Unintended Outcomes

Have there been any unintended (positive or negative) outcomes?

Summary Findings

There is no evidence of significant unintended outcomes arising from the EM program. 

5.2.2.3 Internal and External Factors

What are the factors (both internal and external) that have facilitated or hindered the achievement of expected results?

Summary Findings

Internal factors contributing to EM success include the adoption of the successful AFTER governance approach, the effective use of existing research and testing capacity at NRC, NRCan and Environment Canada’s Emissions Research and Measurement Section (ERMS), and existing linkages with US transportation regulatory authorities developed by AFTER and P&E to jump start the program.  However, the current governance structure does not permit non-government members to be on the EM Management Committee to represent the interests of the private sector or power utilities. 

In terms of positive external factors, EM has successfully engaged electric battery manufacturers and an electric motor manufacturer in projects.  However, relations with the Canadian electric power sector have been mixed.  While very interested in how electric vehicles will affect their industry, utilities are protective of their data and reluctant to share, which has affected progress in some projects.  There has also been a lack of effective co-ordinated public sector support for the development of the policies, regulations and infrastructure identified in the PERD funded 2009 industry-led Electric Vehicle Technology Roadmap as necessary to complement the EM R&D Program.  The low level of government investment and action has contributed to the slow penetration of PHEVs and low level of industry interest. 

Discussion and Analysis

Internal factors

Use of Existing Research Capacity – As a new program, EM was required to take advantage of existing research capacity and build new capacity where required.  The vehicle performance and emissions testing capabilities at Environment Canada ERMS and the modelling expertise at NRCan CanmetENERGY each contributed to the success of projects.  The expertise in electrochemical materials at NRC ICPET and materials and processing at NRC IMI has provided the basis for developing research capacity needed for the development of a range of improved battery components. 

Utilization of Existing Partnerships and Linkages – EM was able to take advantage of existing relationships with the US EPA and DOE that were developed through AFTER and EM Program planning activities.  US agencies were already working with Canadian partners on transportation related emissions and energy efficiency through the Transportation Technology and Fuels Forum (TTFF) and the Canada-US Clean Air Exchange and it was relatively straightforward to extend the discussions to include PHEVs.  Similarly, Canada’s long-term involvement in IEA transportation related initiatives has contributed to enabling EM to become quickly involved.  

Limited Industry Inclusion in Program Oversight and Information Dissemination: The current governance does not permit representatives of non-government stakeholder groups (i.e., manufacturers of electric vehicle components, electric power utilities) to be on the EM Management Committee to represent the interests of these groups. Program staff indicated that this decision was consciously taken in order to avoid the appearance of favouritism towards one company’s interests.  Instead, there is an ex-officio member from Sustainable Development Technologies Canada to represent Canadian industry’s interests.  However, interviewees indicated that this has also meant that there was no direct industry input into the discussions of the EM management committee when allocating funding once the program was underway to ensure it remains linked to broad industry needs.  The program does convene mid-year technical review meetings which those not involved in projects are allowed to attend. That said, several interviewees not directly involved in projects said that they would appreciate being kept better informed about the program and initiatives within their area of interest.  Not having that information increases uncertainty and affects their level of involvement in and support for the Program.

External factors

Lack of Co-ordinated Public Sector Support for Electric Vehicles: The PERD funded 2009 industry-led Electric Vehicle Technology Roadmap identified a suite of initiatives that together would help develop a Canadian electric vehicle industry, support the widespread introduction of PHEVs into the transportation fleet, and contribute to the reduction in use of fossil fuels and GHGs and to a more sustainable energy mix.  The initiatives identified required a substantial investment in Canadian development and manufacture of EVs and energy storage devices.  The Electric Vehicle Technology Roadmap also identified a number of government initiatives that are needed to support the growth of electric vehicles in Canadian transportation.  These include:

• support for R&D and commercialization of advanced EV technologies;
• development of government regulations for EV performance and safety; and
• development of regulatory infrastructure for charging (building codes, electrical codes, power generation standards, etc.).

Several interviewees were concerned about the lack of action on these recommendations.  The EM Program is one element of a comprehensive EM support program. However, without the other elements, the relatively small EM R&D program can make only limited progress.

Linkages to Canadian EV Manufacturing Sector – While the Canadian PHEV related industrial sector is small, there were several firms involved in battery design and manufacturing and electric drive components which participated in the Program through contract research and as partners in research projects led by NRC.  In general, the contract research involved projects closer to market. The participation of these firms contributed to a better understanding of the needs of industry and to the relevancy of the Program to industry.

Good Linkages to Electric Power Utility Sector, but lack of sector data sharing has impeded results – The North American electric power utility sector is very interested in the introduction of PHEVs, as they will have an impact on both generation and distribution infrastructure.  EM was able to engage the utilities in the program.  For one project, led by CEATI^{Footnote 43}, a utility research organization, a number of Canadian and US utilities co-funded several studies on the requirements for charging infrastructure of direct relevance to their needs.  However, interviews and the case study for the modelling project led by NRCan CanmetENERGY found that the unwillingness of the utilities to share data on power consumption impeded the project and reduced the quality of the model.

5.2.2.4 Economic and Efficient

Are the programs and activities the most economic and efficient means of making progress towards intended outcomes?

Summary Finding

The ability of EM to take advantage of the expertise, facilities, relationships and governance procedures developed by other CTS Portfolio Programs in previous funding cycles has contributed to the efficient and effective launching of this new program.  The familiarity of researchers with PERD, the source of most EM project funding also helped to efficiently implement the program.  The requirement by PERD for at least matching cash and in-kind contributions has helped EM leverage funding for projects, resulting in a leverage ratio of CTS to other funding of 1: l.2.

Discussion and Analysis

PERD, the primary source of funding for EM, requires matching contributions as a requirement for all projects. To meet PERD requirements, EM has been able to leverage both financial and in-kind contributions from a range of public and private sector stakeholders.  All projects are led by a federal employee. For most projects, the organization to which the project leader belongs is the primary project co-funder.  A senior scientist from NRC is the EM Program leader and NRC researchers are the project leaders for five of the 16 projects.  Industrial firms and a utility association also co-fund five projects.

From 2007-08 to 2011-12, the EM Program received $9.2 million in CTS funding from PERD, and leveraged a further $10.8 million from partners and stakeholders including OGDs, industry, universities and other sources.  The overall ratio of EM Program funding to other sources was1:1.2.

As discussed above, the EM Program has been able to take advantage of existing relationships developed by AFTER and P&E to quickly and efficiently build relationships with US and international partners.

5.2.3   AFTER

5.2.3.1 Objectives Achievement

To what extent have intended outcomes been achieved as a result of the programs?

Summary Findings

AFTER funded 13 projects under the current funding cycle, focusing on fuel composition, advanced combustion, engine hardware and aftertreatment and health/environment. Immediate and intermediate outcomes fall under two categories, reach and engagement and project results and early outcomes. The evidence suggests that the program has made good progress on its intended outcomes.

AFTER has engaged researchers and policy groups from the federal government departments involved in transportation-related emissions and fuel efficiency in program planning, project selection and carrying out projects.  AFTER also has close working relationships with researchers at the US EPA and DOE.  As described in the Fuel and Technology Alternatives for Buses case study,^{Footnote 44} AFTER maintains linkages with members from many countries participating in the IEA AMF Implementing Agreement.  Through the technology development projects, AFTER maintains linkages to the engine control equipment sector and major vehicle manufacturers.

AFTER projects have produced extensive emissions and fuel efficiency related data for a number of engine technologies and conventional and bio-fuel blends that has been provided to Canadian, US and international regulatory stakeholders.  Other testing of biofuel toxicity has informed Health Canada regulatory needs.  AFTER data is being used by Health Canada, Environment Canada, Transportation Canada and NRCan and has been provided to the US EPA and DOE to inform the development of policies and regulations related to emerging fuels and emissions.  AFTER projects have advanced the development of several engine technologies associated with emissions control, including patenting and in one case licensing and commercial production. 

Discussion and Analysis

Strong stakeholder engagement in program planning and implementation: The AFTER Program Plan 2009/10 to 2011/12 describes how program plans were modified to reflect current stakeholder needs at the beginning of the funding cycle.  The first task was an environmental scan of the changes in the requirements for information by the Canadian and North American transportation sector policy and regulatory community. Federal departments associated with policies and regulations related to transportation emissions (NRCan, EC, Health Canada) participated in planning, and NRC provided leadership for the Program. An additional important source of information is the annual Transportation Technology and Fuels Forum, which is a joint meeting of US and Canadian groups involved in carrying out transportation related research to inform policies and regulations. Information needs identified through these activities included the health effects of transportation-related emissions, and the effects on emissions of the introduction of ethanol blends of gasoline, biodiesel blends and new engine technologies.  The AFTER Program Plan included these research topics.

The results of the 13 AFTER projects fall into two main categories, technology development and performance measurement / testing.

AFTER projects led to advances in technology development: Six of the 13 projects involved technology development.  Private sector firms participated in AFTER projects involving technology development, particularly those projects with a potential early benefit.  For example, private sector firms developing engine monitoring systems participated in the projects involving development of advanced prototype PM and combustion instability sensors.  The combustion instability prototype sensor is being manufactured and sold for use in engine monitoring applications.  A catalytic equipment manufacturer participated in the development of a novel catalyst to remove NOx and has developed a prototype catalytic converter potential commercial application. 

Several projects have led to issuing of patents and some commercialized technologies: Two projects developing sensors to monitor particulate matter levels in diesel engines and combustion instability in internal combustion engines (ICE) resulted in advanced prototype sensors of interest to engine control equipment and automotive vehicle manufacturers. Several patents were issued.  The prototype combustion instability sensor has been commercialized and sold for use in engine testing and development. Other projects focused on development of improved catalytic materials and an improved sensor for monitoring NOx in diesel exhaust.  As shown by the sensor case study^{Footnote 45}, technology development can be long-term; in these examples, research began in 2005 and has only now reached the advanced prototype stage.  AFTER research on a novel catalyst is also being examined by Transport Canada for possible application in reducing NOx emissions from railway diesel engine exhaust. 

AFTER has achieved close relationships with necessary Canadian and international policy groups:  AFTER has developed close relationships with government transportation policy groups. It is important to remember that the research results from AFTER and other CTS programs are provided to stakeholders to inform future policy and regulatory decisions, which then impact fuel efficiency and reduce emissions.  To achieve maximum influence on decision-making, AFTER data must be provided to policy and regulatory groups in a form that they can understand and at an early enough stage when policies and regulations are being formulated.  The close relationship between policy groups and the AFTER program demonstrates the importance of AFTER research to transportation policy development.

International partners such as the US EPA and DOE transportation groups have close relationships with AFTER researchers in the development of knowledge useful to transportation policy and regulatory groups.  Often AFTER projects are complementary to US efforts.  Canadian and US groups share results and identify opportunities for joint projects through the annual TTFF and Canada – US Clean Air Dialogue.  Through the IEA AMF transit bus project, AFTER has also built a partnership with the VTT Technical Research Centre of Finland. 

Project results contributed to the development of policies, regulations and guidelines: Several projects involved gathering data on combustion emissions and fuel efficiency of various diesel engine technologies which is provided directly to EC, Health Canada, NRCan and Transport Canada policy and regulatory groups to support the development of transportation policies and regulations. For example, project data has been shared with the EC Oil, Gas and Alternative Energy Division responsible for fuel regulations, and Environment Canada’s Energy and Transportation Division responsible for transport regulations. The Fuel and Technology Alternatives for Buses project^{Footnote 46} gathered data at the EC ERMS facility on performance and emissions characteristics of a range of diesel fuel blends and transit bus technologies operating under different driving conditions which can be used to determine the best combination of engine and fuel to minimize fuel consumption and emissions.

Another project collected data on emissions from Light and Heavy Duty Vehicle diesel engines using various biodiesel blends and drive cycles, which was provided to the Health Canada Air Health Science Division and contributed to the development of the Renewable Fuels Strategy.  The AFTER report characterising emissions from light duty vehicle diesel engines operating on low sulphur diesel and biofuel blends was heavily cited in Health Canada’s “Risk Assessment of Biodiesel Use in Canada.”  Other projects led by Health Canada researchers under the Health and Environment sub-program collected data on the toxicity, health and environmental effects of emissions from engines using biofuels and other advanced fuels and pollution control technologies, as well as the toxicity of various biodiesels.^{Footnote 47} The data from these projects is being used to inform the development of future regulations on use of biofuel blends.

Results were disseminated to researchers and policy makers in Canada and the US:  The results of AFTER projects are disseminated in a number of ways. In addition to the normal mechanisms of dissemination to other researchers through publications, presentations at conferences, results are shared with policy groups and other stakeholders at semi-annual workshops where the research results are presented and discussed.  In terms of publications, AFTER has produced the second highest quantity of funded papers over the evaluation period (170, following H2FC with 430). As well, these publications are cited 26% more often than  the world average for papers in this field. AFTER also shares results with the US DOE and EPA.  The annual TTFF and the Canada – US Clean Air Dialogue, are attended by Canadian and US transportation related policy, regulatory and research groups, where research results are discussed in terms of the needs of the regulatory communities.  These are important mechanisms, particularly for harmonization of transportation regulations with the US. As described in the case study, the data from the Fuel and Technology Alternatives for Buses project is included in a major report entitled “Fuel and Technology Alternatives for Buses – Overall Energy Efficiency and Emission Performance” being published through the IEA AMF Implementing Agreement. This report will be shared widely with other countries and will inform policies, regulations and guidelines related to transit bus operation.

AFTER has made good progress towards reducing transportation related GHG and other emissions both in contributing to transportation emissions related policies and regulations and in developing new combustion engine control and aftertreatment technologies. AFTER has provided test data on emissions and fuel efficiency to Environment Canada and Natural Resources Canada, and the US Environmental Protection Agency that is being used to inform the development of next generation policies, including the harmonization of Canadian and US regulations. As well, AFTER projects have led to several patents and the development of a combustion instability sensor that has been commercialized and a prototype particulate monitoring sensor that is being examined for possible commercialization. AFTER has also developed a novel catalytic material with improved NOx emissions reduction capability that is being tested for possible commercial use. 

5.2.3.2 Unintended Outcomes

Have there been any unintended (positive or negative) outcomes?

Summary Findings

Positive unintended outcomes from AFTER have been the follow-on applications of data and technology to other CTS portfolio projects. Research outputs from AFTER have contributed to P&E and EM projects.  AFTER project results are also providing the basis for projects being carried out under the new ecoEII initiative.

Discussion and Analysis

While each of the five programs within the CTS portfolio is managed independently, there are opportunities for co-operation and sharing among programs.  There are a number of examples in which information obtained through AFTER projects has contributed to projects in other CTS Portfolio programs.  For example, the IEA AMF Bus project carried out by ERMS produced data and test procedures used by the Electric Mobility Program Projectentitled, “Plug-In Hybrid Electric Vehicles Energy Efficiency and Emissions” which examined emissions from electric hybrid vehicles.  The P&E Project “Identification and Quantification of Urea Thermal Decomposition By-Products in Diesel Engines Emissions – Method Development” also benefitted from data produced through the IEA AMF Bus project. Technology developed under the AFTER program, has also been used in projects in other CTS programs.  For example, the Laser–induced Illumination (LII) instrument has been used in four other P&E and AFTER projects.

AFTER research provides the basis for projects in other programs. For example, research on fuel performance will continue through an ecoEII project on renewable diesel fuels. Data on emissions and fuel efficiency of diesel fuel blends used in advanced diesel engine technologies is also relevant to the Transport Canada ecoTechnology for Vehicles Program.

5.2.3.3 Internal and External Factors

What are the factors (both internal and external) that have facilitated or hindered the achievement of expected results?

Summary Findings

Internal factors contributing to AFTER success include program longevity, the consistency of AFTER Program objectives through several funding cycles and the uniqueness of the Environment Canada ERMS test facilities.

External factors include the strength of linkages to policy and regulatory groups, US and international agencies which value the non-biased credible scientific data provided by AFTER research.

Discussion and Analysis

Internal Factors

Program maturity and consistency – AFTER was created in 2000 as a PERD Program.  While there have been minor changes in direction during the three funding cycles, the general objective has remained constant.  Laboratory equipment has been developed to meet the needs of the projects and relationships have been developed among the research groups in the five government departments participating in AFTER.  The program governance, project selection and oversight procedures are known and acknowledged by program participants as being of high quality.

Unique facilities of key federal partners – The Environment Canada Emissions Research and Measurement Section has unique capability for cold weather measurements that complement measurement and testing facilities in the US and other countries.  ERMS also has state of the art dynamometer testing and emissions measurement capability that is used extensively by AFTER and other programs to measure engine performance in terms of fuel efficiency and emissions.

External Factors

Canada-US policy on harmonizing regulations relies on strong linkages with US agencies – The AFTER Program’s network includes US agencies such as the Environmental Protection Agency responsible for developing American transportation policies and regulations.  AFTER interacts with US scientific and regulatory agencies through the Transportation Technology and Fuels Forum, an annual workshop where Canadian groups involved in transportation science, policy and regulatory issues meet with colleagues from the US DOE and EPA to discuss transportation related issues.  Interviewees noted that, because of Canada’s policy of harmonization with US transportation policies and regulations, these meetings are an important mechanism, to share Canadian data and influence US regulatory decisions.  The Canada –US Clean Air Dialogue is another mechanism for sharing of scientific data from AFTER projects identified by interviewees.  AFTER also provided data to SAE International, an organization that develops transportation related technical guidelines that complement policies and regulations developed by US agencies.

Linkages to international transportation research efforts – AFTER also has international linkages such as the IEA AMF and Combustion Implementing Agreements, which provide international fora for transportation related research.  One major AFTER project is part of a larger IEA AMF study of the fuel efficiency and emissions for variety of transit bus engine technologies, fuels and driving conditions.  The IEAAMF case study showed that, in addition to producing data for a range of conditions found in Canadian bus fleets, AFTER has been able to gain access to a large amount of complementary research data of value to Canadian policy and regulatory groups, as well as transit bus operators.

Linkages to Policy and Regulatory Groups – One of the roles of the members of the AFTER Management Committee is to maintain linkages with their departmental policy and regulatory groups. These linkages contribute to AFTER project selection and provision of research to support policy and regulatory work.  For example, AFTER research on emissions from light duty diesel vehicles operating on low sulphur diesel and biodiesel blends contributed to the Health Canada Risk Assessment of Biodiesel Use in Canada in support of the Renewable Fuels Strategy.  Interviewees noted that AFTER also influences US transportation related emissions and fuel efficiency policies and regulations through joint meetings with US agencies through the Transportation Technology and Fuels Forum (TTFF) and the Canada-US Clean Air Agenda.

5.2.3.4 Economic and Efficient

Are the programs and activities the most economic and efficient means of making progress towards intended outcomes?

Summary Findings

Several features of the AFTER program have contributed to its efficient and economic delivery. PERD, the primary source of AFTER funding, is a mature program that has developed efficient, effective procedures that are well known and understood by the federal R&D community.  Similarly, AFTER’s consistent objectives through three funding cycles have helped stakeholders understand and interact with the program. The requirement by PERD for at least matching funding has helped AFTER leverage funding from federal departments and other sources, resulting in a leverage ratio of 1:1.7. 

Discussion and Analysis

PERD, the primary source of funding for AFTER, requires matching contributions as a requirement for all projects.  To meet PERD requirements, AFTER has been able to leverage both funds and in-kind contributions from a variety of stakeholders.  In many cases, those government departments which led the research projects, such as Health Canada and Environment Canada, co-fund the projects.  Other funders include US government agencies and the IEA AMF. Some projects developing commercial products such as engine sensors or catalysts received contributions from an industrial partner.

Over the period 2007-08 to 2011-12, AFTER received $11.1 million in CTS funding, almost all of which was from PERD.  The Program leveraged in-kind and financial contributions from program partners and stakeholder totalling $18.7 million, for a total Program value of $29.8 million and an overall leverage ratio of 1:1.7. 

As discussed above, the maturity of the AFTER research program, the quality of the research infrastructure available for AFTER projects, its well established linkages to other transportation research, policy and regulatory groups contributed to the efficient and effective carrying out of AFTER projects and dissemination and application of research results.

5.2.4 Particles and Related Emissions (P&E) Program

5.2.4.1 Objectives Achievement

To what extent have intended outcomes been achieved as a result of the programs? 

Summary Findings

The evidence suggests that the P&E program has made good progress on its intended outcomes. Document review and interview findings show that P&E projects were carried out as planned.  Mid-year technical review meetings were conducted to review progress and provided an opportunity to refine project direction.  Projects have led to significant advances in air quality monitoring and modelling.  The data and new knowledge generated have strengthened the basis for decision making regarding environmental, transportation and health policy and regulations.

Another important outcome of the Program is the research network created among Environment Canada, Health Canada and NRC on emissions research.  The P&E program gave Health Canada and Environment Canada researchers broader scope and opportunities to focus on transportation sector emissions and their impacts, leading to better data and predictive models.  Research results have also been published in peer reviewed journals and are being cited 5% more frequently than the world average for papers in this field.  Both Canadian and US policy makers have used P&E findings.  Though a number of program stakeholders believe that use by Canadian policy groups could be improved through disseminating results more broadly and integrated with other related CTS results. 

Discussion and Analysis

Projects were selected in consultation with researchers and built on the previous funding cycle activity:  The P&E projects over the past five years were designed to:characterize exhaust emissions from a range of vehicle classes, engine technologies, fuels and other technologies; model the atmospheric fate of particulate matter and related atmospheric constituents; and identify the potential health and air quality impacts of transportation-related emissions, particularly the toxicity and mutagenicity of emissions related to advanced fuels and emission control technologies.

Key program stakeholders were involved in the development of project plans:  In the most recent funding cycle, 40% of P&E funding was allocated to the ‘Advancing Local-scale Modeling through Inclusion of Transportation Emission Experiments’ (ALMITEE) project.  As described in the ALMITEE case study the project is a multi-investigator, highly integrated research program that was developed by combining a number of smaller, related research projects in 2008-09.  The ALMITEE project evolved through a number of workshops with Environment Canada researchers to identify project tasks and objectives.  Along with Environment Canada staff, Health Canada and NRCan researchers and managers participated in the Program Committee and were involved in the development project plans.

The Program helped develop a network of researchers within, and across, Environment Canada, Health Canada and NRC:  Interviewees stated that without PERD funding this informal network would not have developed and there would be significantly less interaction among researchers at the different departments.  Interviewees report that the new relationships had a positive impact on related research activities at these departments.

The P&E Program gave Environment Canada and Health Canada opportunities to pursue research projects that would otherwise not have been undertaken:  The researchers interviewed for this study noted that the likelihood that they would have been able to pursue projects that address transportation emissions (primarily road transport) was slim, and cited the diesel emissions research that was funded by P&E as an important area where PERD funding was critical.  Much of this research addresses Health Canada’s regulatory requirement to reduce the adverse genetic effects from chemicals (e.g., diesel engine emissions).  PERD funding also allowed Health Canada the opportunity to expand its emissions research to look at cell exposure and whole animal exposure impacts from transportation emissions.  With PERD funding, joint projects between Health Canada and Environment Canada were pursued involving collaboration and exchanges of personnel between lab facilities for up to a month at a time.

The P&E Program supported interaction with the US stakeholder groups:  For example, an industry objective during the most recent funding cycle was to reduce the sulphur content in gasoline.  Environment Canada and the US EPA now have a shared workplan to address these needs, and Canadian researchers are working more closely with their US counterparts.

New methods for characterizing particles and related emissions were developed and applied over the 2007-08 to 2011-12 time period, which support ongoing monitoring and regulatory decision-making:  Emissions characterization research has been the central focus of the Program.  Advanced emission control technologies and new fuel blends, designed to reduce GHG and CAC emissions in accordance with new regulations, also affect the composition of particulate and gaseous emissions and may have unintended negative impacts.  Several P&E projects measured and modeled non-regulated emissions to understand the potential unintended short and long-term effects of new regulations and technologies. 

According to interviewees, detailed studies on a large number of biodiesel feedstocks, blend levels and engine operation conditions provided valuable scientific information in support of emissions regulations as well as the federal government’s Renewable Fuels Strategy (RFS).  The ‘Impact of Ethanol Blends on Emissions from Conventional, Advanced, and Emerging Technology Vehicles’ project performed emissions characterization of ethanol blends in cold Canadian conditions using various vehicle technologies.  These results are necessary for Canadian policy and regulatory decision makers due to Canada’s cold winter climate.  GHG emissions data from these studies were provided to Environment Canada’s Energy and Transportation Division and used to develop GHG emission regulations for new light duty vehicles in Canada.  The Energy and Transportation Division is currently working on tailpipe GHG regulations for heavy-duty vehicles.

New modeling capacity and models were developed:  The ability to model planned changes in fuel and emissions regulations and assess their potential impact on air quality, is a valuable tool in early stages of development.  PERD funding in the 2005-06 to 2008-09 timeframe supported the development of models with the capability to run annual and continental scale scenarios, and to link air quality models with health effect models, providing integrated information to decision makers. 

The ‘Advancing Local-scale Modeling through Inclusion of Transportation Emission Experiments (ALMITEE)’ project is a multi-investigator, highly integrated research program that takes results obtained from lab and field emissions measurements and feeds them into new emissions inventories and modeling programs so that more accurate air quality data is available to regulatory and policy decision makers.  Specific outcomes included:

• Increased understanding of the causes and sources of pollution in South-western Ontario:  The project resulted in a series of 19 peer-reviewed scientific papers that document the advances made in understanding the causes and sources of elevated pollutant events in South-western Ontario.  Research results are being used to improve the modelling system, and develop new approaches to measure and analyze atmospheric and air quality conditions.

• Laboratory and field measurements exposed weaknesses in current modelling systems: Detailed measurements of motor vehicle emissions in the laboratory and in the field in high traffic environments uncovered evidence of an under-appreciated process potentially forming a significant amount of motor-vehicle related organic particulate matter in the atmosphere.  P&E research showed that current values used to predict black carbon emissions from gasoline engine vehicles on the road are significantly under-estimated in the current models.  These results, published in peer-reviewed scientific journals, are relevant to air quality and climate change research, and point towards a potentially significant gap in worldwide emissions inventories.

Improved understanding of health and environmental effects:  P&E funded four projects at Health Canada with the objective to develop and apply measurement and analysis techniques, systems and studies, to identify acute and chronic effects on humans and the environment (air quality) that occur as a result of particles and other pollutants associated with transportation-related sources.  As illustrated by the VitroCell case study example, the P&E Program supported the installation of new technology to determine toxicological effects from engine exhaust using a variety of biodiesel blends.  While the VitroCell device has become increasingly popular in jurisdictions where animal tests are not permitted (e.g., European Community), according to interview findings only six of these devices are being used in North America and this was the first time the device was used in Canada. 

Research results are disseminated via a number of annual information dissemination events, peer review publications and presentations: P&E researchers, in cooperation with AFTER Program researchers, met annually to discuss the mid-year technical reports.  Approximately 30 to 40 representatives of Environment Canada and OGDs (researchers, policy and regulatory managers) attended each meeting.  Researchers and managers interviewed felt that the meetings were well managed and coordinated between P&E and AFTER. 

P&E researchers make other presentations to policy, legislative and regulatory staff (primarily at Environment Canada and Health Canada, but also NRCan and Transport Canada) when possible.  Individual project team members have presented their finding to US agencies (especially border states). 

Research results are published in peer-reviewed journals and presentations of findings are made at national and international scientific meetings.  Projects have led to a significant number of peer reviewed publications and presentations.  For example, in the final year of the ALMITEE project a total of 13 refereed / peer reviewed publications were generated, along with 2 internal technical reports, 4 presentations at conferences and symposia, and 2 invited presentations to stakeholder groups. P&E funded publications appear to have been highly influential in the field. The bibliometric analysis showed that P&E funded publications from 2007-08 to 2010-2011 were cited 92% more often than the world average for papers in this field. Moreover, the program accounted for approximately 13.3% of Canadian publications in this field.

Influence on policy groups has been stronger in US than in the Canadian federal system: Based on interviews and case study findings, it appears that to-date Canadian federal policy makers / regulators have shown less interest in research results than their US counterparts.   In the case of the ALMITEE project, several US States and research organizations have expressed an interest in the project methodology, technology / model, and model results.  The project leader has given two briefings to a South East Michigan stakeholder group (which includes the province of Ontario) and technical presentations to the US Electric Power Research Group (EPRG) and Health Effects Institute.

A number of interviewees felt that OERD should have a more prominent role to play in disseminating research results more broadly, and in a way that P&E findings are integrated with other, relevant CTS results.

The ultimate outcome of the P&E program is a reduction in the health, environmental and economic impacts of transportation-related particulate matter.  The program generates emissions data, models and new knowledge that can inform transportation policies, regulations and related technology planning and development to support the adoption of new, cleaner sustainable transportation fuels and systems.  While there is evidence that P&E research has contributed to policy discussions to-date (e.g., the Renewable Fuels Strategy) it is too soon to assess its long-term impact on the effectiveness of Canada’s transportation related policy, standards and regulations.   

The long-term potential of the program and its results to influence regulations, standards, and policies depends on the extent to which research findings are transferred from the researcher / scientific community to the policy and regulatory agencies in Canada, the US and internationally, through meetings as well as publications.  As noted earlier, stakeholders report that making the connection between science and policy continues to be a challenge and presents a challenge to the achievement of long-term results.

5.2.4.2 Unintended Outcomes

Have there been any unintended (positive or negative) outcomes?

No unintended outcomes were identified.

5.2.4.3 Internal and External Factors

What are the factors (both internal and external) that have facilitated or hindered the achievement of expected results?

Summary Findings

Factors contributing to the success of P&E include: the Program’s ten year history, leverage, research networks / relationships built between participating departments, and program oversight.

The close linkages with AFTER, may have contributed to a perception that P&E activities are more relevant to Environment Canada’s and Health Canada’s mandates and therefore more appropriately funded by departmental A-base than PERD, which has made it more difficult for P&E to secure CTS funding.

Discussion and Analysis

Internal factors

P&E related research has been underway for more than 10 years and this history supported program delivery: The P&E Program was created in 2000 and, while there have been some changes to program delivery over the past 12 years, Program objectives have remained constant.  Program management tools, researcher networks, and new research models developed over the past 12 years contributed to program success and cost-effectiveness.

Program planning supported program delivery:  Three meetings were held with stakeholders (P&E Management Committee, researchers, and Environment Canada regulators) to identify gaps and R&D opportunities.  A CD containing all proposals and reports was provided to workshop participants for their input on future project activity.

Program oversight (committee membership and level of engagement) supported program delivery:  The Program Management Committee includes representatives from Environment Canada, Health Canada, Transport Canada, NRC, and NRCan and one industry representative of an emissions control company.  The Committee remains active and is well engaged in the program, meeting semi-annually to review research project results and directions.  

External Factors

CTS portfolio-level communication could be improved to better support program results:  Several interviewees felt that greater interaction at the CTS portfolio level would help implement programs more efficiently and effectively than is possible in the absence of regular discussions among stakeholders on emerging R&D needs and priorities.  Interviewees noted the importance of modelling to risk assessments for new technologies related to environmental and health impacts, but felt that it was difficult to communicate the value of these activities to the CTS Portfolio beyond Environment Canada and Health Canada.  A number of interviewees stated that in previous PERD cycles there was better consultation with OGDs on plans and priorities. 

Linkages with US agencies have been strengthened through the program:  The P&E Program maintains research and policy linkages with the US DOE and EPA, at the researcher and program management levels.  Canada’s policy to harmonize its transportation regulations and policies with those of the US make these linkages very important to the successful planning and delivery of P&E activities.  It is through these relationships that Canada is able to influence, and anticipate US regulations.

5.2.4.4 Economic and Efficient

Are the programs and activities the most economic and efficient means of making progress towards intended outcomes?

Summary Findings

Evidence that the P&E Program was efficiently delivered includes the level of coordination and collaboration among participating departments and the AFTER Program, the active governance structure, and reporting / information dissemination.  P&E Program partner contributions (cash and in-kind) between 2007-08 and 2011-12 were $14.7 million from Transport Canada, Health Canada, Environment Canada, NRC and industry partners for a leverage ratio of 1:1.6.

Discussion and Analysis

The P&E Program received support from PERD and ecoETI funding programs totalling $9.8 million and accounting for approximately 12% of the total CTS Portfolio funding envelope.  The total investment in P&E activities from all other sources (financial and in-kind) was approximately $14.7 million, for a total overall investment in P&E research of $23.9 million and a leverage ratio of 1:1.6.

The Program engaged researchers primarily from Environment Canada and Health Canada in projects, each providing expertise and lab facilities and equipment (e.g., Environment Canada provides cold temperature testing capability for fuel efficiency).  According to interviewed Committee members a coordinated workplan with the AFTER Program helped ensure Program efficiency.

5.2.5   Advanced Structural Materials-Next Generation Vehicles

5.2.5.1 Objectives Achievement

To what extent have intended outcomes been achieved as a result of the programs? 

Summary Findings

The 11 ASM-NGV projects were defined by Canmet-MTL in consultation with industry partners and input from the Advisory and Industry committees.  The level of industry participation in the Program helps to ensure that projects reflect market realities and address key technical issues.

The Program has made strong progress on the following outcome areas: strengthened R&D collaborations / networks (public sector and private sector); new materials knowledge (e.g., magnesium, aluminum, ultra high strength steel) and R&D capacity; enhanced modeling capacity and research tools; and, new materials.  According to interviewees, the file review, and case studies, ASM-NGV projects have advanced industry’s understanding of the performance of lightweight materials in structural and powertrain applications and under demanding automotive operating conditions (e.g., crash, vibration, corrosion, etc.).

Discussion and Analysis

ASM-NGV program benefited from on-going industry participation in the program’s design and management:  The Program engaged stakeholders through its Industry Advisory Committee (10 members representing materials, auto parts supply and auto manufacturers) and a broader Advisory Committee that also includes co-delivery partners including Auto21, Automotive Partnerships Canada (APC), the US DOE, and NRC, NSERC, and Transport Canada.  Appropriately, there is significant overlap between the ASM-NGV and APC Advisory Committees’ memberships.  The Committees meet twice a year and those members interviewed for this study were satisfied with their opportunity to provide input to program planning and review results.

Projects were defined in consultation with industry partners and science based partners in US and Chinese governments: According to Program Annual Reports, seven projects were funded in 2007-08 and in 2010-11 four new projects were added (addressing new lightweight materials for electric motor components, improved thermoelectric materials for energy recovery and issues of materials compatibility with biofuels).^{Footnote 48}

The largest of the 11 ASM-NGV projects was the Magnesium Front End R&D (MFERD) initiative, which involves more than 100 researchers from China, USA and Canada.  MFERD is an on-going international collaborative effort sponsored by NRCan, the Chinese Ministry of Science and Technology (MOST)^{Footnote 49}, the US DOE and the United States Automotive Materials Partnership (USAMP) which is a consortium of Chrysler, Ford and General Motors.  NRCan supported the concept for the two-phase, $21 million research collaboration first through CliMRI and then ASM-NGV.  Phase I (2007-08 to 2009-10) focused on fundamental materials research; and Phase II (2010-11 to 2011-12) on manufacturing and assembly (the decision to implement Phase II was based on the progress of Phase I).

The MFERD project is aimed at developing the necessary materials knowledge and manufacturing expertise to support the production of major front-end components using magnesium alloys instead of conventional steels. The overall weight reduction is expected to be in the range of 30 to 40 kg per vehicle.  An international team led by General Motors developed the project’s ten task areas.  Canadian companies involved in the project (Centre Line, Promatek (Magma-Cosma), Husky and Meridian Technologies) represent different steps in the value chain.

The Turbocharger project^{Footnote 50} includes a significant investment of approximately 10% of ASM-NGV program spending and was defined in consultation with a number of industry partners representing various points on the supply chain. Extensive consultations were held during the planning phase with project partners (Dana Corporation and Novelis), and other stakeholders including original equipment manufacturers and parts suppliers (GM, Ford, Magna), and university researchers at McGill and McMaster universities. Turbocharging technologies are one strategy for increasing engine efficiency and reducing engine weight, and can deliver GHG reductions similar to competitive technologies (e.g., hybrid electric/combustion engines), at a lower cost. 

Projects responded to industry needs:  Based on interviews with program stakeholders (industry, OGDs and other research programs) ASM-NGV R&D proposals targeted key research needs, as identified by government labs and industry.  A review of program documentation shows significant R&D partnerships with industry and universities.

ASM-NGV helped support and develop a number of significant research networks and collaborations, both nationally and internationally. 

• Participation in the Clean Energy Dialogue (CED) began in March 2011 when Canadian federal departments (NRCan, Transport Canada, NRC) met with their US counterparts to identify possible research opportunities.  The Program leader is the key point of contact for CED materials related research and co-chaired a workshop on future collaboration in the automotive materials area and participated in several US DOE workshops on materials.

• The Program network includes the International Energy Agency - Advanced Materials for Transportation (IEA-AMT) Implementing Agreement.  NRCan’s participation and contributions to the IEA-AMT Implementing Agreement meetings in late 2010 and March 2011 built on the ASM-NGV Program strategy and project results.

• The MFERD project led to improved levels of co-ordination and collaboration (nationally and internationally).  Prior to MFERD, magnesium research in Canada was carried out with less formal coordination / collaboration by various organizations (e.g., Canmet-MTL, NRC-AMTC, NRC-IMI), and research consortia including Auto21 and the UBC NSERC Network for magnesium research (MagNet).  During project planning university interests were represented through Auto21, MagNet and the participation of some university researchers.  Industry consultations included Original Equipment Manufacturers (OEMs), Tier 1 suppliers, and materials companies.  The result is an international R&D project network with more than 100 researchers.

The Program supported the development of materials knowledge and R&D capacity in public and private sector organizations:  The use of advanced, lightweight material in structural and powertrain applications is vital to reducing vehicle weight and increasing powertrain efficiency.  However, information available on the performance of lightweight and powertrain materials under the demanding automotive operating conditions (e.g., crash, vibration, corrosion, etc.) is limited.  Basic (pre-competitive) ASM-NGV R&D projects, involving materials manufacturers, part suppliers and Original equipment manufacturers, have addressed these gaps for select materials.  For example:

• The understanding of how magnesium parts would perform in crash sensitive applications was minimal and the MFERD project helped fill some of the gaps.  New understanding of the behaviour of magnesium parts when subjected to dynamic loads similar to crashes was developed.  Other information gaps relate to how magnesium corrodes in demanding environments.  The project developed test methods to evaluate various materials and compared a number of corrosion protection techniques available on magnesium parts, and helped to re-direct research into multi-material components (i.e., a mix of magnesium and aluminum to achieve weight reduction while meeting crashworthiness / safety standards).

• With the relocation of MTL from Ottawa to Hamilton, some of the Turbocharger project R&D tasks were transferred to industrial partners, Dana and Novelis.  The companies were able to significantly advance the high service temperature alloy development and condensate corrosion areas of focus.  Novelis cast, hot- and cold-rolled, and heat-treated several variants of 3XXX aluminum alloys and Dana redesigned and built a scientific grade machine for condensate corrosion studies that was commissioned early in 2012.  The materials testing capability developed by the Turbocharger project partners (i.e., high temperature performance in corrosive environments) may extend to components with analogous operating requirements such as engine blocks, exhaust systems and fuel cell related infrastructure.

Modelling capacity and new research tools helped reduce the risks associated with materials R&D:  Developing new alloys and processing technologies is time consuming, expensive, and often less successful than planned, making it a high-risk endeavour.  Developing new simulation tools based on integrated computational materials engineering (ICME) can reduce the time and effort spent on developing new alloys.   Also, understanding how new alloys will behave in complex systems (i.e., automobile applications) requires models that can be applied in a predictive manner to assess material and component performance under adverse conditions (e.g., in crashes, under high temperature or pressure, etc.).  According to interviews, few industry members have the required expertise in complex modelling and Canmet-MTL plays a unique role by providing this capacity.  

New knowledge of material performance re-focused R&D strategies:  As noted above, the ASM-NGV projects are at earlier R&D stages involving proof of concept and validation experiments and thus new products are not expected in the immediate-term.  Materials’ R&D is targeted at specific automotive applications and the expectation is that project findings will help R&D performers refine / better focus their R&D strategies.   In the case of the MFERD project, research results proved that magnesium could not be used in some applications as hoped (due to its chemical and mechanical properties once formed), alternate possible material combinations were identified (e.g., multi-material options that would combine Mg with other metals such as aluminum) that could allow parts manufacturers to light-weight front-end components while maintaining safety standards.

Dissemination of research results has been limited in some cases to the direct use by project participants:  Due to intellectual property issues and proprietary material formulations, some research results cannot be widely shared (which is to be expected given the nature of ASM-NGV research).  Information on testing protocols, crash test results and material performance is shared more broadly through project reports, presentations and papers.  As noted earlier, annual meetings of the Advisory Committee are held to discuss research results and re-focus projects if needed.  According to the bibliometric analysis, publications resulting from ASM-NGV funded research between 2007 and 2011 are influential in the field. They are cited 67% more often than the world average for papers in this field. Moreover, the program accounted for approximately 20.5% of Canadian publications in this field.

It is too soon to see long-term impacts on GHG and CAC emissions or significant economic benefit.  Interview and document review findings show that ASM-NGV is well connected to Canadian industry and related research programs in the US.  The extent to which these technologies will contribute to a sustainable transportation sector, with net economic benefits to Canadians, will depend on commercial and market factors (e.g., new regulations, price of oil, competitiveness of the Canadian auto sector).  The achievement of long-term outcomes, related to GHG and CAC emissions and economic benefits, will follow from the future integration of lightweight materials in automotive structural applications.

Projects have led to further R&D investments by USDOE and the new federal ecoEII program, and also helped focus NSERC / university research.  Companies involved in MFERD have applied R&D results to further research activity.  For example, Cosma Engineering (Vehma International) is working with USAMP to develop front-end designs for a Ford 150 and a Cadillac.  While the designs will be proprietary, the R&D results related to weight reduction and crash performance will be shared.  The materials testing capability developed by the Turbocharger project partners (i.e., high temperature performance in corrosive environments) may extend to components with analogous operating requirements such as engine blocks, exhaust systems and fuel cell related infrastructure.

5.2.5.2 Unintended Outcomes

Have there been any unintended (positive or negative) outcomes?

• No unintended outcomes were identified for ASM-NGV.

5.2.5.3 Internal and External Factors

What are the factors (both internal and external) that have facilitated or hindered the achievement of expected results?

Summary Findings

A number of internal factors positively affected the achievement of Program results, including program governance, NRCan capacity (facilities and expertise), and sound program planning.  External factors with a positive impact on results included the program’s long history, industry involvement and alignment with international R&D priorities.  The financial challenges in the automotive sector and general economic conditions over the past five years had a negative impact on the Program.

Internal factors

Program governance supported program delivery:  The level of participation by industry, OGDs and other funding programs was identified as a significant positive factor.  The Advisory Committee includes representatives from the US DOE, Auto 21, APC and industry associations and meets twice a year.  Stakeholder feedback on program governance was that the planning and oversight processes were clear, and that the committees (which include automakers, suppliers, original equipment manufacturers) have a strong understanding of the North American auto market. 

Unique Canmet-MTL facilities and expertise were critical to the success of some ASM-NGV projects: Industry interviewees report that Canmet-MTL fills a niche role, providing unique access to computational methods, models and pilot scale testing facilities (e.g., twin roll casters) and expertise (e.g., casting, computational methods and modelling) needed by designers to help predict the performance of new materials in complex automotive systems under a range of conditions (e.g., crashworthiness, high temperature or pressure).  Some large original equipment manufacturers have in-house expertise in this area but smaller companies cannot maintain this specialized expertise and rely on universities and Canmet-MTL.

Relocation of facilities to Hamilton has meant closer ties with manufacturers but also impacted delivery of projects in the near term: According to key interviews and project documentation, the relocation of MTL facilities from Ottawa to Hamilton, Ontario and the resulting re-staffing (only 40% of MTL staff were retained) affected the delivery of some projects.  In some cases, research was continued in industry labs with the loan of NRCan equipment.  MTLs new facility houses several unique pieces, and combinations, of equipment in North America.  The move brings the lab closer to its industry stakeholder base and McMaster University, which is now building the McMaster Automotive Resource Centre adjacent to Canmet-MTL.

The number and type of industry partners likely had an impact on the overall success of a given project:  A requirement for each ASM-NGV project is that there be at least one industry participant. In cases where only one firm participated, several interviewees felt the overall risk to the project increased.  Projects with multiple industry partners, representing various steps in the vertical supply chain (e.g., Turbocharger project and MFERD), were viewed by these interviewees as being more successful in terms of meeting project objectives and disseminating research results.

External factors

Program history and experience supported project definition and selection: The ASM-NGV program followed from CLiMRI (which ran from 1999-2003 and then 2003-2007).  At this time there was an internal re-organization at CANMET-MTL to better address client needs and an Automotive Program was initiated.  Some of the projects within this Program were ASM-NGV funded.  One MTL manager estimates that the total value of the Automotive Program was $3 million, $1 million from the ASM-NGV program and the balance from A-base.  The ASM-NGV Program built on the eight-years of CLiMRI experience and the established relationships with the private sector and other program stakeholders.

Canadian research priorities are aligned with (or consider) international priorities and investments:  International interest in magnesium R&D in the auto sector allowed the Program to leverage R&D funds with investments made in the US and China. 

The Canada-US Clean Energy Dialogue (CED) was another opportunity to collaborate with the US.  The CED was initiated in 2009 to ‘strengthen bilateral collaboration on clean energy technologies and seek solutions for reducing greenhouse gas (GHG) emissions to accelerate the transition to a low-carbon economy’.^{Footnote 51}  Both countries, and domestic auto manufacturers, are investing heavily in developing alternative power trains (including for electric vehicles), more fuel-efficient vehicle designs, and low-emission engines and fuels.  Through these collaborative initiatives, favourable investment pathways can be narrowed and technology deployment barriers can be overcome.  

Over 2010-11 and 2011-12, ASM-NGV Program Leader and other NRCan staff participated in workshops to identify bilateral R&D collaboration opportunities, including joint exploration of advanced lightweight magnesium fabrication methods.  DOE and NRCan signed several bilateral R&D agreements under which they will collaborate on lightweight vehicles.  (CED Action Plan II began in 2012 and will run for two years.)

Economic conditions adversely affected the automotive sector’s R&D investments over the past five years:  A number of companies were not able to meet their original commitments to project activities as a result of the economic downturn in 2008.  The downturn led some companies to declare bankruptcy and others to reduce their longer-term R&D investments.   The most adversely affected project was the project to develop ‘New Polymer and Processing Methods for Mass Transit Applications’.  The initiative was designed to improve the environmental and mechanical performance of composite materials, particularly for mass transit and commercial vehicle applications.  When in-kind project contributions were included the total value of the proposed project was $1.1 million.  As project status reports and ASM-NGV Program Annual Reports show, delays due to uncertainties with PERD funding and the 2009 recession, led to a scaling back of the multi-partner component (i.e., in the planning phase (2007-08), nine companies expressed an interest, by 2008-09 only four companies continued to be interested and a consortium agreement was expected in 2009, and by 2010 one partner company had filed for bankruptcy and the remaining potential partners could not be convinced to join).  The result was a project with approximately one-half the original budget with support from PERD, NRC-IMI A-base and the Transportation Development Centre of Transport Canada. 

5.2.5.4 Economic and Efficient

Are the programs and activities the most economic and efficient means of making progress towards intended outcomes?

Summary Findings

Evidence that ASM-NGV has been carried out efficiently includes the level of coordination and collaboration with other federal initiatives, active governance structure, and industry participation as discussed above. Program leverage included approximately $1 million a year in A-base funds and industry in-kind and financial contributions. The contribution to ASM-NGV projects from non-PERD sources was $16 million, for a leverage ratio of 1:3.3.

Discussion and Analysis

PERD is the only funding program that provided support to ASM-NGV (almost $5 million over five years), and the Program represents 7.7% of the CTS funding envelope.  Each project involved industry members who provided in-kind and / or financial contributions.  The largest contributions were provided by the US and China to an international magnesium research project.  The contribution to ASM-NGV projects from non-PERD sources was $16 million, for a leverage ratio of 1:3.3.

The Program’s Advisory Committee was active over the past five years and included industry representation from original equipment manufacturers, parts suppliers, and materials companies.  This oversight helped ensure that industry and other stakeholders were kept current with, and able to provide input to, annual reviews of project results, and MTL / ASM-NGV input to related initiatives (e.g., participation in CED workshops, IEA meetings).

The Program has a balance of large and smaller projects; the larger investments have enabled project teams to collaborate with international stakeholders and leverage their PERD funding with funding from other Canadian initiatives.

6.0 Summary Conclusions and Recommendations

6.1   Conclusions

Relevance

Overall, documentary, interview and case study evidence suggests that a role for the Federal Government is justified by on-going sector needs.  The role for NRCan, and in particular the roles of OERD and PERD, vis-à-vis other actors (e.g., other government departments) has been well co-ordinated and worked well in most Program areas.  However, the number of other federally administered programs and other departments with an active interest in CTS research areas, may have introduced some confusion with respect to federal R&D priorities and strategies for clean transportation systems. 

Performance

Case studies, interviews and document reviews suggest that it is too early to show major commercial impacts for many of the projects.  The nature of project results has varied by program, however it appears that impacts to date have been largely accruing to direct participants – leading to further projects.  That said, across the Programs, interviewees suggested that OERD could play a more significant role in communicating these results within the sector and across the CTS portfolio. OERD has in the past convened CTS Portfolio Workshops and disseminated portfolio newsletters among portfolio project participants and the federal R&D community, though the last of these were in 2009 and 2011 respectively. In four of the programs, project results have contributed to the discussion and development of transportation policy and regulations (e.g., energy efficiency, fuels, engine systems). 

Interviews, document review, contribution and comparative analysis suggest that while the long established OERD-PERD process has worked comparatively well in most areas reviewed (not withstanding the complex programming situation), there is a need to consider CTS’s relationship to other federal transportation initiatives. Missing from the CTS portfolio level discussion is a relationship to federal strategy or guide for transportation R&D, one that identifies the criteria or rationale for funding allocations to each component program/activity in the context of what other federal initiatives are pursuing. There appears to be confusion with regard to Federal strategy, which creates a risk of inefficiencies in funding.  Interviews and documents suggest that placing the five Programs under the ‘CTS Portfolio’ umbrella created expectations concerning coordination, priority setting and funding allocation at the Portfolio level that were not met.

6.2   Recommendations

In light of the above findings, the evaluation makes the following recommendations:

1. NRCan OERD should clarify the CTS Portfolio’s position within the overall federal transportation R&D context by consulting with other federal departments that have related transportation R&D mandates in the planning process and on a regular basis while projects are underway.  Note that policy, codes, standards and regulatory needs should continue to be considered alongside overall energy performance targets, technology development and commercial goals.

2. NRCan OERD should ensure consistency in project selection processes across all CTS programs so that they are transparent to all program partners.

3. OERD should establish a strategy to disseminate the knowledge and transfer the technology progress gained on a project level more broadly across industry and policy makers within Canada.

Annex A: Evaluation Case Study Selections

The table below presents the case studies analyzed in this evaluation. In total, 14 case studies were selected.

Case Study	Description	Value	Timeframe
H2FC
Enabling Long Term Durability for Next Generation Co-Generation Fuel Cells (43.003)	This project focused predominantly on co-generation fuel cell requirements. There was a requirement to address critical durability gaps that existed in terms of reformatting tolerant MEA designs and MEA components that adversely affect catalyst degradation.	$950,000 (PERD) + $600,000 (industry, including associations)	2007-08 to March 2012
Fuel Cell Humidifier Development and Commercialization (43.004)	Humidification is one of the key components of a fuel cell system that affects performance, lifetime, size, cost and (ultimately) commercialization. The industry partner had developed a gas-to-gas membrane humidifier for fuel cell systems with a proven ability to achieve cost and performance targets needed for commercial systems.  PERD funding was needed for further development work to optimize performance, integrate the humidifier with customers’ systems, demonstrate reliability and prepare for manufacturing and commercialization.	$730,000 (PERD) + $822,000 (financial and in-kind from industry, university)	January 2009 – March 2012
Canadian Hydrogen Airports Project (40.009)	The project technology providers were amongst the most advanced firms in their respective hydrogen field of expertise. All technologies have been developed and many have been integrated in various commercial and industrial applications. Unfortunately, these technologies are not yet commercially viable, due to economics and lack of adequate infrastructure.  This project was intended to demonstrate viable, economic system for supplying hydrogen fuel to vehicles, power supply technologies and an enabler of deployment of fuel cell opportunities.	$ 1, 870, 000 (ecoETI) + $3, 000 000 (financial and in-kind from industry and provincial partners)	April 2008 – Anticipated completion year 2011
Innovative Processes for Prototyping High Volume, Low-Cost Durable Membrane-Electrode Assemblies (MEAs) for Stationary and Automotive Fuel Cells (40.012)	Proton Exchange Membrane Fuel Cells (PEMFCs) are considered a viable and innovative power source for future clean transportation and stationary applications. The objectives were to develop unique, high-volume manufacturing processes to produce low-cost, durable proton exchange membranes (PEM) and membrane-electrode-assemblies, and to install an accelerated durability testing (ADT) system at NRC.	$496,300 (CEF) + $291,900 (NRC A-base and industry)	November 2010 – March 2012
EM
High Performance Soft Magnetic Composites for Electric Motors of Hybrid Vehicles (52.003)	The perceived technology market niche was the interest in making a composite of a Sintered Flaky-Soft Magnetic Composites (SF-SMC) using a metallurgy, instead of water ionized iron components in order to increase the mechanical and magnetic properties of the weight reduced composite and therefore increase fuel efficiency. The component used was Iron-3%Silicon. This addition to the SFC composite process was hoped to reduce manufacturing costs.	$466 000 (PERD) + $212 000 (financial support from industry, university, and NGO partners)	July 2008- March 2012
Plug-in Hybrid Electric Vehicles (PHEVs) Energy Efficiency and Emissions (54.001)	This project was led by Environment Canada with some involvement by Transport Canada and US EPA Office of Transportation and Air Quality. This project will primarily conduct R&D on plug-in hybrid electric vehicles, but will also invest efforts on medium and heavy duty hybrid vehicles in anticipation that opportunities may arise for the modification or conversion of these hybrids in to the plug-in configuration. (Note that at least one medium duty demonstration is underway in NY City). The intent of this research is threefold, first to establish appropriate testing mechanisms for these vehicle systems under Canadian context, secondly to develop an understanding of the emissions and fuel economy of these systems under laboratory and real world conditions, and finally to foster the development of these systems in the private sector.	$ 432,000 (PERD) + $565, 000 (financial and in-kind support from industry, US EPA, TC, NRC, and EC)	Started April 2007 completed in March 2012
Impact of PHEV Charging on the Grid	The project examining the impact on electrical generation and distribution and GHG emissions due to penetration of PHEVs involved the development of software models by NRCan researchers. Models were developed for each of the provinces as well as Canada as a whole.	$432 224 (PERD) + $435,000 (financial and in-kind support from industry, NRCan A base, university, and IEA partners)	2006-07 to Current phase completed in March, 2012
Electric Mobility Research and Development Case Study	This case study examines three projects carrying out research and development on battery technology for PHEVs.  Two projects involve early and mid stage R&D carried out at the NRC aimed at new and improved technology for next generation batteries. The third project is applied close to market R&D being carried out by a battery manufacturer.	$3,282,000	2008 to March, 2012
P&E
Advanced Local Scale Modelling Through Transportation Emission Experiments (ALMITEE), Environment Canada (C12.007)	New policies and regulations to support the development of Canada’s clean transportation sector place new demands on existing air quality models and modeling capacity.  This project set out to develop a model that can predict changes in urban and regional air quality caused by specific changes in the transportation system (e.g., fuels, vehicle types, people and goods movement patterns) at high-resolution.	$2,193,000 (PERD)+ $1,222,000 (EC A-Base) + $152,000 (ecoEII and university in-kind)	April 2009 - March 2012
Deployment of the VitroCell System for In Vitro Toxicity Assessment of Vehicular Emissions at an Air-liquid Interface (C14.004A and B), Health Canada	Routine in vitro toxicological analyses of diluted vehicular exhaust, the results of which can be used to support transportation policy decisions, have been hampered by technical difficulties.  This project addressed the need to deploy a novel in vitro exposure apparatus (the VitroCell) that permits the delivery of diluted vehicular emissions to cultured cells at an air-liquid interface.  The VitroCell project fills a knowledge gap by providing improved testing of emissions with results that allow for a better understanding of their impact on human health.	$$283,400 (PERD) + $152,000 (HC and EC in-kind)	2009-10 to 2011-12
ASM-NGV
Lightweight Thermal Management Systems for Turbocharger Technologies (32.002)	Turbocharging technologies are an effective strategy for increasing engine efficiency. The North American auto market has few examples of turbocharged vehicles, with the exception of a European imports. The Turbocharger project aims to build a knowledge base for incorporating lightweight material alternatives into the design of improved turbocharger components (specifically thermal management systems) both with and without exhaust gas recirculation (EGR).	$485,000 (PERD) + $450,000 (MTL A-base) + $582,000 (industry in-kind)	April 2008 to March 2012
Magnesium Front End Research & Development (MFERD) Project (31.001)	This Project is an on-going, international collaborative effort jointly sponsored by Natural Resources Canada (NRCan), the Chinese Ministry of Science and Technology (MOST)Footnote 52, the United States Department of Energy (USDOE) and the United States Automotive Materials Partnership (USAMP, a consortium of Chrysler, Ford and General Motors).  This project explores the potential use of magnesium alloys for structural applications as one option to meet the auto sector’s vehicle weight reduction goals of 30% to 50% without compromising vehicle safety.	$1,485,000 (PERD)+ $4,645,000 (NRCan A-Base, Auto21, university, industry)	April 2007 – March 2012
AFTER
Fuel and Technology Alternatives for Buses	It is important to know the impact of various renewable fuels on GHG and other emissions from the various types of diesel engines in use, as they may affect policy and regulatory decisions.  This project addresses this issue through the investigation of the impact of fuel composition, particularly biodiesel blends and oil sands derived fuels, on fuel efficiency and emissions from various diesel engine configurations.	$183,000 (PERD) + $1,260,000 (financial and in-kind support from EC, TC, Canmet Abase, industry, and IEA)	2008-09 to April, 2012
Combustion Engine Sensors	Current and next generation diesel engines are required to meet tighter and tighter emission restrictions.  In the design and operation of current generation diesel engines, there is a trade-off between emissions of NOx and particulate matter (PM) or soot, both of which are considered as harmful pollutants.   A PM sensor can monitor soot levels and be used in an On-Board Diagnostic system to adjust EGR levels to create the optimum balance between NOx and PM emissions.  This case study examines two AFTER projects that involve development of sensors for combustion engines.	$430,000 (PERD) + $790,000 (financial and in-kind support from RMC and industry partners)	Current phase completed in March, 2012